You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/10 09:20:48 UTC
[GitHub] [airflow] kumargauravin opened a new pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
kumargauravin opened a new pull request #10273:
URL: https://github.com/apache/airflow/pull/10273
Initial version of code for review.
Needs to be tested with real account which has permissions which I do not have at this time.
Some ToDo left added as comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-996095985
Rest some suggestion were thr it may have to be rebased.
On Thu, Dec 16, 2021 at 2:02 PM Kumar Gaurav ***@***.***>
wrote:
> Test cases if you could please I am occupied with something else.
>
> Thanks in advance.
>
> On Thu, Dec 16, 2021 at 6:06 AM Malthe Borch ***@***.***>
> wrote:
>
>> @kumargauravin <https://github.com/kumargauravin> I am interested in
>> this functionality. Does the code basically work and what's needed from
>> here is to get the pull request into good merge shape (i.e., sort out
>> whatever issues have been mentioned).
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/apache/airflow/pull/10273#issuecomment-995688612>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ADK64R34MYROAQFNLH35EG3URHB3HANCNFSM4PZYJCCQ>
>> .
>> Triage notifications on the go with GitHub Mobile for iOS
>> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
>> or Android
>> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>>
>>
> --
> Regards,
> Kumar Gaurav
> +1 469 354 2101
>
--
Regards,
Kumar Gaurav
+1 469 354 2101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r468176857
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
@kumargauravin , I actually don't think it's ok to provide access key in the way you're currently doing it. We have `connection` table to manage authentication information. You can take inspiration from the other hooks in the Azure package. That's what I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r468189087
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
Check the video here https://github.com/apache/airflow/blob/master/BREEZE.rst for setting up breeze environment.
Try using the Azure Batch hook https://github.com/apache/airflow/blob/master/airflow/providers/microsoft/azure/hooks/azure_batch.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-996095684
Test cases if you could please I am occupied with something else.
Thanks in advance.
On Thu, Dec 16, 2021 at 6:06 AM Malthe Borch ***@***.***>
wrote:
> @kumargauravin <https://github.com/kumargauravin> I am interested in this
> functionality. Does the code basically work and what's needed from here is
> to get the pull request into good merge shape (i.e., sort out whatever
> issues have been mentioned).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/pull/10273#issuecomment-995688612>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADK64R34MYROAQFNLH35EG3URHB3HANCNFSM4PZYJCCQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
Regards,
Kumar Gaurav
+1 469 354 2101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle edited a comment on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-717785881
For testing the task handler you can check out [test_es_task_handler.py](https://github.com/apache/airflow/blob/master/tests/providers/elasticsearch/log/test_es_task_handler.py) for example. It contains unittest by simply mocking elasticsearch. You would habe to mock azlaws. You can do this by simply using `unittest.mock`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-706789472
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-714502050
hi,
Yes not much knowledge about testing if someone can help getting this pass
I will get idea.
Regards,
Kumar
On Thu, Oct 22, 2020 at 2:17 PM Felix Uellendall <no...@github.com>
wrote:
> Hey @kumargauravin <https://github.com/kumargauravin> how is this going?
> Do you need any help?
>
> The tests are failing because you must add tests for the provider class
> you added. The config test is also failing. Can you fix that?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/pull/10273#issuecomment-714336167>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADK64R4L2S7CFS6XGUMCPT3SL7WQRANCNFSM4PZYJCCQ>
> .
>
--
Regards,
Kumar Gaurav
+1 469 354 2101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-751239984
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-671249829
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-671262962
You need to install pre-commit in your development environment https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#pre-commit-hooks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-671645954
Hi,
Let’s set priorities.
1. Pylint: I will go with the video and see what I understand.
2. Complete ToDo sections and how we can test it.
3. What else needs to be done in order to use the PI.
4. Anything else we require to add to PR. Any other SOP issues you see!
5. Movement of config to table as per example provided.
Can you contribute also like in any one points like pylint will be helpful.
Add any missing things and let’s get this close and merged so that we can
use!!!
Regards,
Kumar
On Mon, Aug 10, 2020 at 4:18 PM Kamil Breguła <no...@github.com>
wrote:
> *@mik-laj* commented on this pull request.
> ------------------------------
>
> In airflow/providers/microsoft/azure/hooks/laws.py
> <https://github.com/apache/airflow/pull/10273#discussion_r468191537>:
>
> > + self.conn_id = remote_conn_id
> + self.account_id = account_id
> + self.access_key = access_key
> + self.table_name = table_name
> +
> + def build_signature(self, date, content_length, method, content_type, resource):
> + x_headers = 'x-ms-date:' + date
> + string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
> + bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
> + decoded_key = base64.b64decode(self.access_key)
> + encoded_hash = base64.b64encode(
> + hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
> + ).decode()
> + authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
> + return authorization
> +
>
> In the case of Stackdriver, we used configurations via airflow.cfg. It is
> easier to use, but it is not a generally accepted rule.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/pull/10273#discussion_r468191537>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADK64R7FNRVPNAU72DSAPTLSABP2VANCNFSM4PZYJCCQ>
> .
>
--
Regards,
Kumar Gaurav
+1 469 354 2101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r468147109
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
@mik-laj @turbaszek what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r477182196
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,118 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+from datetime import datetime
+
+
+import hashlib
+import hmac
+import base64
+import requests
+
+from airflow.hooks.base_hook import BaseHook
+
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self, remote_conn_id, account_id, access_key, table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ """Builds signature for Azure HTTP Data Collector API"""
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = (
+ method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ ) # pylint: disable=line-too-long
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(
+ decoded_key, bytes_to_hash, digestmod=hashlib.sha256
+ ).digest() # pylint: disable=line-too-long
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
+ @staticmethod
+ def _clean_execution_date(execution_date: datetime) -> str:
+ """
+ Clean up an execution date so that it is safe to query in log analytics
+ by formatting it correctly
+
+ :param execution_date: execution date of the dag run.
+ """
+ return execution_date.strftime("%Y-%m-%d %H:%M:%S")
+
+ def post_log(self, log, ti, ssl_verify=True):
+ """
+ Post data to Azure Log Analytics
+ """
+ execution_date = self._clean_execution_date(ti.execution_date)
+ body = {
+ "dag_id": ti.dag_id,
+ "task_id": ti.task_id,
+ "execution_date": execution_date,
+ "try_number": ti.try_number,
+ "part": 1,
+ "raw_data": log,
+ }
+ # TODO: Break content into said size like 20M.
+ custom_table_name = self.table_name
+ method = 'POST'
+ content_type = 'application/json'
+ resource = '/api/logs'
+ rfc1123date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
+ content_length = len(body)
+ signature = self.build_signature(rfc1123date, content_length, method, content_type, resource)
+ uri = (
+ 'https://' + self.account_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
+ )
+
+ headers = {
+ 'content-type': content_type,
+ 'Authorization': signature,
+ 'Log-Type': custom_table_name,
+ 'x-ms-date': rfc1123date,
+ }
+ # TODO: Option to print to console instead API.
+ try:
+ response = requests.post(uri, data=body, headers=headers, verify=ssl_verify)
+ response.raise_for_status()
Review comment:
I think thats what a basic [HttpHook](https://github.com/apache/airflow/blob/master/airflow/providers/http/hooks/http.py) also does? WDYT of inheriting from `HttpHook` instead of `BaseHook`? There are a lot of similiarities.
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,118 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+from datetime import datetime
+
+
+import hashlib
+import hmac
+import base64
+import requests
+
+from airflow.hooks.base_hook import BaseHook
+
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self, remote_conn_id, account_id, access_key, table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ """Builds signature for Azure HTTP Data Collector API"""
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = (
+ method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ ) # pylint: disable=line-too-long
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(
+ decoded_key, bytes_to_hash, digestmod=hashlib.sha256
+ ).digest() # pylint: disable=line-too-long
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
+ @staticmethod
+ def _clean_execution_date(execution_date: datetime) -> str:
+ """
+ Clean up an execution date so that it is safe to query in log analytics
+ by formatting it correctly
+
+ :param execution_date: execution date of the dag run.
+ """
+ return execution_date.strftime("%Y-%m-%d %H:%M:%S")
+
+ def post_log(self, log, ti, ssl_verify=True):
+ """
+ Post data to Azure Log Analytics
+ """
+ execution_date = self._clean_execution_date(ti.execution_date)
+ body = {
+ "dag_id": ti.dag_id,
+ "task_id": ti.task_id,
+ "execution_date": execution_date,
+ "try_number": ti.try_number,
+ "part": 1,
+ "raw_data": log,
+ }
+ # TODO: Break content into said size like 20M.
+ custom_table_name = self.table_name
+ method = 'POST'
+ content_type = 'application/json'
+ resource = '/api/logs'
+ rfc1123date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
+ content_length = len(body)
+ signature = self.build_signature(rfc1123date, content_length, method, content_type, resource)
+ uri = (
+ 'https://' + self.account_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
+ )
Review comment:
```suggestion
uri = f'https://{self.account_id}.ods.opinsights.azure.com{resource}?api-version=2016-04-01'
```
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,118 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on a change in pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r477338126
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,118 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+from datetime import datetime
+
+
+import hashlib
+import hmac
+import base64
+import requests
+
+from airflow.hooks.base_hook import BaseHook
+
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self, remote_conn_id, account_id, access_key, table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ """Builds signature for Azure HTTP Data Collector API"""
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = (
+ method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ ) # pylint: disable=line-too-long
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(
+ decoded_key, bytes_to_hash, digestmod=hashlib.sha256
+ ).digest() # pylint: disable=line-too-long
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
+ @staticmethod
+ def _clean_execution_date(execution_date: datetime) -> str:
+ """
+ Clean up an execution date so that it is safe to query in log analytics
+ by formatting it correctly
+
+ :param execution_date: execution date of the dag run.
+ """
+ return execution_date.strftime("%Y-%m-%d %H:%M:%S")
+
+ def post_log(self, log, ti, ssl_verify=True):
+ """
+ Post data to Azure Log Analytics
+ """
+ execution_date = self._clean_execution_date(ti.execution_date)
+ body = {
+ "dag_id": ti.dag_id,
+ "task_id": ti.task_id,
+ "execution_date": execution_date,
+ "try_number": ti.try_number,
+ "part": 1,
+ "raw_data": log,
+ }
+ # TODO: Break content into said size like 20M.
+ custom_table_name = self.table_name
+ method = 'POST'
+ content_type = 'application/json'
+ resource = '/api/logs'
+ rfc1123date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
+ content_length = len(body)
+ signature = self.build_signature(rfc1123date, content_length, method, content_type, resource)
+ uri = (
+ 'https://' + self.account_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
+ )
+
+ headers = {
+ 'content-type': content_type,
+ 'Authorization': signature,
+ 'Log-Type': custom_table_name,
+ 'x-ms-date': rfc1123date,
+ }
+ # TODO: Option to print to console instead API.
+ try:
+ response = requests.post(uri, data=body, headers=headers, verify=ssl_verify)
+ response.raise_for_status()
Review comment:
OK will try that out. Thanks for your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-714336167
Hey @kumargauravin how is this going? Do you need any help?
The tests are failing because you must add tests for the provider class you added. The config test is also failing. Can you fix that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-718741228
Sure Felix,
Haven't tried will try soon and update.
Regards,
Kumar
On Wed, Oct 28, 2020 at 2:14 PM Felix Uellendall <no...@github.com>
wrote:
> For testing the task handler you can check out test_es_task_handler.py
> <https://github.com/apache/airflow/blob/master/tests/providers/elasticsearch/log/test_es_task_handler.py>
> for example. It contains unittest by simply mocking elasticsearch. You
> would habe to mock azlaws.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/pull/10273#issuecomment-717785881>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADK64R73WNSLQPAHZMTBWMDSM7KV3ANCNFSM4PZYJCCQ>
> .
>
--
Regards,
Kumar Gaurav
+1 469 354 2101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-717785881
For testing the task handler you can check out [test_es_task_handler.py](https://github.com/apache/airflow/blob/master/tests/providers/elasticsearch/log/test_es_task_handler.py) for example. It contains unittest by simply mocking elasticsearch. You would habe to mock azlaws.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r468191537
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
In the case of Stackdriver, we used configurations via airflow.cfg. It is easier to use, but it is not a generally accepted rule.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r468176857
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
@kumargauravin , I actually don't think it's ok to provide access key in the way you're currently doing it. We have `connection` table to manage authentication information. You can take inspiration from the other hooks in the Azure package. That's what I think. :\
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r467961714
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
I see a lot of breaking changes commits per version of these libraries.
Further, the real thing to look for a library would be if it handles the split functionality due to API limit. Which I did not found here. But will check more. For our purpose and full control though, I think this will provide more control. Let me know your thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r468182356
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
OK. That is a nice idea. Can you refer to anyone and I will try to work in same way. Also for pylint, I may need some help setting up on that breeze setup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kumargauravin commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
kumargauravin commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-678727037
@feluelle @ephraimbuddy Request to check once and suggest. I want to complete once cycle of build tests and then continue to work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #10273:
URL: https://github.com/apache/airflow/pull/10273
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] malthe commented on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
malthe commented on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-995688612
@kumargauravin I am interested in this functionality. Does the code basically work and what's needed from here is to get the pull request into good merge shape (i.e., sort out whatever issues have been mentioned).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10273: Azure Log Aanalytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r467795320
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+from datetime import datetime
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self,remote_conn_id,account_id,access_key,table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
Review comment:
Can you check if we can do this with any of these libraries:
https://pypi.org/project/azure-mgmt-loganalytics/0.7.0/
https://pypi.org/project/azure-loganalytics/0.1.0/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle edited a comment on pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#issuecomment-717785881
For testing the task handler you can check out [test_es_task_handler.py](https://github.com/apache/airflow/blob/master/tests/providers/elasticsearch/log/test_es_task_handler.py) for example. It contains unittest by simply mocking elasticsearch. You would have to mock azlaws. You can do this by simply using `unittest.mock`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #10273: Azure Log Analytics Workspace Task Logs Ingestion
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #10273:
URL: https://github.com/apache/airflow/pull/10273#discussion_r513263047
##########
File path: airflow/providers/microsoft/azure/hooks/laws.py
##########
@@ -0,0 +1,118 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""
+This module contains integration with Azure LAWS.
+
+"""
+from datetime import datetime
+
+
+import hashlib
+import hmac
+import base64
+import requests
+
+from airflow.hooks.base_hook import BaseHook
+
+
+class LawsHook(BaseHook):
+ """
+ Interacts with Azure LAWS through api.
+
+ :param remote_conn_id: Not Used
+ :type remote_conn_id: str
+ :param account_id: Laws Account Id
+ :type account_id: str
+ :param access_key: Access Key
+ :type access_key: str
+ :param table_name: <Table Name>_CL
+ :type table_name: str
+ """
+
+ def __init__(self, remote_conn_id, account_id, access_key, table_name):
+ super().__init__()
+ self.conn_id = remote_conn_id
+ self.account_id = account_id
+ self.access_key = access_key
+ self.table_name = table_name
+
+ def build_signature(self, date, content_length, method, content_type, resource):
+ """Builds signature for Azure HTTP Data Collector API"""
+ x_headers = 'x-ms-date:' + date
+ string_to_hash = (
+ method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
+ ) # pylint: disable=line-too-long
+ bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
+ decoded_key = base64.b64decode(self.access_key)
+ encoded_hash = base64.b64encode(
+ hmac.new(
+ decoded_key, bytes_to_hash, digestmod=hashlib.sha256
+ ).digest() # pylint: disable=line-too-long
+ ).decode()
+ authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash)
+ return authorization
+
+ @staticmethod
+ def _clean_execution_date(execution_date: datetime) -> str:
+ """
+ Clean up an execution date so that it is safe to query in log analytics
+ by formatting it correctly
+
+ :param execution_date: execution date of the dag run.
+ """
+ return execution_date.strftime("%Y-%m-%d %H:%M:%S")
+
+ def post_log(self, log, ti, ssl_verify=True):
+ """
+ Post data to Azure Log Analytics
+ """
+ execution_date = self._clean_execution_date(ti.execution_date)
+ body = {
+ "dag_id": ti.dag_id,
+ "task_id": ti.task_id,
+ "execution_date": execution_date,
+ "try_number": ti.try_number,
+ "part": 1,
+ "raw_data": log,
+ }
+ # TODO: Break content into said size like 20M.
+ custom_table_name = self.table_name
+ method = 'POST'
+ content_type = 'application/json'
+ resource = '/api/logs'
+ rfc1123date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
+ content_length = len(body)
+ signature = self.build_signature(rfc1123date, content_length, method, content_type, resource)
+ uri = (
+ 'https://' + self.account_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
+ )
+
+ headers = {
+ 'content-type': content_type,
+ 'Authorization': signature,
+ 'Log-Type': custom_table_name,
+ 'x-ms-date': rfc1123date,
+ }
+ # TODO: Option to print to console instead API.
+ try:
+ response = requests.post(uri, data=body, headers=headers, verify=ssl_verify)
+ response.raise_for_status()
Review comment:
Did that work? Or haven't you tried that yet?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org