You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@fluo.apache.org by GitBox <gi...@apache.org> on 2021/10/26 19:33:31 UTC
[GitHub] [fluo-muchos] brianloss opened a new pull request #414: Add optional Application Insights on azure
brianloss opened a new pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414
This PR allows for the Azure Application Insights Java agent to be
configured to run with the manager and tablet servers. When specified,
this will create an Application Insights object along with the cluster,
and metrics will be sent there. The default configuration sends
micrometer metrics, but the user could configure differently (e.g., to
send certain JMX metrics).
* Create workbook if either oms integration or app insights are used.
* Create application insights resource as part of azure cluster setup
* Configure Accumulo to run the application insights Java agent
* Clean up parameter to azureDeployLinuxCounters.json template to remove
duplication of the creation of the log analytics workspace name.
* Save the log analytics workspace resource ID in muchos.props along
with the workspace ID and key, since this value is needed to set up
the application insights resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r739326968
##########
File path: ansible/roles/azure/tasks/create_application_insights.yml
##########
@@ -0,0 +1,52 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create an Azure Application Insights resource.
+# 2. Update the az_appinsights_connection_string property in muchos.props
+
+- import_tasks: log_analytics_ws_common.yml
+- import_tasks: application_insights_common.yml
+
+- name: Create Application Insights resource using REST API
+ azure_rm_resource:
+ resource_group: "{{ resource_group }}"
+ provider: insights
+ resource_type: components
+ resource_name: "{{ app_insights_name }}"
+ api_version: "2020-02-02"
+ idempotency: yes
+ state: present
+ body:
+ location: "{{ location }}"
+ kind: java
+ properties:
+ ApplicationId: "{{ app_insights_name }}"
+ Application_Type: other
+ Flow_Type: Bluefield
+ Request_Source: rest
+ WorkspaceResourceId: "{{ az_logs_resource_id if az_logs_resource_id else az_logs_ws_resource_id }}"
+ register: app_insights
+
+- name: Update az_appinsights_connection_string in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^az_appinsights_connection_string\s*=\s*|^[#]\s*az_appinsights_connection_string\s*=\s*'
+ line: "az_appinsights_connection_string = {{ app_insights.response.properties.ConnectionString }}"
+ insertafter: '\[azure\]'
Review comment:
Updated in the latest commit. I changed the regex to `\[azure\]|az_app_insights_version\s*=`. insertafter uses the last match of that regex by default, so this will insert after the az_app_insights_version line if it's there (since that's in the [azure] section and therefore will be the later match). Otherwise, it will insert right after the [azure] section header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r739328508
##########
File path: ansible/roles/accumulo/templates/accumulo-env.sh
##########
@@ -121,6 +126,18 @@ case "$cmd" in
true
;;
esac
+{% if cluster_type == 'azure' and az_use_app_insights %}
+
+# Enable the application insights agent for tablet servers
+case "$cmd" in
+ tserver|master|manager)
Review comment:
Enabled for gc in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r738773420
##########
File path: ansible/roles/azure/tasks/create_application_insights.yml
##########
@@ -0,0 +1,52 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create an Azure Application Insights resource.
+# 2. Update the az_appinsights_connection_string property in muchos.props
+
+- import_tasks: log_analytics_ws_common.yml
+- import_tasks: application_insights_common.yml
+
+- name: Create Application Insights resource using REST API
+ azure_rm_resource:
+ resource_group: "{{ resource_group }}"
+ provider: insights
+ resource_type: components
+ resource_name: "{{ app_insights_name }}"
+ api_version: "2020-02-02"
+ idempotency: yes
+ state: present
+ body:
+ location: "{{ location }}"
+ kind: java
+ properties:
+ ApplicationId: "{{ app_insights_name }}"
+ Application_Type: other
+ Flow_Type: Bluefield
+ Request_Source: rest
+ WorkspaceResourceId: "{{ az_logs_resource_id if az_logs_resource_id else az_logs_ws_resource_id }}"
+ register: app_insights
+
+- name: Update az_appinsights_connection_string in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^az_appinsights_connection_string\s*=\s*|^[#]\s*az_appinsights_connection_string\s*=\s*'
+ line: "az_appinsights_connection_string = {{ app_insights.response.properties.ConnectionString }}"
+ insertafter: '\[azure\]'
Review comment:
This was a fix for an issue I saw where the lines to replace weren't there at all in muchos.props. That generally shouldn't be the case, since they are included in muchos.props.example. But in clearing the value out, I deleted the line and then it got added at the end of the file with the hosts, and setup failed. az_app_insights_version has a default value so it doesn't _have_ to be in muchos.props. It's also unlikely that value would be removed since it's in muchos.props.example, but I figured it might be better to be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] keith-turner commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r738697738
##########
File path: ansible/roles/accumulo/templates/accumulo-env.sh
##########
@@ -121,6 +126,18 @@ case "$cmd" in
true
;;
esac
+{% if cluster_type == 'azure' and az_use_app_insights %}
+
+# Enable the application insights agent for tablet servers
+case "$cmd" in
+ tserver|master|manager)
Review comment:
Could `gc` be added?
##########
File path: ansible/roles/azure/tasks/create_application_insights.yml
##########
@@ -0,0 +1,52 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create an Azure Application Insights resource.
+# 2. Update the az_appinsights_connection_string property in muchos.props
+
+- import_tasks: log_analytics_ws_common.yml
+- import_tasks: application_insights_common.yml
+
+- name: Create Application Insights resource using REST API
+ azure_rm_resource:
+ resource_group: "{{ resource_group }}"
+ provider: insights
+ resource_type: components
+ resource_name: "{{ app_insights_name }}"
+ api_version: "2020-02-02"
+ idempotency: yes
+ state: present
+ body:
+ location: "{{ location }}"
+ kind: java
+ properties:
+ ApplicationId: "{{ app_insights_name }}"
+ Application_Type: other
+ Flow_Type: Bluefield
+ Request_Source: rest
+ WorkspaceResourceId: "{{ az_logs_resource_id if az_logs_resource_id else az_logs_ws_resource_id }}"
+ register: app_insights
+
+- name: Update az_appinsights_connection_string in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^az_appinsights_connection_string\s*=\s*|^[#]\s*az_appinsights_connection_string\s*=\s*'
Review comment:
could this be simplified to something like the following?
```suggestion
regexp: '^([#]\s*)?az_appinsights_connection_string\s*=\s*'
```
##########
File path: ansible/roles/azure/tasks/create_application_insights.yml
##########
@@ -0,0 +1,52 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create an Azure Application Insights resource.
+# 2. Update the az_appinsights_connection_string property in muchos.props
+
+- import_tasks: log_analytics_ws_common.yml
+- import_tasks: application_insights_common.yml
+
+- name: Create Application Insights resource using REST API
+ azure_rm_resource:
+ resource_group: "{{ resource_group }}"
+ provider: insights
+ resource_type: components
+ resource_name: "{{ app_insights_name }}"
+ api_version: "2020-02-02"
+ idempotency: yes
+ state: present
+ body:
+ location: "{{ location }}"
+ kind: java
+ properties:
+ ApplicationId: "{{ app_insights_name }}"
+ Application_Type: other
+ Flow_Type: Bluefield
+ Request_Source: rest
+ WorkspaceResourceId: "{{ az_logs_resource_id if az_logs_resource_id else az_logs_ws_resource_id }}"
+ register: app_insights
+
+- name: Update az_appinsights_connection_string in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^az_appinsights_connection_string\s*=\s*|^[#]\s*az_appinsights_connection_string\s*=\s*'
+ line: "az_appinsights_connection_string = {{ app_insights.response.properties.ConnectionString }}"
+ insertafter: '\[azure\]'
Review comment:
Could this be the following? not sure the order it runs relative to create_log_analytics_ws.yml supports this.
```suggestion
insertafter: '^az_app_insights_version'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r738771283
##########
File path: ansible/roles/azure/tasks/create_application_insights.yml
##########
@@ -0,0 +1,52 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create an Azure Application Insights resource.
+# 2. Update the az_appinsights_connection_string property in muchos.props
+
+- import_tasks: log_analytics_ws_common.yml
+- import_tasks: application_insights_common.yml
+
+- name: Create Application Insights resource using REST API
+ azure_rm_resource:
+ resource_group: "{{ resource_group }}"
+ provider: insights
+ resource_type: components
+ resource_name: "{{ app_insights_name }}"
+ api_version: "2020-02-02"
+ idempotency: yes
+ state: present
+ body:
+ location: "{{ location }}"
+ kind: java
+ properties:
+ ApplicationId: "{{ app_insights_name }}"
+ Application_Type: other
+ Flow_Type: Bluefield
+ Request_Source: rest
+ WorkspaceResourceId: "{{ az_logs_resource_id if az_logs_resource_id else az_logs_ws_resource_id }}"
+ register: app_insights
+
+- name: Update az_appinsights_connection_string in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^az_appinsights_connection_string\s*=\s*|^[#]\s*az_appinsights_connection_string\s*=\s*'
Review comment:
I think that would work, but will have to test to make sure the regex lib used by ansible handles that. I doubt it wouldn't though. I was following the pattern from several other places where lines are modified in muchos.props.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r738770272
##########
File path: ansible/roles/accumulo/templates/accumulo-env.sh
##########
@@ -121,6 +126,18 @@ case "$cmd" in
true
;;
esac
+{% if cluster_type == 'azure' and az_use_app_insights %}
+
+# Enable the application insights agent for tablet servers
+case "$cmd" in
+ tserver|master|manager)
Review comment:
Yeah, it definitely could be. I was thinking initially this would be used for capturing opentelemetry data when that work is done, and also some profiling integration that will be coming. gc didn't seem to matter for either of those use cases, but I can add it for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] keith-turner merged pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
keith-turner merged pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #414: Add optional Application Insights on azure
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #414:
URL: https://github.com/apache/fluo-muchos/pull/414#discussion_r739328262
##########
File path: ansible/roles/azure/tasks/create_application_insights.yml
##########
@@ -0,0 +1,52 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create an Azure Application Insights resource.
+# 2. Update the az_appinsights_connection_string property in muchos.props
+
+- import_tasks: log_analytics_ws_common.yml
+- import_tasks: application_insights_common.yml
+
+- name: Create Application Insights resource using REST API
+ azure_rm_resource:
+ resource_group: "{{ resource_group }}"
+ provider: insights
+ resource_type: components
+ resource_name: "{{ app_insights_name }}"
+ api_version: "2020-02-02"
+ idempotency: yes
+ state: present
+ body:
+ location: "{{ location }}"
+ kind: java
+ properties:
+ ApplicationId: "{{ app_insights_name }}"
+ Application_Type: other
+ Flow_Type: Bluefield
+ Request_Source: rest
+ WorkspaceResourceId: "{{ az_logs_resource_id if az_logs_resource_id else az_logs_ws_resource_id }}"
+ register: app_insights
+
+- name: Update az_appinsights_connection_string in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^az_appinsights_connection_string\s*=\s*|^[#]\s*az_appinsights_connection_string\s*=\s*'
Review comment:
Updated in latest commit. I ended up going with `^\s*[#]?\s*az_appinsights_connection_string\s*=` since there could be whitespace before or after the `#` if it exists. If there's no `#`, then `\s*\s*` is superfluous but will still match optional leading whitespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org