You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "yswdqz (via GitHub)" <gi...@apache.org> on 2023/02/11 16:47:33 UTC
[GitHub] [skywalking] yswdqz opened a new pull request, #10380: Support monitoring DynamoDB.
yswdqz opened a new pull request, #10380:
URL: https://github.com/apache/skywalking/pull/10380
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [x] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [x] If it's UI related, attach the screenshots below.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
This PR is not finished yet, but I've encountered some problems, so I'm sending this PR first in order to ask some questions and report on my progress.
Problems:
1. I design the ui with reference to Amazon CloudWatch, and they use a method called `PERIOD`, Is there such a function in mal?
doc is [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html)
2. The read usage meter is like
But the real data is always 1
I think it should be a horizontal line rather than a diagonal line? And is it because of my mal's `avg` function?
(also, I find I'm still not familiar with the metric model, are there any articles that go into detail about the metric model?)
3. About how to set service level and instance level, I think that DynamoDB is based on table. So I set table as instance level, and set AWS account ID as service level , is that ok?
TODO:
1. set up service level dashboard.
2. try to test left metrics :
metrics I have not tested:
①read/write_throttled_requests
②read/write_throttle_events
③read/wirte system error
④user error
⑤conditional_check_failed_requests
⑥transaction_conflict
3. add ui menu
4. remove redundant comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10380: Support monitoring DynamoDB.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10380:
URL: https://github.com/apache/skywalking/pull/10380#discussion_r1103720880
##########
docs/en/setup/backend/backend-aws-dynamodb-monitoring.md:
##########
@@ -0,0 +1,38 @@
+# AWS DynamoDb monitoring
+SkyWalking leverages Amazon Kinesis Data Filehose with [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/) to transfer the metrics into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+1. Amazon CloudWatch fetches metrics from DynamoDB and pushes metrics to SkyWalking OAP Server via Amazon Kinesis data firehose.
+2. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
+
+### Set up
+1. Enable [AWS CloudWatch](https://aws.amazon.com/cn/cloudwatch/)
+2. Create [Amazon Kinesis Data Filehose](https://aws.amazon.com/cn/kinesis/data-firehose/), set source to `Direct PUT`, set destination to `HTTP Endpoint`, and set `HTTP EndPoint url` to `aws-firehose-receiver`'s port (refer to [aws-firehose-receiver](aws-firehose-receiver.md))
+
+ Note that AWS requires that the `HTTP Endpoint url` must be HTTPS and the port needs to be 443, so you can load the certificate in [aws-firehose-receiver](aws-firehose-receiver.md) and set the port to 443. Also, you can use another gateway to accept the request and route it to `aws-filehose-receiver`.
+3. Create a [metric stream](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html), set namespace to DynanoDB, and set `Kinesis Data Firehose` to the firehose you just created.
+4. Config [aws-firehose-receiver](aws-firehose-receiver.md) to receive data.
+
+### DynamoDB Monitoring
+DynamoDB monitoring provides monitoring of the status and resources of the DynamoDB server. AWS user id is cataloged as a `Layer: AWS_DYNAMODB` `Service` in OAP.
+Each DynamoDB table is cataloged as an `Instance` in OAP.
Review Comment:
The table should be closer to the `endpoint` concept. Because table is a logic concept rather than a physical deployment concept like pods or processes in a cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10380: Support monitoring DynamoDB.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10380:
URL: https://github.com/apache/skywalking/pull/10380#issuecomment-1426918708
Is https://github.com/apache/skywalking/pull/10302#issuecomment-1424142361 addressed and fixed? I think we need that fix.
@pg-yang Is the e2e mock data load for AWS firehose ready to use? This PR should be covered by that style e2e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10380: Support monitoring DynamoDB.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10380:
URL: https://github.com/apache/skywalking/pull/10380#discussion_r1103720431
##########
docs/menu.yml:
##########
@@ -199,6 +199,8 @@ catalog:
path: "/en/setup/backend/backend-mysql-monitoring"
- name: "PostgreSQL Server"
path: "/en/setup/backend/backend-postgresql-monitoring"
+ - name: "AWS-DynamoDB Server"
Review Comment:
```suggestion
- name: "AWS DynamoDB"
```
This should be placed in the AWS monitoring too.
<img width="291" alt="image" src="https://user-images.githubusercontent.com/5441976/218288490-2f54b77b-2894-431e-846b-d5b03e29053a.png">
Also, you should update UI repository to add this feature into two places(database and AWS catalogs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz closed pull request #10380: Support monitoring DynamoDB.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz closed pull request #10380: Support monitoring DynamoDB.
URL: https://github.com/apache/skywalking/pull/10380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10380: Support monitoring DynamoDB.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10380:
URL: https://github.com/apache/skywalking/pull/10380#discussion_r1103720677
##########
docs/en/setup/backend/backend-aws-dynamodb-monitoring.md:
##########
@@ -0,0 +1,38 @@
+# AWS DynamoDb monitoring
+SkyWalking leverages Amazon Kinesis Data Filehose with [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/) to transfer the metrics into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+1. Amazon CloudWatch fetches metrics from DynamoDB and pushes metrics to SkyWalking OAP Server via Amazon Kinesis data firehose.
+2. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
+
+### Set up
+1. Enable [AWS CloudWatch](https://aws.amazon.com/cn/cloudwatch/)
+2. Create [Amazon Kinesis Data Filehose](https://aws.amazon.com/cn/kinesis/data-firehose/), set source to `Direct PUT`, set destination to `HTTP Endpoint`, and set `HTTP EndPoint url` to `aws-firehose-receiver`'s port (refer to [aws-firehose-receiver](aws-firehose-receiver.md))
+
+ Note that AWS requires that the `HTTP Endpoint url` must be HTTPS and the port needs to be 443, so you can load the certificate in [aws-firehose-receiver](aws-firehose-receiver.md) and set the port to 443. Also, you can use another gateway to accept the request and route it to `aws-filehose-receiver`.
Review Comment:
```suggestion
Note that AWS requires that the `HTTP Endpoint URL` must be through HTTPS listening at 443, therefore need to load the certificate in [aws-firehose-receiver](aws-firehose-receiver.md) and set the port to 443.
Or, you can use another gateway to accept the requests and route them to `aws-filehose-receiver`.
```
I remember @pg-yang mentioned 443 needs to be some OS user group. Should we mention this? @kezhenxu94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] pg-yang commented on pull request #10380: Support monitoring DynamoDB.
Posted by "pg-yang (via GitHub)" <gi...@apache.org>.
pg-yang commented on PR #10380:
URL: https://github.com/apache/skywalking/pull/10380#issuecomment-1426933765
> @pg-yang Is the e2e mock data load for AWS firehose ready to use? This PR should be covered by that style e2e.
I use OTEL mock service to test S3 like EKS. Because if we directly mock firehose service, we have to copy OTEL v0.7.0 into mock service for rewriting some fields such as `time`, and we already have `OtelMetricsConvertorTest` to test conversion logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10380: Support monitoring DynamoDB.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10380:
URL: https://github.com/apache/skywalking/pull/10380#issuecomment-1436936730
This PR context seems broken. It reports codes have been changed on upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org