You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "ChrisssWong (via GitHub)" <gi...@apache.org> on 2023/03/21 13:36:53 UTC
[GitHub] [skywalking] ChrisssWong opened a new issue, #10571: [Bug] The difference in data structure of log storage between MySQL and ES causes the issue that the TraceId cannot be displayed in the UI log function of Skywalking.
ChrisssWong opened a new issue, #10571:
URL: https://github.com/apache/skywalking/issues/10571
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
OAP server (apache/skywalking)
### What happened
Skywalking can store application logs, which are stored in memory by default but can be modified to be stored in MySQL or Elasticsearch.
When the storage endpoint is set to MySQL, the TraceId can be displayed in the UI log function of Skywalking.
However, if the storage endpoint is set to ES, the TraceId cannot be displayed as shown below:
After reviewing the data structures of Skywalking in MySQL and ES, we found that in MySQL, it looks like this:
```sql
CREATE TABLE log
(
id varchar(512) NOT NULL
PRIMARY KEY,
unique_id varchar(200) NULL,
service_id varchar(200) NULL,
service_instance_id varchar(200) NULL,
endpoint_id varchar(200) NULL,
-- It‘s stored trace_id
trace_id varchar(150) NULL,
trace_segment_id varchar(150) NULL,
span_id int NULL,
content_type int NULL,
content mediumtext NULL,
timestamp bigint NULL,
tags_raw_data mediumtext NULL,
tags_0 varchar(200) NULL,
tags_1 varchar(200) NULL,
tags_2 varchar(200) NULL,
tags_3 varchar(200) NULL,
tags_4 varchar(200) NULL,
tags_5 varchar(200) NULL,
tags_6 varchar(200) NULL,
tags_7 varchar(200) NULL,
tags_8 varchar(200) NULL,
tags_9 varchar(200) NULL,
tags_10 varchar(200) NULL,
tags_11 varchar(200) NULL,
tags_12 varchar(200) NULL,
tags_13 varchar(200) NULL,
tags_14 varchar(200) NULL,
tags_15 varchar(200) NULL,
tags_16 varchar(200) NULL,
tags_17 varchar(200) NULL,
tags_18 varchar(200) NULL,
tags_19 varchar(200) NULL,
time_bucket bigint NULL
);
```
But in ES, it is stored like this:
```
{
"_index": "skywalking_log-20230321",
"_type": "_doc",
"_id": "bbdd208df28c4521aa436dbfd9a26a3a",
"_version": 1,
"_seq_no": 132,
"_primary_term": 1,
"found": true,
"_source": {
"unique_id": "bbdd208df28c4521aa436dbfd9a26a3a",
"span_id": 0,
"service_instance_id": "Z29sZGVuLXNlcnZpY2UtZmFjZQ==.1_MWJlOGE0N2QxOTM1NDM1NGEzMzJmMGNkM2ViYzQ4MzZAMTcyLjE3LjAuMQ==",
"content": "2023-03-21 16:31:46.935 [TID:ee137820a72441a4a119ede353229b10.92.16793875068960035] INFO",
"tags": [
"level=INFO"
],
"content_type": 1,
"tags_raw_data": "Cg0KBWxldmVsEgRJTkZPCi4KBmxvZ2dlchIkY29tLnprc2ouY29tbW9uLmxvZy5hc3BlY3QuTG9nQXNwZWN0Ch4KBnRocmVhZBIUaHR0cC1uaW8tOTIxMS1leGVjLTI=",
"service_id": "Z29sZGVuLXNlcnZpY2UtZmFjZQ==.1",
"time_bucket": 20230321163146,
"timestamp": 1679387506935
}
}
```
In other words, the index structure of the logs in ES does not contain the TraceId field.
To enable the TraceId field to be added to the index structure and displayed in the UI interface of Skywalking for easy searching, certain settings need to be modified.
Why isn't the TraceId field set in the sw_log index of ES? Are there any other ways to set it?
### What you expected to happen
I I want to see TraceId in the UI log function of Skywalking.
Thanks you!
### How to reproduce
Reproduction Steps:
1.Use the version of skywalking-api is 8.9.0，apache-skywalking-apm-8.9.0.tar.gz
2.First use MySQL as the storage endpoint, and later switch to ES as the storage endpoint.
3.View logs in Skywalking Web-UI interface, TraceID don't display.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10571: [Bug] The difference in data structure of log storage between MySQL and ES causes the issue that the TraceId cannot be displayed in the UI log function of Skywalking.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10571:
URL: https://github.com/apache/skywalking/issues/10571#issuecomment-1477877643
I could only say
1. 8.9.0 is too old. 8.x is EOL for a long time.
2. Endpoint ID is both in MySQL and ElasticSearch.
3. What you read is the data of the row, but not the data structure. Check the ElasticSearch template.
Notice, this is not a bug report. Discussion panel is for this kind of QA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #10571: [Bug] The difference in data structure of log storage between MySQL and ES causes the issue that the TraceId cannot be displayed in the UI log function of Skywalking.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #10571: [Bug] The difference in data structure of log storage between MySQL and ES causes the issue that the TraceId cannot be displayed in the UI log function of Skywalking.
URL: https://github.com/apache/skywalking/issues/10571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] ChrisssWong commented on issue #10571: [Bug] The difference in data structure of log storage between MySQL and ES causes the issue that the TraceId cannot be displayed in the UI log function of Skywalking.
Posted by "ChrisssWong (via GitHub)" <gi...@apache.org>.
ChrisssWong commented on issue #10571:
URL: https://github.com/apache/skywalking/issues/10571#issuecomment-1477909024
Ok, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org