You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/06/10 06:23:20 UTC
[GitHub] [skywalking] momo0313 opened a new issue #4893: Why oap analyzer ignore numbers?
momo0313 opened a new issue #4893:
URL: https://github.com/apache/skywalking/issues/4893
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [ ] Bug
- [x] Requirement
- [ ] Feature or performance improvement
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
Skywalking version 6.6 with Elasticsearch 6.8 for storage.
Firstly, on the "trace" page of skywaking UI，try to search a endpoint name like "/xnv/abc_def/QW030384" , and the results are not expected, something like:
```
/xnv/abc_def/QW051234
/xnv/abc_def/QW051235
...
```
Secondly, try to analyzer the above endpoint name with the default oap_analyzer using kibana devtool:
```bash
GET sw_segment-20200610/_analyze
{
"text": "/xnv/abc_def/QW030384",
"analyzer" : "oap_analyzer"
}
```
the results are:
```
{
"tokens" : [
{
"token" : "xnv",
"start_offset" : 1,
"end_offset" : 4,
"type" : "word",
"position" : 0
},
{
"token" : "abc",
"start_offset" : 5,
"end_offset" : 8,
"type" : "word",
"position" : 1
},
{
"token" : "def",
"start_offset" : 9,
"end_offset" : 12,
"type" : "word",
"position" : 2
},
{
"token" : "qw",
"start_offset" : 13,
"end_offset" : 15,
"type" : "word",
"position" : 3
}
]
}
```
The above results say that, with the oap_analyzer, the endpoint name "/xnv/abc_def/QW030384" is analyzed to some tokens: "xnv" "abc" "def" "qw",
**the numbers in the endpoint name are ignored.**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-642507208
Seems make sense to me. Does standard analyzer support `/`, `-`, `.` and white space? I think these are common in the operation name. Or do we need to customize stopwords?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-642562648
Dot is required. Think about the package name, class name. Such as grpc or dubbo case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-659110347
https://github.com/apache/skywalking-query-protocol/blob/master/trace.graphqls#L39-L40
This thing has been supported at the protocol level and OAP backend codes. If you want to enhance this, this is only some kind of frontend work.
Could we close this discussion, as it is not related to the topic anymore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-647134359
@momo0313 As 8.0.1 is out, we could continue this. Is this analyzer support `_`? I think once we have `/`, `-`, `.` and `_`, it should be perfect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 commented on issue #4893: Why oap analyzer ignore numbers?
Posted by GitBox <gi...@apache.org>.
momo0313 commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-642448872
> I think this is the rule we are using `setting.put("analysis.analyzer.oap_analyzer.type", "stop")` in `StorageEsInstaller#createSetting`. I didn't write these codes, could you do a little more research about the difference? Then we could have a more reasonable choice.
We can refer to [analysis-stop-analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stop-analyzer.html) and [analysis-standard-analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html) to get the difference of the "stop" and the "standard" type analyzer.
In short the "standard" type analyzer will not ignore the numbers, and in this case we can query by numbers in the endpoint name.
It's indeed a matter of choice, but I tend to choose the "standard" analyzer. We usually query with a complete endpoint name, it is confused that the results are not matching the numbers in the querying string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap analyzer ignore numbers?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-642059697
I think this is the rule we are using `setting.put("analysis.analyzer.oap_analyzer.type", "stop")` in `StorageEsInstaller#createSetting`. I didn't write these codes, could you do a little more research about the difference? Then we could have a more reasonable choice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap analyzer ignore numbers?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-641767561
Trace could be queried through the endpoint id, that is an accurate way. https://github.com/apache/skywalking-query-protocol/blob/master/trace.graphqls#L39
But we haven't provided it in the UI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap analyzer ignore numbers?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-641766917
What do you suggest? If we do the exact match, most likely people are hard to find the endpoint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
momo0313 commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-642552416
The standard analyzer supports `/` and `-`, while `.` and `_` are not support. There is an example
```
GET _analyze
{
"text": "/xyz/a/tmo,pp-abd_def/QA0203.940"
}
```
The above sentence would produce the following terms:
`xyz`,`a`, `tmo`, `pp`, `abc_def`, `qa0203.940`
Although, in this case, we have to query some strings like "abc_def" or "qa0203.940" to get the right results, it's better than the case we can not query it by numbers. Actually, we usually copy and paste the complete endpoint and query it, so the `_` and `.` may not be a problem.
Do not think we need to customize any stopwords, any words like `a`, `the`.. are significant in the endpoint, service name, instance name and so on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 edited a comment on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
momo0313 edited a comment on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-658591759
With some more research, the "pattern analyzer" may NOT be a good choice. As described in [pattern analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-analyzer.html), it uses Java Regular Expressions. We have to think about the performance, further more, a badly written regular expression could run very slowly or even throw a StackOverflowError and cause the node it is running on to exit suddenly.
Actually, I don't have good idea about this issue now. I'm inclined to support the both "fuzzy" and "accurate" type query. If the user have no background knowledges of the analyzer of Elasticsearch, they can choose "accurate" type, input the complete endpoint to query and then get the accurate result, otherwise, they can choose the fuzzy type as now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 commented on issue #4893: Why oap analyzer ignore numbers?
Posted by GitBox <gi...@apache.org>.
momo0313 commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-641804278
> What do you suggest? If we do the exact match, most likely people are hard to find the endpoint.
In my opinion, the oap_analyzer should not ignore the numbers in the endpoint name because it is allowed.
Just like the Elasticsearch default "standard" analyzer:
"text": "/xnv/abc_def/QW030384" will be analyzed to tokens: xnv abe_def qw030384.
In this case, we can query the endpoint with xnv OR abe_dev OR QW030384
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 edited a comment on issue #4893: Why oap analyzer ignore numbers?
Posted by GitBox <gi...@apache.org>.
momo0313 edited a comment on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-641804278
> What do you suggest? If we do the exact match, most likely people are hard to find the endpoint.
In my opinion, the oap_analyzer should not ignore the numbers in the endpoint name because it is allowed.
Just like the Elasticsearch default "standard" analyzer:
"text": "/xnv/abc_def/QW030384" will be analyzed to tokens: xnv abe_def qw030384.
In this case, we can query the endpoint with a combination of some analyzed tokens in right order, like /xnv OR /xnv/abc_dev OR QW030384 OR /xnv/abc_dev/QW030384 and so on
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
momo0313 commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-643040440
AFAIK, The [pattern analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-analyzer.html) may be a choice, and it support `/` `-` '.' and white space. The default regex of pattern analyzer is `\W`, same as `[^A-Za-z0-9_]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-654308375
@momo0313 Any update?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 commented on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
momo0313 commented on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-658591759
With some more research, the "pattern analyzer" may NOT be a good choice. As described in [pattern analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-analyzer.html), it uses Java Regular Expressions. We have to think about the performance, further more, a badly written regular expression could run very slowly or even throw a StackOverflowError and cause the node it is running on to exit suddenly.
Actually, I have not idea about this issue now, and I change my point of view to: The user should not have to know about the background knowledges of the analyzer of Elasticsearch, they just input the complete endpoint to query and then get the accurate result. Or we support the both by adding two query types, "fuzzy" or "accurate"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #4893:
URL: https://github.com/apache/skywalking/issues/4893
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] momo0313 edited a comment on issue #4893: Why oap use the "stop" analyzer instead of "standand" one for the endpoint name field in elasticsearch storage?
Posted by GitBox <gi...@apache.org>.
momo0313 edited a comment on issue #4893:
URL: https://github.com/apache/skywalking/issues/4893#issuecomment-658591759
With some more research, the "pattern analyzer" may NOT be a good choice. As described in [pattern analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-analyzer.html), it uses Java Regular Expressions. We have to think about the performance, further more, a badly written regular expression input by user could run very slowly or even throw a StackOverflowError and cause the node it is running on to exit suddenly.
Actually, I don't have good idea about this issue now. I'm inclined to support the both "fuzzy" and "accurate" type query. If the user have no background knowledges of the analyzer of Elasticsearch, they can choose "accurate" type, input the complete endpoint to query and then get the accurate result, otherwise, they can choose the fuzzy type as now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org