You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/02/25 06:48:17 UTC
[GitHub] [skywalking] lvcarlei opened a new issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
lvcarlei opened a new issue #6441:
URL: https://github.com/apache/skywalking/issues/6441
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
___
### Bug
- Which version of SkyWalking, OS, and JRE?
SkyWalking version 8.3.0
Alpine Linux v3.8
java version "1.8.0_192"
- What happened?
ERROR 2021-02-25 14:41:03:221 AnalysisSaveAsync-235 InstMethodsInter : class[class org.apache.hadoop.hbase.client.HTable] before method[getScanner] intercept failure
java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:163)
at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:143)
at org.apache.skywalking.apm.agent.core.context.ContextManager.createExitSpan(ContextManager.java:112)
at org.apache.skywalking.apm.plugin.hbase.HTableInterceptor.beforeMethod(HTableInterceptor.java:72)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java)
at executor.aicoin.utils.HbaseUtil.scanLast(HbaseUtil.java:226)
at executor.aicoin.service.AnalysisService.parseDataT(AnalysisService.java:57)
at executor.aicoin.service.AnalysisService.parseData(AnalysisService.java:682)
at executor.aicoin.service.AnalysisService$$FastClassBySpringCGLIB$$c21a2f26.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)
at org.apache.skywalking.apm.plugin.spring.async.SWCallable.call(SWCallable.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] nisiyong commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
nisiyong commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801076665
> 1. the plugin should be renamed to `hbase-1.x-2.x-plugin`
I prefer the first one since the`HTable` public methods still have no change, I think HBase Client Core API won't change often.
And if we need to change the plugin name, why not just name it `hbase-plugin` directly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-785674094
I don't know actually. Look like. Maybe version mismatch, usage mismatch. Same as previous comments, we need you to check locally with our existing plugin codes and test cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] nisiyong commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
nisiyong commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801083893
Maybe `hbase-1.x-2.x-plugin` will be more clearly, like `tomcat-7.x-8.x-plugin`.
After renaming the plugin, just use 2.x dependency in the pom.xml? No 1.x dependency anymore, but the plugin still suits 1.x.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-785662971
This should be a plugin bug, we need your help to locate this issue, and send a pull request. The key is, why we don't get the real `peer`(hbase server IPs).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] nisiyong commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
nisiyong commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801137798
Okay, I will do it tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] nisiyong edited a comment on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
nisiyong edited a comment on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801011967
> I don't know actually. Look like. Maybe version mismatch, usage mismatch. Same as previous comments, we need you to check locally with our existing plugin codes and test cases.
I met this exception on our production agent log, too. Today I debug the plugin and figure it out.
The `hbase-1.x-plugin` only suits the HBase client 1.x, cause after HBase 2.x, `HTable` has changed its constructor function. So when use hbase-client-2.x will meet this exception.
## Environment
- **SkyWalking Agent**: 8.2.0
- **HBase Client**: 2.1.0 (1.x works fine)
## Agent Log
```
ERROR 2021-03-17 18:31:53:844 http-nio-8080-exec-4 InstMethodsInter : class[class org.apache.hadoop.hbase.client.HTable] before method[delete] intercept failure
java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:163)
at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:143)
at org.apache.skywalking.apm.agent.core.context.ContextManager.createExitSpan(ContextManager.java:126)
at org.apache.skywalking.apm.plugin.hbase.HTableInterceptor.beforeMethod(HTableInterceptor.java:72)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
at org.apache.hadoop.hbase.client.HTable.delete(HTable.java)
at org.apache.skywalking.apm.testcase.hbase.controller.HBaseController.hbaseCase(HBaseController.java:91)
```
## 1.x HTable contructor (from version 1.0.0 to 1.6.0)
source code: [HTable.java](https://github.com/apache/hbase/blob/rel/1.6.0/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java)
```java
@InterfaceAudience.Private
public HTable(TableName tableName, final ClusterConnection connection,
final ConnectionConfiguration tableConfig,
final RpcRetryingCallerFactory rpcCallerFactory,
final RpcControllerFactory rpcControllerFactory,
final ExecutorService pool) throws IOException {
...
}
```
## 2.x HTable contructor (from version 2.0.0 to 2.1.9)
source code: [HTable.java](https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java)
```java
@InterfaceAudience.Private
protected HTable(final ClusterConnection connection,
final TableBuilderBase builder,
final RpcRetryingCallerFactory rpcCallerFactory,
final RpcControllerFactory rpcControllerFactory,
final ExecutorService pool) {
...
}
```
## 2.x HTable contructor (after version 2.2.0 include 2.2.0)
source code: [HTable.java](https://github.com/apache/hbase/blob/rel/2.4.1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java)
```java
@InterfaceAudience.Private
protected HTable(final ConnectionImplementation connection,
final TableBuilderBase builder,
final RpcRetryingCallerFactory rpcCallerFactory,
final RpcControllerFactory rpcControllerFactory,
final ExecutorService pool) {
...
}
```
The above difference cause `org.apache.skywalking.apm.plugin.hbase.HTableInterceptor#onConstruct` could not set the zk address in `EnhancedInstance`, and when invoke `ContextManager.createExitSpan` will pass the empty peer.
```java
public void onConstruct(EnhancedInstance objInst, Object[] allArguments) throws Throwable {
Configuration connection = ((ClusterConnection) allArguments[1]).getConfiguration();
Field field = connection.getClass().getDeclaredField("overlay");
field.setAccessible(true);
Properties properties = (Properties) field.get(connection);
for (Map.Entry entry : properties.entrySet()) {
if ("hbase.zookeeper.quorum".equals(entry.getKey())) {
objInst.setSkyWalkingDynamicField(entry.getValue().toString());
}
}
}
```
I have fixed the plugin now, but I need some help. The current plugin named `hbase-1.x.-plugin`, I want to add a new plugin `hbase-2.x-plugin`, but I was confused which plugin will be work at runtime because both of them enhance the same class `org.apache.hadoop.hbase.client.HTable`.
I want to know more about how does the plugin work on multi-version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #6441:
URL: https://github.com/apache/skywalking/issues/6441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801035130
There are 2 ways
1. Don't focus on the version match mechanism. You could intercept the new construct directly. But as you are sharing the same target class, you need to change the old `HTableInstrumentation` and add a `ConstructorInterceptPoint`. Then, the plugin should be renamed to `hbase-1.x-2.x-plugin`.(documents should be updated too)
2. If you insist to use version match mechanism to create a new plugin called `hbase-2.x-plugin`, then, read `Set up witnessClasses and/or witnessMethods if the instrumentation should be activated in specific versions.` section in the plug dev doc, https://skywalking.apache.org/docs/main/v8.4.0/en/guides/java-plugin-development-guide/. You need to find special class or method to identify the version in the runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801125815
> Maybe `hbase-1.x-2.x-plugin` will be more clearly, like `tomcat-7.x-8.x-plugin`.
> After renaming the plugin, just use 2.x dependency in the pom.xml? No 1.x dependency anymore, but the plugin still suits 1.x.
Compiling dependency doesn't matter. Remember to add 2.0 into plugin tests(e2e). We need them to verify.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] nisiyong commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
nisiyong commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801011967
> I don't know actually. Look like. Maybe version mismatch, usage mismatch. Same as previous comments, we need you to check locally with our existing plugin codes and test cases.
I met this exception on our production agent log, too. Today I debug the plugin and figure it out.
The `hbase-1.x-plugin` only suits the HBase client 1.x, cause after HBase 2.x, `HTable` has changed its constructor function. So when use hbase-client-2.x will meet this exception.
## Environment
- **SkyWalking Agent**: 8.2.0
- **HBase Client**: 2.1.0 (1.x works fine)
## Agent Log
```
ERROR 2021-03-17 18:31:53:844 http-nio-8080-exec-4 InstMethodsInter : class[class org.apache.hadoop.hbase.client.HTable] before method[delete] intercept failure
java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:163)
at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:143)
at org.apache.skywalking.apm.agent.core.context.ContextManager.createExitSpan(ContextManager.java:126)
at org.apache.skywalking.apm.plugin.hbase.HTableInterceptor.beforeMethod(HTableInterceptor.java:72)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
at org.apache.hadoop.hbase.client.HTable.delete(HTable.java)
at org.apache.skywalking.apm.testcase.hbase.controller.HBaseController.hbaseCase(HBaseController.java:91)
```
## 1.x HTable contructor (version 1.6.0, the latest 1.x)
source code: [HTable.java](https://github.com/apache/hbase/blob/rel/1.6.0/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java)
```java
@InterfaceAudience.Private
public HTable(TableName tableName, final ClusterConnection connection,
final ConnectionConfiguration tableConfig,
final RpcRetryingCallerFactory rpcCallerFactory,
final RpcControllerFactory rpcControllerFactory,
final ExecutorService pool) throws IOException {
...
}
```
## 2.x HTable contructor (version 2.0.0, the earliest 2.x)
source code: [HTable.java](https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java)
```java
@InterfaceAudience.Private
protected HTable(final ClusterConnection connection,
final TableBuilderBase builder,
final RpcRetryingCallerFactory rpcCallerFactory,
final RpcControllerFactory rpcControllerFactory,
final ExecutorService pool) {
...
}
```
The above difference cause `org.apache.skywalking.apm.plugin.hbase.HTableInterceptor#onConstruct` could not set the zk address in `EnhancedInstance`, and when invoke `ContextManager.createExitSpan` will pass the empty peer.
```java
public void onConstruct(EnhancedInstance objInst, Object[] allArguments) throws Throwable {
Configuration connection = ((ClusterConnection) allArguments[1]).getConfiguration();
Field field = connection.getClass().getDeclaredField("overlay");
field.setAccessible(true);
Properties properties = (Properties) field.get(connection);
for (Map.Entry entry : properties.entrySet()) {
if ("hbase.zookeeper.quorum".equals(entry.getKey())) {
objInst.setSkyWalkingDynamicField(entry.getValue().toString());
}
}
}
```
I have fixed the plugin now, but I need some help. The current plugin named `hbase-1.x.-plugin`, I want to add a new plugin `hbase-2.x-plugin`, but I was confused which plugin will be work at runtime because both of them enhance the same class `org.apache.hadoop.hbase.client.HTable`.
I want to know more about how does the plugin work on multi-version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-801127433
@nisiyong You actually need another issue to describe the issue. This issue actually is related to ElasticSearch client plugin somehow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] lvcarlei commented on issue #6441: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
Posted by GitBox <gi...@apache.org>.
lvcarlei commented on issue #6441:
URL: https://github.com/apache/skywalking/issues/6441#issuecomment-785672616
Is this also a plugin bug?
class[class org.elasticsearch.client.RestHighLevelClient] before method[search] intercept failure
java.lang.NullPointerException
at org.apache.skywalking.apm.plugin.elasticsearch.v6.interceptor.RestHighLevelClientSearchMethodsInterceptor.beforeMethod(RestHighLevelClientSearchMethodsInterceptor.java:45)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java)
at executor.aicoin.utils.hioes.EsUtil.getKlineRecordBytime(EsUtil.java:67)
at executor.aicoin.service.KlineAnalysisService.getFormulaKlineResultByTime(KlineAnalysisService.java:291)
at executor.aicoin.service.KlineAnalysisService.calKlineT(KlineAnalysisService.java:262)
at executor.aicoin.service.KlineAnalysisService.calKline(KlineAnalysisService.java:389)
at executor.aicoin.service.KlineAnalysisService$$FastClassBySpringCGLIB$$376ed08d.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)
at org.apache.skywalking.apm.plugin.spring.async.SWCallable.call(SWCallable.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org