You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2021/09/02 16:13:35 UTC
[GitHub] [incubator-inlong] LeBW opened a new issue #1522: TubeMQ doesn't receive messages from Dataproxy
LeBW opened a new issue #1522:
URL: https://github.com/apache/incubator-inlong/issues/1522
<!--
Thanks for contributing to InLong!
Please explain your issue precisely, and if possible provide a reproducer
snippet (this helps resolve issues much quicker).
-->
#### Expected behavior
TubeMQ should receive messages from Dataproxy.
#### Actual behavior
TubeMQ doesn't receive messages from Dataproxy
#### Steps to reproduce
1. Deploy InLong with docker according to [here](https://inlong.apache.org/zh-cn/docs/example.html)
2. Adding content in `collect-data/test.log`
Then TubeMQ can't receive the message, like this
```
root@49c0683df4a2:/opt/tubemq-server# curl http://127.0.0.1:8081/broker.htm?method=admin_query_broker_all_store_info&topicName=test
{"result":true,"errCode":0,"errMsg":"Success!","dataSet":[],"totalCnt":0}
```
The logs of dataproxy is:
```
2021-09-02 14:49:00,215 INFO config.ConfigManager$ReloadConfigWorker (ConfigManager.java:checkWithManager:257): start to request http://manager:8083/api/inlong/manager/openapi/dataproxy/getConfig to get config info
2021-09-02 14:49:43,241 INFO config.ConfigManager$ReloadConfigWorker (ConfigManager.java:checkWithManager:257): start to request http://manager:8083/api/inlong/manager/openapi/dataproxy/getConfig to get config info
2021-09-02 14:49:59,591 INFO source.ServerMessageHandler (ServerMessageHandler.java:channelOpen:214): connections=2,maxConnections=5000
2021-09-02 14:50:05,588 INFO source.ServerMessageHandler (ServerMessageHandler.java:messageReceived:534): message received
2021-09-02 14:50:10,616 INFO file.EventQueueBackingStoreFile (EventQueueBackingStoreFile.java:beginCheckpoint:230): Start checkpoint for FLUME_HOME/file/ch-msg5/check/checkpoint, elements to sync = 1
2021-09-02 14:50:10,651 INFO file.EventQueueBackingStoreFile (EventQueueBackingStoreFile.java:checkpoint:255): Updating checkpoint metadata: logWriteOrderID: 1630053997386, queueSize: 1, queueHead: 99999999
2021-09-02 14:50:10,677 INFO file.Log (Log.java:writeCheckpoint:1065): Updated checkpoint for file: FLUME_HOME/file/ch-msg5/data/log-1 position: 461 logWriteOrderID: 1630053997386
2021-09-02 14:50:19,561 INFO config.ConfigManager$ReloadConfigWorker (ConfigManager.java:checkWithManager:257): start to request http://manager:8083/api/inlong/manager/openapi/dataproxy/getConfig to get config info
2021-09-02 14:50:55,119 INFO config.ConfigManager$ReloadConfigWorker (ConfigManager.java:checkWithManager:257): start to request http://manager:8083/api/inlong/manager/openapi/dataproxy/getConfig to get config info
```
The logs of agent is:
```
2021-09-02 14:49:59,593 INFO [task-running-thread-3] (reader.TextFileReader@init:140): file name for task is /data/collect-data/test.log, md5 is bc5309bee3c74
2021-09-02 14:50:00,593 INFO [TimeoutScanThread] (threads.TimeoutScanThread@run:180): TimeoutScanThread thread=116's currentBufferSize = 0
2021-09-02 14:50:01,751 INFO [SendHBThread] (network.ClientMgr@removeBadRealClient:664): this host dataproxy is active! so keep it
2021-09-02 14:50:01,754 INFO [SendHBThread] (network.ClientMgr@replaceBadConnectionHB:727): hasn't bad host! so keep it
2021-09-02 14:50:05,587 INFO [ProxySink-running-thread-1] (sinks.ProxySink@lambda$null$1:130): send bid b_test with message size 10, the job id is job_1630594
2021-09-02 14:50:12,837 INFO [TimeoutScanThread] (threads.TimeoutScanThread@run:180): TimeoutScanThread thread=91's currentBufferSize = 0
2021-09-02 14:50:13,329 INFO [SendHBThread] (network.ClientMgr@removeBadRealClient:664): this host dataproxy is active! so keep it
2021-09-02 14:50:13,330 INFO [SendHBThread] (network.ClientMgr@replaceBadConnectionHB:727): hasn't bad host! so keep it
2021-09-02 14:50:13,593 INFO [TimeoutScanThread] (threads.TimeoutScanThread@run:180): TimeoutScanThread thread=102's currentBufferSize = 0
2021-09-02 14:50:16,593 INFO [SendHBThread] (network.ClientMgr@removeBadRealClient:664): this host dataproxy is active! so keep it
2021-09-02 14:50:16,593 INFO [SendHBThread] (network.ClientMgr@replaceBadConnectionHB:727): hasn't bad host! so keep it
2021-09-02 14:50:20,596 INFO [TimeoutScanThread] (threads.TimeoutScanThread@run:180): TimeoutScanThread thread=116's currentBufferSize = 0
2021-09-02 14:50:20,754 INFO [SendHBThread] (network.ClientMgr@removeBadRealClient:664): this host dataproxy is active! so keep it
2021-09-02 14:50:20,754 INFO [SendHBThread] (network.ClientMgr@replaceBadConnectionHB:727): hasn't bad host! so keep it
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] LeBW closed issue #1522: TubeMQ doesn't receive messages from Dataproxy
Posted by GitBox <gi...@apache.org>.
LeBW closed issue #1522:
URL: https://github.com/apache/incubator-inlong/issues/1522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gosonzhang commented on issue #1522: TubeMQ doesn't receive messages from Dataproxy
Posted by GitBox <gi...@apache.org>.
gosonzhang commented on issue #1522:
URL: https://github.com/apache/incubator-inlong/issues/1522#issuecomment-912927983
@LeBW In the bin directory of tubemq, use tubemq-producer-test.sh to send data to the Topic test corresponding to b_test. Can the data be successfully produced?
If data not successfully produced, post the relevant error message; if you can, it means that there is an error when dataproxy reports data to tubemq, send and let me see the topic.properities file in the conf directory of dataproxy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] LeBW edited a comment on issue #1522: TubeMQ doesn't receive messages from Dataproxy
Posted by GitBox <gi...@apache.org>.
LeBW edited a comment on issue #1522:
URL: https://github.com/apache/incubator-inlong/issues/1522#issuecomment-912977549
> @LeBW In the bin directory of tubemq, use tubemq-producer-test.sh to send data to the Topic test corresponding to b_test. Can the data be successfully produced?
>
> If data not successfully produced, post the relevant error message; if you can, it means that there is an error when dataproxy reports data to tubemq, send and let me see the topic.properities file in the conf directory of dataproxy.
I try it to send data to the topic, and it's ok. Here is the log.
```
root@49c0683df4a2:/opt/tubemq-server/bin# ./tubemq-producer-test.sh --master-servers localhost:8715 --topicName test
[2021-09-04 13:48:53,152] INFO Created Session Factory, the config is: {"masterInfo":["localhost:8715"],"rpcReadTimeoutMs":3000,"rpcConnProcessorCnt":17,"rpcNettyWorkMemorySize":10485760,"rpcRspCallBackThreadCnt":3,"nettyWriteBufferHighWaterMark":52428800,"nettyWriteBufferLowWaterMark":5242880,"maxRegisterRetryTimes":5,"regFailWaitPeriodMs":1000,"maxHeartBeatRetryTimes":5,"heartbeatPeriodMs":13000,"heartbeatPeriodAfterFail":60000,"linkStatsDurationMs":60000,"linkStatsForbiddenDurationMs":1800000,"linkStatsMaxAllowedFailTimes":5,"linkStatsMaxForbiddenRate":0.3,"maxSentForbiddenRate":0.15,"maxForbiddenCheckDuration":30000,"sessionStatisticCheckDuration":1000,"sessionWarnForbiddenRate":0.2,"sessionWarnDelayedMsgCount":300000,"linkMaxAllowedDelayedMsgCount":50000,"sessionMaxAllowedDelayedMsgCount":500000,"unAvailableFbdDurationMs":50000,"enableUserAuthentic":false,"usrName":"","usrPassWord":"","localAddress":"172.27.0.3","TLSConfig":{"tlsEnable":false,"tlsPort":8124,"tlsTrustStorePath
":"","tlsTrustStorePassword":"","tlsTwoWayAuthEnable":false,"tlsKeyStorePath":"","tlsKeyStorePassword":""}} (org.apache.inlong.tubemq.client.factory.TubeBaseSessionFactory)
[2021-09-04 13:48:53,200] INFO [Producer] Producer status from ready to running! (org.apache.inlong.tubemq.client.producer.ProducerManager)
[2021-09-04 13:48:53,202] INFO [Publish begin 2] publish topicSet [test], address = org.apache.inlong.tubemq.client.producer.ProducerManager@6c3708b3 (org.apache.inlong.tubemq.client.producer.ProducerManager)
[2021-09-04 13:48:53,208] INFO [Publish begin 2] already get meta info, topicSet: [test], waste time 6 Ms (org.apache.inlong.tubemq.client.producer.ProducerManager)
Required send count VS sent message count = -2 : 5000 (5000:0:0)
Required send count VS sent message count = -2 : 10000 (10000:0:0)
```
And the `topic.properties` is:
```
root@0a950ba5d730:/opt/inlong-dataproxy# cat conf/topics.properties
b_test=test
b_test_1=test_1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] LeBW commented on issue #1522: TubeMQ doesn't receive messages from Dataproxy
Posted by GitBox <gi...@apache.org>.
LeBW commented on issue #1522:
URL: https://github.com/apache/incubator-inlong/issues/1522#issuecomment-912977549
> @LeBW In the bin directory of tubemq, use tubemq-producer-test.sh to send data to the Topic test corresponding to b_test. Can the data be successfully produced?
>
> If data not successfully produced, post the relevant error message; if you can, it means that there is an error when dataproxy reports data to tubemq, send and let me see the topic.properities file in the conf directory of dataproxy.
I try it to send data to the topic, and it's ok. Here is the log.
```
root@49c0683df4a2:/opt/tubemq-server/bin# ./tubemq-producer-test.sh --master-servers localhost:8715 --topicName test
[2021-09-04 13:48:53,152] INFO Created Session Factory, the config is: {"masterInfo":["localhost:8715"],"rpcReadTimeoutMs":3000,"rpcConnProcessorCnt":17,"rpcNettyWorkMemorySize":10485760,"rpcRspCallBackThreadCnt":3,"nettyWriteBufferHighWaterMark":52428800,"nettyWriteBufferLowWaterMark":5242880,"maxRegisterRetryTimes":5,"regFailWaitPeriodMs":1000,"maxHeartBeatRetryTimes":5,"heartbeatPeriodMs":13000,"heartbeatPeriodAfterFail":60000,"linkStatsDurationMs":60000,"linkStatsForbiddenDurationMs":1800000,"linkStatsMaxAllowedFailTimes":5,"linkStatsMaxForbiddenRate":0.3,"maxSentForbiddenRate":0.15,"maxForbiddenCheckDuration":30000,"sessionStatisticCheckDuration":1000,"sessionWarnForbiddenRate":0.2,"sessionWarnDelayedMsgCount":300000,"linkMaxAllowedDelayedMsgCount":50000,"sessionMaxAllowedDelayedMsgCount":500000,"unAvailableFbdDurationMs":50000,"enableUserAuthentic":false,"usrName":"","usrPassWord":"","localAddress":"172.27.0.3","TLSConfig":{"tlsEnable":false,"tlsPort":8124,"tlsTrustStorePath
":"","tlsTrustStorePassword":"","tlsTwoWayAuthEnable":false,"tlsKeyStorePath":"","tlsKeyStorePassword":""}} (org.apache.inlong.tubemq.client.factory.TubeBaseSessionFactory)
[2021-09-04 13:48:53,200] INFO [Producer] Producer status from ready to running! (org.apache.inlong.tubemq.client.producer.ProducerManager)
[2021-09-04 13:48:53,202] INFO [Publish begin 2] publish topicSet [test], address = org.apache.inlong.tubemq.client.producer.ProducerManager@6c3708b3 (org.apache.inlong.tubemq.client.producer.ProducerManager)
[2021-09-04 13:48:53,208] INFO [Publish begin 2] already get meta info, topicSet: [test], waste time 6 Ms (org.apache.inlong.tubemq.client.producer.ProducerManager)
Required send count VS sent message count = -2 : 5000 (5000:0:0)
Required send count VS sent message count = -2 : 10000 (10000:0:0)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org