You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/12/27 02:50:50 UTC
[GitHub] [iotdb] ccliushou opened a new issue, #8618: 关于0.13版本的TsFile 同步方式bug
ccliushou opened a new issue, #8618:
URL: https://github.com/apache/iotdb/issues/8618
0.13版本中，通过 tools\start-sync-client 的方式进行TsFile同步，这种方式未来是否还会继续提供？？1.0中的Pipe、PipeSink方式是未来的支持方式吗？？
0.13版本中，TsFile 同步发现bug，操作方式如下：
1、根据文档描述，配置了 发送端和接收端；
2、执行tools\start-sync-client ，发送端将数据（root.XXX存储组）同步到接收端；
3、接收端，通过delete STORAGE group root.XXX 方式，删除存储组；
4、发送端，再次执行 tools\start-sync-client，无法再次同步 数据到接收端。
发送端日志：
2022-12-27 10:47:43,118 [main] INFO o.a.i.d.s.c.SyncSenderDescriptor:82 - Start to read sync config file D:\apache-iotdb-0.13.0-all-bin\conf\iotdb-sync-client.properties
2022-12-27 10:47:43,154 [main] INFO o.a.i.d.c.IoTDBDescriptor:128 - Start to read config file file:D:/apache-iotdb-0.13.0-all-bin/conf/iotdb-engine.properties
2022-12-27 10:47:43,165 [main] INFO o.a.i.d.c.IoTDBDescriptor:1296 - allocateMemoryForRead = 1125226905
2022-12-27 10:47:43,165 [main] INFO o.a.i.d.c.IoTDBDescriptor:1297 - allocateMemoryForWrite = 1500302540
2022-12-27 10:47:43,166 [main] INFO o.a.i.d.c.IoTDBDescriptor:1298 - allocateMemoryForSchema = 375075635
2022-12-27 10:47:43,168 [main] INFO o.a.i.t.c.c.TSFileDescriptor:123 - try loading iotdb-engine.properties from D:\apache-iotdb-0.13.0-all-bin\conf\iotdb-engine.properties
2022-12-27 10:47:43,173 [main] INFO o.a.i.d.c.IoTDBDescriptor:260 - IoTDB enable memory control: true
2022-12-27 10:47:43,201 [Sync-Client] INFO o.a.i.d.c.IoTDBThreadPoolFactory:174 - new single scheduled thread pool: timedForceMLogThread
2022-12-27 10:47:43,223 [Sync-Client] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: mlog.bin size is 10086971
2022-12-27 10:47:43,850 [Sync-Client] INFO o.a.i.m.c.MetricConfigDescriptor:75 - Start to read config file D:\apache-iotdb-0.13.0-all-bin\conf\iotdb-metric.yml
2022-12-27 10:47:43,873 [Sync-Client] INFO o.a.i.d.c.IoTDBThreadPoolFactory:195 - new scheduled thread pool: sync-client-timer
2022-12-27 10:47:43,955 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:481 - Receiver has received schema successfully.
2022-12-27 10:47:43,956 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:265 - There are 1 data dirs to be synced.
2022-12-27 10:47:43,956 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:268 - Start to sync data in data dir D:\apache-iotdb-0.13.0-all-bin\data\data, the process is 1/1
2022-12-27 10:47:43,957 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.r.SyncSenderLogAnalyzer:89 - log file D:\apache-iotdb-0.13.0-all-bin\data\data\sync-sender\192.168.235.134_9555\sync.log doesn't exist.
2022-12-27 10:47:43,963 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.m.SyncFileManager:94 - Start to get current local files in data folder D:\apache-iotdb-0.13.0-all-bin\data\data
2022-12-27 10:47:43,965 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.m.SyncFileManager:175 - Start to get last local files from last local file info D:\apache-iotdb-0.13.0-all-bin\data\data\sync-sender\192.168.235.134_9555\last_local_files.txt
2022-12-27 10:47:43,976 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:543 - Sync process starts to transfer data of storage group root.rczz, it has 1 virtual storage groups.
2022-12-27 10:47:43,976 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:597 - There has no deleted files to be synced in storage group root.rczz
2022-12-27 10:47:43,977 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:626 - There has no new tsfiles to be synced in storage group root.rczz
2022-12-27 10:47:43,978 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:580 - Sync process finished the task to sync data of storage group root.rczz.
2022-12-27 10:47:43,980 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:287 - Finish to sync data in data dir D:\apache-iotdb-0.13.0-all-bin\data\data, the process is 1/1
2022-12-27 10:47:43,988 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:299 - Sync process has finished.
接收端日志：
2022-12-27 10:47:43,886 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:98 - Start to sync with sender 192.168.235.1_8506306abea44db18da7f89dabf66265
2022-12-27 10:47:43,944 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:292 - Start to load metadata in sync process.
2022-12-27 10:47:43,944 [Sync] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: mlog.bin size is 0
2022-12-27 10:47:43,965 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:155 - Sync process started to receive data of storage group root.rczz
2022-12-27 10:47:43,977 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:336 - Sync process with sender 192.168.235.1_8506306abea44db18da7f89dabf66265 finished.
2022-12-27 10:47:44,003 [pool-25-IoTDB-Load-TsFile-1] INFO o.a.i.d.s.r.l.FileLoader:177 - Sync loading process for 192.168.235.1_8506306abea44db18da7f89dabf66265 has finished.
2022-12-27 10:48:21,873 [pool-3-IoTDB-timedQuerySqlCount-1] INFO o.a.i.d.s.b.QueryFrequencyRecorder:42 - Query count in current 1 minute 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] ccliushou commented on issue #8618: 关于0.13版本的TsFile 同步方式bug
Posted by GitBox <gi...@apache.org>.
ccliushou commented on issue #8618:
URL: https://github.com/apache/iotdb/issues/8618#issuecomment-1374396783
首先感谢您的回答。针对数据同步，我又进行了测试，发现还是存在一定的问题。测试的版本为：0.13.0
测试过程如下：
1、根据文档描述，配置了 发送端和接收端；
2、删除接收端所有的存储组；
3、在发送端，执行tools\start-sync-client ，发送端将数据（root.XXX存储组）同步到接收端；在两端分别确认时间序列的数据量，是一致的，两端的数据量都是：54178；
4、在接收端，通过语句：delete from root.XXX.dev01 where time < 1654085712597 ，删除部分数据，查询接收端的数据量为：44179
5、关闭发送端的同步脚本窗口；
5、删除发送端IOTDB_HOME/data/data下所有sync开头的文件夹
6、删除发送端IOTDB_HOME/data/system/sync文件夹
7、再次运行发送端的同步脚本tools\start-sync-client，确认提示为成功；
8、在接收端，执行：flush
9、在接收端，查询数据量，还是44179。期望得到的数据量应该是：54178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on issue #8618: 关于0.13版本的TsFile 同步方式bug
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on issue #8618:
URL: https://github.com/apache/iotdb/issues/8618#issuecomment-1365679382
- 整文件级别的同步功能会继续支持
- pipe 模式会继续支持，以实现定制化同步。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] ccliushou commented on issue #8618: 关于0.13版本的TsFile 同步方式bug
Posted by GitBox <gi...@apache.org>.
ccliushou commented on issue #8618:
URL: https://github.com/apache/iotdb/issues/8618#issuecomment-1367167273
刚刚又操作了一次。应该确认是bug。操作过程如下：
1、根据文档描述，配置了 发送端和接收端；
2、执行tools\start-sync-client ，发送端将数据（root.XXX存储组）同步到接收端；
3、接收端，通过delete STORAGE group root.XXX 方式，删除存储组；
4、重启接收端；
5、发送端，再次执行 tools\start-sync-client，无法再次同步 数据到接收端。
接收端，第二次接收的关键日志：
2022-12-29 16:46:26,278 [main] INFO o.a.i.d.s.t.ThriftService:111 - IoTDB: start SYNC ServerService...
2022-12-29 16:46:26,282 [main] INFO o.a.i.d.c.IoTDBThreadPoolFactory:232 - new SynchronousQueue thread pool: Sync-Client
2022-12-29 16:46:26,283 [Sync] INFO o.a.i.d.s.t.ThriftServiceThread:241 - The SYNC ServerService service thread begin to run...
2022-12-29 16:46:26,383 [main] INFO o.a.i.d.s.t.ThriftService:135 - IoTDB: start SYNC ServerService successfully, listening on ip 0.0.0.0 port 9555
2022-12-29 16:46:26,384 [main] INFO o.a.i.d.c.IoTDBThreadPoolFactory:68 - new fixed thread pool: UpgradeThread, thread number: 1
2022-12-29 16:46:26,385 [main] INFO o.a.i.d.s.UpgradeSevice:97 - finish counting upgrading files, total num:0
2022-12-29 16:46:26,385 [main] INFO o.a.i.d.s.UpgradeSevice:76 - Waiting for upgrade task pool to shut down
2022-12-29 16:46:26,385 [main] INFO o.a.i.d.s.UpgradeSevice:78 - Upgrade service stopped
2022-12-29 16:46:26,386 [main] INFO o.a.i.d.c.IoTDBThreadPoolFactory:68 - new fixed thread pool: Settle, thread number: 1
2022-12-29 16:46:26,388 [main] INFO o.a.i.d.s.SettleService:150 - Waiting for settle task pool to shut down
2022-12-29 16:46:26,388 [main] INFO o.a.i.d.s.SettleService:152 - Settle service stopped
2022-12-29 16:46:26,391 [main] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: tlog.bin size is 0
2022-12-29 16:46:26,393 [main] INFO o.a.i.d.e.c.ContinuousQueryTaskPoolManager:44 - ContinuousQueryTaskPoolManager is initializing, thread number: 2
2022-12-29 16:46:26,394 [main] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: cqlog.bin size is 0
2022-12-29 16:46:26,395 [main] INFO o.a.i.d.c.IoTDBThreadPoolFactory:174 - new single scheduled thread pool: CQ-Task-Submit-Thread
2022-12-29 16:46:26,396 [main] INFO o.a.i.d.e.c.ContinuousQueryService:136 - Continuous query service started.
2022-12-29 16:46:26,396 [main] INFO o.a.i.db.service.IoTDB:187 - Congratulation, IoTDB is set up successfully. Now, enjoy yourself!
2022-12-29 16:46:26,396 [main] INFO o.a.i.db.service.IoTDB:133 - IoTDB has started.
2022-12-29 16:46:35,063 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:98 - Start to sync with sender 192.168.235.1_0412c25e03e84bcd8dcd633a611b1183
2022-12-29 16:46:35,071 [Sync] INFO o.a.i.d.c.d.DirectoryManager:128 - create folder ./../data/data/sequence failed. Is the folder existed: true
2022-12-29 16:46:35,072 [Sync] INFO o.a.i.d.c.d.DirectoryManager:128 - create folder ./../data/data/unsequence failed. Is the folder existed: true
2022-12-29 16:46:35,140 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:292 - Start to load metadata in sync process.
2022-12-29 16:46:35,140 [Sync] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: mlog.bin size is 0
2022-12-29 16:46:35,162 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:336 - Sync process with sender 192.168.235.1_0412c25e03e84bcd8dcd633a611b1183 finished.
2022-12-29 16:46:35,182 [pool-17-IoTDB-Load-TsFile-1] INFO o.a.i.d.s.r.l.FileLoader:177 - Sync loading process for 192.168.235.1_0412c25e03e84bcd8dcd633a611b1183 has finished.
发送端：
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on issue #8618: 关于0.13版本的TsFile 同步方式bug
Posted by GitBox <gi...@apache.org>.
yschengzi commented on issue #8618:
URL: https://github.com/apache/iotdb/issues/8618#issuecomment-1370589054
不好意思，漏了一些步骤，请按下面的步骤重新让发送端接收端保持一致
1.关闭发送端的同步脚本
2.删除发送端IOTDB_HOME/data/data下所有sync开头的文件夹（一般是sync-sender）
3.删除发送端IOTDB_HOME/data/system/sync文件夹
4.重启接收端
5.重启发送端同步脚本
注意让发送端接收端数据完全保持一致请执行flush命令，如果还未保持一致请等待10分钟，同步脚本每10分钟扫描一次发送端
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on issue #8618: 关于0.13版本的TsFile 同步方式bug
Posted by GitBox <gi...@apache.org>.
yschengzi commented on issue #8618:
URL: https://github.com/apache/iotdb/issues/8618#issuecomment-1367795067
同步工具的机制可以理解为接收端重复执行一次发送端的操作
在你的步骤中，发送端创建了存储组，接收端重复执行会创建一样的存储组
因此如果在接收端删掉对应存储组，接收端是不会再次创建的
这么做的原因：
1.如果需要发送端接收端时刻保持一致，需要时刻监控发送端接收端状态并且比对，这样对系统影响太大
2.时刻保持一致将导致无法在接收端根据需要修改接收端上的数据
如果想让发送端接收端保持一致，可以：
1.找到IOTDB_HOME/data/data目录下，删除所有sync开头的文件夹（发送端）
2.重启sync工具
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] ccliushou commented on issue #8618: 关于0.13版本的TsFile 同步方式bug
Posted by GitBox <gi...@apache.org>.
ccliushou commented on issue #8618:
URL: https://github.com/apache/iotdb/issues/8618#issuecomment-1369282310
感谢您的回答。我按照这个步骤操作了，但，接收端还是没有那数据：
1.找到IOTDB_HOME/data/data目录下，删除所有sync开头的文件夹（发送端）
2.重启sync工具
看接收端的输出的日志，还是感觉有bug：
2023-01-03 08:04:36,565 [main] INFO o.a.i.db.service.IoTDB:133 - IoTDB has started.
2023-01-03 08:04:53,284 [pool-16-IoTDB-RPC-Client-1] INFO o.a.i.d.s.b.ServiceProvider:206 - IoTDB: Login status: Login successfully. User : root, opens Session-1
2023-01-03 08:04:53,406 [pool-16-IoTDB-RPC-Client-1] INFO o.a.i.d.s.t.i.TSServiceImpl:1181 - IoTDB server version: 0.13.3
2023-01-03 08:04:57,244 [pool-16-IoTDB-RPC-Client-1] INFO o.a.i.d.c.IoTDBThreadPoolFactory:68 - new fixed thread pool: Query, thread number: 4
2023-01-03 08:05:33,030 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:98 - Start to sync with sender 192.168.235.1_8506306abea44db18da7f89dabf66265
2023-01-03 08:05:33,046 [Sync] INFO o.a.i.d.c.d.DirectoryManager:128 - create folder ./../data/data/sequence failed. Is the folder existed: true
2023-01-03 08:05:33,047 [Sync] INFO o.a.i.d.c.d.DirectoryManager:128 - create folder ./../data/data/unsequence failed. Is the folder existed: true
2023-01-03 08:05:33,215 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:292 - Start to load metadata in sync process.
2023-01-03 08:05:33,215 [Sync] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: mlog.bin size is 0
2023-01-03 08:05:33,279 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:155 - Sync process started to receive data of storage group root.rczz
2023-01-03 08:05:33,299 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:336 - Sync process with sender 192.168.235.1_8506306abea44db18da7f89dabf66265 finished.
2023-01-03 08:05:33,364 [pool-17-IoTDB-Load-TsFile-1] INFO o.a.i.d.s.r.l.FileLoader:177 - Sync loading process for 192.168.235.1_8506306abea44db18da7f89dabf66265 has finished.
2023-01-03 08:05:36,029 [pool-3-IoTDB-timedQuerySqlCount-1] INFO o.a.i.d.s.b.QueryFrequencyRecorder:42 - Query count in current 1 minute 1
2023-01-03 08:05:54,762 [pool-16-IoTDB-RPC-Client-1] INFO o.a.i.d.s.b.ServiceProvider:232 - Session-1 is closing
2023-01-03 08:06:36,030 [pool-3-IoTDB-timedQuerySqlCount-1] INFO o.a.i.d.s.b.QueryFrequencyRecorder:42 - Query count in current 1 minute 3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org