You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "SunnySaman (via GitHub)" <gi...@apache.org> on 2023/03/10 07:27:10 UTC
[GitHub] [iotdb] SunnySaman opened a new issue, #9284: [Bug] StorageEngine failed to recover
SunnySaman opened a new issue, #9284:
URL: https://github.com/apache/iotdb/issues/9284
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
- IoTDB：`0.13.3`
- OS：`CentOS Linux release 7.5.1804 (Core)`
- JDK:
```
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9
```
### Describe the bug and provide the minimal reproduce step
生产环境有一个节点不知道什么原因有可能是断电或物理机重启导致IoTDB在重启后，无法写入数据，查询部分正常。
执行Insert语句返回的错误信息：
```
Msg: 304: Path *** does not exist
```
重新启动iotdb服务之后：在日志中发现了以下信息：
- log_error.log
```
2023-03-10 14:37:05,903 [pool-12-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.t.r.TsFileSequenceReader:280 - Something error happened while reading file metadata of file /var/lib/iotdb/data/data/sequence/root.fz4.pv.s_9/0/0/1675621909201-1-0-32.tsfile
2023-03-10 14:37:05,949 [Thread-1] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-1-28
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:402)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:400)
... 1 common frames omitted
Caused by: java.nio.BufferUnderflowException: null
at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:647)
at java.base/java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:439)
at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readLong(ReadWriteIOUtils.java:546)
at org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode.deserializeFrom(MetadataIndexNode.java:105)
at org.apache.iotdb.tsfile.file.metadata.TsFileMetadata.deserializeFrom(TsFileMetadata.java:55)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readFileMetadata(TsFileSequenceReader.java:275)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getAllTimeseriesMetadata(TsFileSequenceReader.java:935)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromReader(TsFileRecoverPerformer.java:183)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromFile(TsFileRecoverPerformer.java:175)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:108)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recoverTsFiles(VirtualStorageGroupProcessor.java:781)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recover(VirtualStorageGroupProcessor.java:519)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.<init>(VirtualStorageGroupProcessor.java:402)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:777)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted
```
### What did you expect to see?
- 写入成功；
### What did you see instead?
- 请分析一下什么情况下造成此种现象？
- 如何规避？
- 出现之后如何解决？
- 是否可以直接删除 错误日志中 提到的那个文件？
- 如果删除这个文件之后会出现什么问题？
- 有没有建议好的策略能够减小该异常的影响范围？
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] ishadowland commented on issue #9284: [Bug] StorageEngine failed to recover 导致无法写入数据
Posted by "ishadowland (via GitHub)" <gi...@apache.org>.
ishadowland commented on issue #9284:
URL: https://github.com/apache/iotdb/issues/9284#issuecomment-1552520237
我也遇到了相同的问题
复现条件是在大量数据写入的情况下 强制重启服务器
iotdb版本1.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SunnySaman commented on issue #9284: [Bug] StorageEngine failed to recover 导致无法写入数据
Posted by "SunnySaman (via GitHub)" <gi...@apache.org>.
SunnySaman commented on issue #9284:
URL: https://github.com/apache/iotdb/issues/9284#issuecomment-1474778364
又遇见一次 `StorageEngine failed to recover.` 导致无法写入数据的问题。但是这次没有说明是什么文件无法恢复，怎么解决呢？
```
2023-03-18 16:30:59,788 [Thread-1] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-1-110
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:410)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Negative position
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:408)
... 1 common frames omitted
Caused by: java.lang.IllegalArgumentException: Negative position
at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:785)
at org.apache.iotdb.tsfile.read.reader.LocalTsFileInput.read(LocalTsFileInput.java:90)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readTailMagic(TsFileSequenceReader.java:229)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.loadMetadataSize(TsFileSequenceReader.java:202)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.<init>(TsFileSequenceReader.java:139)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromReader(TsFileRecoverPerformer.java:181)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromFile(TsFileRecoverPerformer.java:175)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:108)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recoverTsFiles(VirtualStorageGroupProcessor.java:785)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recover(VirtualStorageGroupProcessor.java:526)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.<init>(VirtualStorageGroupProcessor.java:404)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:784)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:245)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SunnySaman commented on issue #9284: [Bug] StorageEngine failed to recover 导致无法写入数据
Posted by "SunnySaman (via GitHub)" <gi...@apache.org>.
SunnySaman commented on issue #9284:
URL: https://github.com/apache/iotdb/issues/9284#issuecomment-1474778484
@qiaojialin 乔老师，请分析下此类问题吧。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SunnySaman commented on issue #9284: [Bug] StorageEngine failed to recover
Posted by "SunnySaman (via GitHub)" <gi...@apache.org>.
SunnySaman commented on issue #9284:
URL: https://github.com/apache/iotdb/issues/9284#issuecomment-1463453063
**修正：iotdb 版本为 0.13.3-SNAPSHOT版本，我把这个数据包拿到0.13.3 版本还原了。但是出现的错误日志信息都是一样的。**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SunnySaman commented on issue #9284: [Bug] StorageEngine failed to recover
Posted by "SunnySaman (via GitHub)" <gi...@apache.org>.
SunnySaman commented on issue #9284:
URL: https://github.com/apache/iotdb/issues/9284#issuecomment-1463444301
日志显示` 1675621909201-1-0-32.tsfile `这个文件recover 异常，其中` 1675621909201-1-0-32.tsfile.resource` 文件大小为`0kb`，我把这个`1675621909201-1-0-32.tsfile.resource` 文件删除之后重启之后数据库恢复正常。
- 请问这个resource文件是做什么用的？
- 和tsfile 文件有什么关系？
- 什么情况下会生成 resource 文件？
- 我看这个数据目录下有多个tsfile 文件及其对应的 resource 文件，请问什么情况下会 完结 一个tsfile 文件和resource 文件？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org