You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "jt2594838 (via GitHub)" <gi...@apache.org> on 2024/07/12 06:59:11 UTC
[PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
jt2594838 opened a new pull request, #12912:
URL: https://github.com/apache/iotdb/pull/12912
As the title indicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #12912:
URL: https://github.com/apache/iotdb/pull/12912#discussion_r1677278700
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALMetaData.java:
##########
@@ -131,38 +131,45 @@ public long getFirstSearchIndex() {
public static WALMetaData readFromWALFile(File logFile, FileChannel channel) throws IOException {
if (channel.size() < WALWriter.MAGIC_STRING_V2_BYTES || !isValidMagicString(channel)) {
- throw new IOException(String.format("Broken wal file %s", logFile));
+ throw new IOException(
+ String.format("Broken wal file %s, size %d", logFile, logFile.length()));
}
// load metadata size
- ByteBuffer metadataSizeBuf = ByteBuffer.allocate(Integer.BYTES);
+ WALMetaData metaData = null;
long position;
- WALFileVersion version = WALFileVersion.getVersion(channel);
- position =
- channel.size()
- - Integer.BYTES
- - (version == WALFileVersion.V2
- ? WALWriter.MAGIC_STRING_V2_BYTES
- : WALWriter.MAGIC_STRING_V1_BYTES);
- channel.read(metadataSizeBuf, position);
- metadataSizeBuf.flip();
- // load metadata
- int metadataSize = metadataSizeBuf.getInt();
- ByteBuffer metadataBuf = ByteBuffer.allocate(metadataSize);
- channel.read(metadataBuf, position - metadataSize);
- metadataBuf.flip();
- WALMetaData metaData = WALMetaData.deserialize(metadataBuf);
- // versions before V1.3, should recover memTable ids from entries
- if (metaData.memTablesId.isEmpty()) {
- int offset = Byte.BYTES;
- for (int size : metaData.buffersSize) {
- channel.position(offset);
- ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
- channel.read(buffer);
- buffer.clear();
- metaData.memTablesId.add(buffer.getLong());
- offset += size;
+ try {
+ ByteBuffer metadataSizeBuf = ByteBuffer.allocate(Integer.BYTES);
+ WALFileVersion version = WALFileVersion.getVersion(channel);
+ position =
+ channel.size()
+ - Integer.BYTES
+ - (version == WALFileVersion.V2
+ ? WALWriter.MAGIC_STRING_V2_BYTES
+ : WALWriter.MAGIC_STRING_V1_BYTES);
+ channel.read(metadataSizeBuf, position);
+ metadataSizeBuf.flip();
+ // load metadata
+ int metadataSize = metadataSizeBuf.getInt();
+ ByteBuffer metadataBuf = ByteBuffer.allocate(metadataSize);
+ channel.read(metadataBuf, position - metadataSize);
+ metadataBuf.flip();
+ metaData = WALMetaData.deserialize(metadataBuf);
+ // versions before V1.3, should recover memTable ids from entries
+ if (metaData.memTablesId.isEmpty()) {
+ int offset = Byte.BYTES;
+ for (int size : metaData.buffersSize) {
+ channel.position(offset);
+ ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ channel.read(buffer);
+ buffer.clear();
+ metaData.memTablesId.add(buffer.getLong());
+ offset += size;
+ }
}
+ } catch (IllegalArgumentException e) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #12912:
URL: https://github.com/apache/iotdb/pull/12912#issuecomment-2227637678
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12912) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12912&metric=new_coverage&view=list) (required ≥ 80%)
 [D Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12912) (required ≥ A)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12912)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #12912:
URL: https://github.com/apache/iotdb/pull/12912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
Posted by "OneSizeFitsQuorum (via GitHub)" <gi...@apache.org>.
OneSizeFitsQuorum commented on code in PR #12912:
URL: https://github.com/apache/iotdb/pull/12912#discussion_r1675728798
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALMetaData.java:
##########
@@ -131,38 +131,45 @@ public long getFirstSearchIndex() {
public static WALMetaData readFromWALFile(File logFile, FileChannel channel) throws IOException {
if (channel.size() < WALWriter.MAGIC_STRING_V2_BYTES || !isValidMagicString(channel)) {
- throw new IOException(String.format("Broken wal file %s", logFile));
+ throw new IOException(
+ String.format("Broken wal file %s, size %d", logFile, logFile.length()));
}
// load metadata size
- ByteBuffer metadataSizeBuf = ByteBuffer.allocate(Integer.BYTES);
+ WALMetaData metaData = null;
long position;
- WALFileVersion version = WALFileVersion.getVersion(channel);
- position =
- channel.size()
- - Integer.BYTES
- - (version == WALFileVersion.V2
- ? WALWriter.MAGIC_STRING_V2_BYTES
- : WALWriter.MAGIC_STRING_V1_BYTES);
- channel.read(metadataSizeBuf, position);
- metadataSizeBuf.flip();
- // load metadata
- int metadataSize = metadataSizeBuf.getInt();
- ByteBuffer metadataBuf = ByteBuffer.allocate(metadataSize);
- channel.read(metadataBuf, position - metadataSize);
- metadataBuf.flip();
- WALMetaData metaData = WALMetaData.deserialize(metadataBuf);
- // versions before V1.3, should recover memTable ids from entries
- if (metaData.memTablesId.isEmpty()) {
- int offset = Byte.BYTES;
- for (int size : metaData.buffersSize) {
- channel.position(offset);
- ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
- channel.read(buffer);
- buffer.clear();
- metaData.memTablesId.add(buffer.getLong());
- offset += size;
+ try {
+ ByteBuffer metadataSizeBuf = ByteBuffer.allocate(Integer.BYTES);
+ WALFileVersion version = WALFileVersion.getVersion(channel);
+ position =
+ channel.size()
+ - Integer.BYTES
+ - (version == WALFileVersion.V2
+ ? WALWriter.MAGIC_STRING_V2_BYTES
+ : WALWriter.MAGIC_STRING_V1_BYTES);
+ channel.read(metadataSizeBuf, position);
+ metadataSizeBuf.flip();
+ // load metadata
+ int metadataSize = metadataSizeBuf.getInt();
+ ByteBuffer metadataBuf = ByteBuffer.allocate(metadataSize);
+ channel.read(metadataBuf, position - metadataSize);
+ metadataBuf.flip();
+ metaData = WALMetaData.deserialize(metadataBuf);
+ // versions before V1.3, should recover memTable ids from entries
+ if (metaData.memTablesId.isEmpty()) {
+ int offset = Byte.BYTES;
+ for (int size : metaData.buffersSize) {
+ channel.position(offset);
+ ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ channel.read(buffer);
+ buffer.clear();
+ metaData.memTablesId.add(buffer.getLong());
+ offset += size;
+ }
}
+ } catch (IllegalArgumentException e) {
Review Comment:
maybe we can catch Exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #12912:
URL: https://github.com/apache/iotdb/pull/12912#issuecomment-2225033667
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12912) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12912&metric=new_coverage&view=list) (required ≥ 80%)
 [D Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12912) (required ≥ A)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12912)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix uncaught IllegalArgumentException when reading broken WAL files [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #12912:
URL: https://github.com/apache/iotdb/pull/12912#issuecomment-2225107501
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/12912?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `76.66667%` with `7 lines` in your changes missing coverage. Please review.
> Project coverage is 41.70%. Comparing base [(`173b24f`)](https://app.codecov.io/gh/apache/iotdb/commit/173b24f50b8888609bc3780eb5e1acdb692e12ed?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`d7ceb00`)](https://app.codecov.io/gh/apache/iotdb/commit/d7ceb0094831dfd316f358f68f1e6e481b7e7f61?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 2 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/iotdb/pull/12912?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...b/storageengine/dataregion/wal/io/WALMetaData.java](https://app.codecov.io/gh/apache/iotdb/pull/12912?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Fwal%2Fio%2FWALMetaData.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3dhbC9pby9XQUxNZXRhRGF0YS5qYXZh) | 76.66% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/12912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12912 +/- ##
============================================
- Coverage 41.70% 41.70% -0.01%
Complexity 71 71
============================================
Files 3322 3322
Lines 210230 210236 +6
Branches 24971 24971
============================================
- Hits 87680 87673 -7
- Misses 122550 122563 +13
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/12912?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org