You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/06/22 15:14:40 UTC
[GitHub] [accumulo] milleruntime opened a new issue #2172: Update the LogReader utility
milleruntime opened a new issue #2172:
URL: https://github.com/apache/accumulo/issues/2172
The recent changes to sorted recovery in #2117 broke the `LogReader` utility. It is still using `RecoveryLogReader` to read from sorted WALs. `LogReader` needs to be changed to use the updated `RecoveryLogsIterator`.
<pre>
accumulo wal-info /accumulo/recovery/9cbe3382-7b2a-477e-b1f6-5edeaa401001
2021-06-22T11:12:33,271 [start.Main] ERROR: Thread 'wal-info' died.
java.io.FileNotFoundException: File does not exist: hdfs://localhost:8020/accumulo/recovery/9cbe3382-7b2a-477e-b1f6-5edeaa401001/part-r-00000.rf/data
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1729) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1722) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1737) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1863) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:460) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:433) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:403) ~[hadoop-client-api-3.3.0.jar:?]
at org.apache.accumulo.tserver.log.RecoveryLogReader.<init>(RecoveryLogReader.java:137) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.log.RecoveryLogReader.<init>(RecoveryLogReader.java:120) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.logger.LogReader.execute(LogReader.java:150) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:126) ~[accumulo-start-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) [?:?]
</pre>
Also if you specifiy one of the sorted WALs as a file to read, this error is reported.
<pre>
accumulo wal-info /accumulo/recovery/9cbe3382-7b2a-477e-b1f6-5edeaa401001/part-r-00000.rf
2021-06-22T11:13:23,898 [start.Main] ERROR: Thread 'wal-info' died.
java.lang.IllegalArgumentException: Unsupported write ahead log version ######
at org.apache.accumulo.tserver.log.DfsLogger.getDecryptingStream(DfsLogger.java:378) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.logger.LogReader.execute(LogReader.java:134) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:126) ~[accumulo-start-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) [?:?]
</pre>
We may want to keep the old technique for reading the map files and give an option to delete the old files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2172: Update the LogReader utility
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2172:
URL: https://github.com/apache/accumulo/issues/2172#issuecomment-869657285
> I currently have a workable solution for the first half of the ticket (still needs some revision) but I am currently looking into the second part. It appears the Log File Header is incorrect or at least not something `DfsLogger.getDecryptingStream` expects. I am still investigating but if someone has an idea on a possible solution, I am all ears.
The `DfsLogger` class won't be able to read the sorted recovery RFiles, it can only read the original WALs. One solution I was thinking was to just refactor `LogReader` to use `RecoveryLogsIterator` to read the sorted recovery RFiles if that is what the user passes as a parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] Manno15 commented on issue #2172: Update the LogReader utility
Posted by GitBox <gi...@apache.org>.
Manno15 commented on issue #2172:
URL: https://github.com/apache/accumulo/issues/2172#issuecomment-868731317
I currently have a workable solution for the first half of the ticket (still needs some revision) but I am currently looking into the second part. It appears the Log File Header is incorrect or at least not something `DfsLogger.getDecryptingStream` expects. I am still investigating but if someone has an idea on a possible solution, I am all ears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] Manno15 edited a comment on issue #2172: Update the LogReader utility
Posted by GitBox <gi...@apache.org>.
Manno15 edited a comment on issue #2172:
URL: https://github.com/apache/accumulo/issues/2172#issuecomment-868731317
I currently have a workable solution for the first half of the ticket (still needs some revision) but I am currently looking into the second part. It appears the Log File Header is incorrect or at least not something `DfsLogger.getDecryptingStream` expects. I am still investigating but if someone has an idea on a possible solution, I am all ears.
EDIT: MagicBuffer doesn't equal either magic4 or magic3 inside `DfsLogger.getDecryptingStream`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2172: Update the LogReader utility
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2172:
URL: https://github.com/apache/accumulo/issues/2172#issuecomment-869657285
> I currently have a workable solution for the first half of the ticket (still needs some revision) but I am currently looking into the second part. It appears the Log File Header is incorrect or at least not something `DfsLogger.getDecryptingStream` expects. I am still investigating but if someone has an idea on a possible solution, I am all ears.
The `DfsLogger` class won't be able to read the sorted recovery RFiles, it can only read the original WALs. One solution I was thinking was to just refactor `LogReader` to use `RecoveryLogsIterator` to read the sorted recovery RFiles if that is what the user passes as a parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] Manno15 closed issue #2172: Update the LogReader utility
Posted by GitBox <gi...@apache.org>.
Manno15 closed issue #2172:
URL: https://github.com/apache/accumulo/issues/2172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org