You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2019/08/13 08:35:42 UTC
[GitHub] [hadoop] adoroszlai opened a new pull request #1287: HDDS-1956.
Aged IO Thread exits on first read
adoroszlai opened a new pull request #1287: HDDS-1956. Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287
## What changes were proposed in this pull request?
1. Aged IO thread exits on first read due to `ArrayIndexOutOfBoundsException`. This is caused by using `index` as the key name, while `readKey` expects a string separated by `_`.
2. After fixing the above, it randomly exits due to `OMException: Key not found`. This is an off-by-one error. The first key written has `index=1`, but the test randomly attempts to read `index=0`.
3. Add trace level message for read/write/delete operations.
4. Include thread name in log output pattern for `integration-test`.
5. Fix some log messages in `MiniOzoneChaosCluster`
https://issues.apache.org/jira/browse/HDDS-1956
## How was this patch tested?
Manually ran `TestMiniChaosOzoneCluster`:
```
mvn -Phdds -pl :hadoop-ozone-integration-test -Dtest=TestMiniChaosOzoneCluster test
```
Stopped it after a few minutes (see [HDDS-1952](https://issues.apache.org/jira/browse/HDDS-1952)).
Verified that aged IO thread was writing and reading keys (until it started failing due to chaos):
```
2019-08-13 10:19:18,307 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) - LOADGEN: Writing key pool-245-thread-1_0
2019-08-13 10:19:18,433 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) - LOADGEN: Written key pool-245-thread-1_0
2019-08-13 10:19:22,473 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:22,992 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0
...
2019-08-13 10:19:23,837 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) - LOADGEN: Writing key pool-245-thread-1_1
2019-08-13 10:19:23,870 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) - LOADGEN: Written key pool-245-thread-1_1
2019-08-13 10:19:23,891 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_1
2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_1
2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_1
2019-08-13 10:19:24,044 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org