You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by GitBox <gi...@apache.org> on 2021/12/16 23:02:27 UTC
[GitHub] [trafficserver] masaori335 opened a new pull request #8559: Clear random header value by AIO read error
masaori335 opened a new pull request #8559:
URL: https://github.com/apache/trafficserver/pull/8559
We observed a crash loop on start-up with a bad disk. This is the fix for it.
# Backtrace
```
[ 00 ] libpthread-2.17.so waitpid
[ 01 ] traffic_server crash_logger_invoke(int, siginfo_t*, void*) ( Crash.cc:165 )
[ 02 ] libpthread-2.17.so
[ 03 ] traffic_server CacheDisk::~CacheDisk() ( List.h:293 )
[ 04 ] traffic_server CacheDisk::~CacheDisk() ( CacheDisk.cc:112 )
[ 05 ] traffic_server CacheProcessor::diskInitialized() ( Cache.cc:767 )
[ 06 ] traffic_server CacheDisk::openStart(int, void*) ( CacheDisk.cc:213 )
[ 07 ] traffic_server AIOCallbackInternal::io_complete(int, void*) ( eventsystem/I_Continuation.h:160 )
[ 08 ] traffic_server EThread::process_event(Event*, int) ( I_Continuation.h:160 )
[ 09 ] traffic_server EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) ( UnixEThread.cc:170 )
[ 10 ] traffic_server EThread::execute_regular() ( UnixEThread.cc:230 )
[ 11 ] traffic_server spawn_thread_internal(void*) ( Thread.cc:85 )
[ 12 ] libpthread-2.17.so start_thread
```
# Log
```
[ET_AIO 0:99] WARNING: cache disk operation failed READ -1 61
[ET_NET 9] WARNING: could not read disk header for disk /dev/disk/****: declaring disk bad
[ET_NET 9] WARNING: failed operation: READ (opcode=1), span: /dev/disk/**** (fd=99)
```
# Conditions
1. Some HDD on a box got broken
2. (re)start ATS for some reason
# Scenario
1. On `ET_NET` thread, `CacheDisk::open` schedule AIO read to load disk header info into `DiskHeader *header` member.
2. On `ET_AIO ` thread, `cache_op` try to read the disk but got an error (errno=61). When this error happens, the given buffer (`header`) has a random value.
3. On `ET_NET` thread, `CacheDisk::openStart` handles the callback and declares it's a bad disk because of the result. It deletes the `CacheDisk` object.
4. The `CacheDisk` destructor refers to the `header` which has a random value on a loop and got heap-buffer-overflow.
# Affected Version
8.1.x, 9.0.x, and master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #8559: Clear random header value by AIO read error
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #8559:
URL: https://github.com/apache/trafficserver/pull/8559#issuecomment-1032009712
@ezelkow1 Here you're > https://github.com/apache/trafficserver/pull/8653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ezelkow1 commented on pull request #8559: Clear random header value by AIO read error
Posted by GitBox <gi...@apache.org>.
ezelkow1 commented on pull request #8559:
URL: https://github.com/apache/trafficserver/pull/8559#issuecomment-1031846483
If we want this for 8.x can we get a backport?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] zwoop commented on pull request #8559: Clear random header value by AIO read error
Posted by GitBox <gi...@apache.org>.
zwoop commented on pull request #8559:
URL: https://github.com/apache/trafficserver/pull/8559#issuecomment-1006047269
Cherry-picked to v9.1.x branch.
Cherry-picked to v9.2.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 merged pull request #8559: Clear random header value by AIO read error
Posted by GitBox <gi...@apache.org>.
masaori335 merged pull request #8559:
URL: https://github.com/apache/trafficserver/pull/8559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org