You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/07/03 10:36:10 UTC
[GitHub] [incubator-kvrocks] git-hulk created a discussion: We found SEGV fault in tcl tests
GitHub user git-hulk created a discussion: We found SEGV fault in tcl tests
We got below SEGV fault reports in tcl test cases when enabling the ASAN,
it looks the migration behavior went wrong after the master failed to send
the write batch. But I'm not sure how to reproduce this bug, so just record
the error stack here. Welcome to help if you have any idea.
cc @ShooterIT @ChrisZMF
```
tests/tmp/server.21529.21/stderr:==30388==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000f0 (pc 0x55c5e3f1823b bp 0x7fb31cbe58b0 sp 0x7fb31cbe5888 T103)
\ndetail reports:\n
E20220703 10:01:54.671126 25061 replication.cc:128] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E20220703 10:01:54.672175 25046 replication.cc:128] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E20220703 10:01:54.672446 25096 replication.cc:128] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E20220703 10:02:52.464046 32227 slot_migrate.cc:494] [migrate] Got invalid response: -MOVED 10 127.0.0.1:28122, line length: 25
E20220703 10:02:52.464298 32227 slot_migrate.cc:792] [migrate] Wrong response
E20220703 10:02:52.464344 32227 slot_migrate.cc:588] [migrate] Failed to migrate complex key: 7Ok
E20220703 10:02:52.464365 32227 slot_migrate.cc:310] [migrate] Failed to migrate key: 7Ok
E20220703 10:02:52.464390 32227 slot_migrate.cc:181] [migrate] Failed to send snapshot
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20220703 10:02:26.069008 30652 redis_cmd.cc:4401] [replication] Succeed sending file MANIFEST-000011 to 127.0.0.1
ASAN:DEADLYSIGNAL
=================================================================
==30388==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000f0 (pc 0x55c5e3f1823b bp 0x7fb31cbe58b0 sp 0x7fb31cbe5888 T103)
==30388==The signal is caused by a READ memory access.
==30388==Hint: address points to the zero page.
E20220703 10:01:57.912034 25900 slot_migrate.cc:241] [migrate] Failed to connect destination server
E20220703 10:01:57.912315 25900 slot_migrate.cc:170] [migrate] Failed to start migrating
E20220703 10:01:50.633563 24830 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:26.272025 30649 replication.cc:698] [replication] Failed to parallel fetch files while fetch file err: read size: File exists
E20220703 10:02:36.297154 30649 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:37.313882 30649 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:00.684429 26903 replication.cc:77] Ping slave[127.0.0.1:60450] err: Broken pipe, would stop the thread
E20220703 10:02:14.635962 29872 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:15.636242 29872 replication.cc:248] [replication] Failed to connect the master, err: Connection refused
E20220703 10:01:47.295190 23137 redis_cmd.cc:1582] Failed to execute redis command: blpop, err: Invalid argument: WRONGTYPE Operation against a key holding the wrong kind of value
E20220703 10:01:49.304493 23137 redis_cmd.cc:1582] Failed to execute redis command: brpop, err: Invalid argument: WRONGTYPE Operation against a key holding the wrong kind of value
E20220703 10:02:11.485483 29160 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:56.062144 798 slot_migrate.cc:638] [migrate] Stop migrating complex key due to task stopped
E20220703 10:02:56.062376 798 slot_migrate.cc:588] [migrate] Failed to migrate complex key: 4ji
E20220703 10:02:56.062402 798 slot_migrate.cc:310] [migrate] Failed to migrate key: 4ji
E20220703 10:02:56.062427 798 slot_migrate.cc:181] [migrate] Failed to send snapshot
E20220703 10:02:00.316570 27423 replication.cc:248] [replication] Failed to connect the master, err: Connection refused
E20220703 10:02:01.316888 27423 replication.cc:248] [replication] Failed to connect the master, err: Connection refused
E20220703 10:02:04.277168 27444 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:08.042352 28215 replication.cc:162] [replication] connection error/eof, reconnect the master
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20220703 10:02:51.241149 21564 event_listener.cc:132] [event_listener/table_file_deleted] db: ./tests/tmp/server.21533.1/db, sst file: ./tests/tmp/server.21533.1/db/000017.sst, status: OK
E20220703 10:02:51.850924 552 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:46.880172 32246 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:59.856895 1032 slot_migrate.cc:494] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E20220703 10:02:59.857090 1032 slot_migrate.cc:256] [migrate] Failed to notify the destination to prepare to import data
E20220703 10:02:59.857113 1032 slot_migrate.cc:170] [migrate] Failed to start migrating
E20220703 10:02:59.857223 1032 slot_migrate.cc:494] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E20220703 10:02:59.858108 1032 slot_migrate.cc:494] [migrate] Got invalid response: -ERR invalid password, line length: 21
E20220703 10:02:59.858148 1032 slot_migrate.cc:406] [migrate] Failed to auth destination server with 'pass', stop migrating slot 22
E20220703 10:02:59.858170 1032 slot_migrate.cc:170] [migrate] Failed to start migrating
E20220703 10:02:59.858280 1032 slot_migrate.cc:494] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E20220703 10:02:05.646659 26764 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:14.635977 29873 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:44.471841 31018 slot_migrate.cc:476] [migrate] Failed to read response, Err: Connection reset by peer
E20220703 10:02:44.472028 31018 slot_migrate.cc:[792](https://github.com/apache/incubator-kvrocks/runs/7167947569?check_suite_focus=true#step:9:793)] [migrate] Wrong response
E20220703 10:02:44.472072 31018 slot_migrate.cc:588] [migrate] Failed to migrate complex key: 0QG
E20220703 10:02:44.472093 31018 slot_migrate.cc:310] [migrate] Failed to migrate key: 0QG
E20220703 10:02:44.472122 31018 slot_migrate.cc:181] [migrate] Failed to send snapshot
E20220703 10:02:44.472157 31018 slot_migrate.cc:420] [migrate] Failed to send import command to destination, slot: 8, error: Broken pipe
sanitizer error was reported, exiting...
```
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/710
----
This is an automatically sent email for dev@kvrocks.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk deleted a discussion: We found SEGV fault in tcl tests
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk deleted a discussion: We found SEGV fault in tcl tests
We got below SEGV fault reports in tcl test cases when enabling the ASAN,
it looks the migration behavior went wrong after the master failed to send
the write batch. But I'm not sure how to reproduce this bug, so just record
the error stack here. Welcome to help if you have any idea.
cc @ShooterIT @ChrisZMF
```
tests/tmp/server.21529.21/stderr:==30388==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000f0 (pc 0x55c5e3f1823b bp 0x7fb31cbe58b0 sp 0x7fb31cbe5888 T103)
\ndetail reports:\n
E20220703 10:01:54.671126 25061 replication.cc:128] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E20220703 10:01:54.672175 25046 replication.cc:128] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E20220703 10:01:54.672446 25096 replication.cc:128] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E20220703 10:02:52.464046 32227 slot_migrate.cc:494] [migrate] Got invalid response: -MOVED 10 127.0.0.1:28122, line length: 25
E20220703 10:02:52.464298 32227 slot_migrate.cc:792] [migrate] Wrong response
E20220703 10:02:52.464344 32227 slot_migrate.cc:588] [migrate] Failed to migrate complex key: 7Ok
E20220703 10:02:52.464365 32227 slot_migrate.cc:310] [migrate] Failed to migrate key: 7Ok
E20220703 10:02:52.464390 32227 slot_migrate.cc:181] [migrate] Failed to send snapshot
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20220703 10:02:26.069008 30652 redis_cmd.cc:4401] [replication] Succeed sending file MANIFEST-000011 to 127.0.0.1
ASAN:DEADLYSIGNAL
=================================================================
==30388==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000f0 (pc 0x55c5e3f1823b bp 0x7fb31cbe58b0 sp 0x7fb31cbe5888 T103)
==30388==The signal is caused by a READ memory access.
==30388==Hint: address points to the zero page.
E20220703 10:01:57.912034 25900 slot_migrate.cc:241] [migrate] Failed to connect destination server
E20220703 10:01:57.912315 25900 slot_migrate.cc:170] [migrate] Failed to start migrating
E20220703 10:01:50.633563 24830 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:26.272025 30649 replication.cc:698] [replication] Failed to parallel fetch files while fetch file err: read size: File exists
E20220703 10:02:36.297154 30649 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:37.313882 30649 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:00.684429 26903 replication.cc:77] Ping slave[127.0.0.1:60450] err: Broken pipe, would stop the thread
E20220703 10:02:14.635962 29872 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:15.636242 29872 replication.cc:248] [replication] Failed to connect the master, err: Connection refused
E20220703 10:01:47.295190 23137 redis_cmd.cc:1582] Failed to execute redis command: blpop, err: Invalid argument: WRONGTYPE Operation against a key holding the wrong kind of value
E20220703 10:01:49.304493 23137 redis_cmd.cc:1582] Failed to execute redis command: brpop, err: Invalid argument: WRONGTYPE Operation against a key holding the wrong kind of value
E20220703 10:02:11.485483 29160 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:56.062144 798 slot_migrate.cc:638] [migrate] Stop migrating complex key due to task stopped
E20220703 10:02:56.062376 798 slot_migrate.cc:588] [migrate] Failed to migrate complex key: 4ji
E20220703 10:02:56.062402 798 slot_migrate.cc:310] [migrate] Failed to migrate key: 4ji
E20220703 10:02:56.062427 798 slot_migrate.cc:181] [migrate] Failed to send snapshot
E20220703 10:02:00.316570 27423 replication.cc:248] [replication] Failed to connect the master, err: Connection refused
E20220703 10:02:01.316888 27423 replication.cc:248] [replication] Failed to connect the master, err: Connection refused
E20220703 10:02:04.277168 27444 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:08.042352 28215 replication.cc:162] [replication] connection error/eof, reconnect the master
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20220703 10:02:51.241149 21564 event_listener.cc:132] [event_listener/table_file_deleted] db: ./tests/tmp/server.21533.1/db, sst file: ./tests/tmp/server.21533.1/db/000017.sst, status: OK
E20220703 10:02:51.850924 552 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:46.880172 32246 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:59.856895 1032 slot_migrate.cc:494] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E20220703 10:02:59.857090 1032 slot_migrate.cc:256] [migrate] Failed to notify the destination to prepare to import data
E20220703 10:02:59.857113 1032 slot_migrate.cc:170] [migrate] Failed to start migrating
E20220703 10:02:59.857223 1032 slot_migrate.cc:494] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E20220703 10:02:59.858108 1032 slot_migrate.cc:494] [migrate] Got invalid response: -ERR invalid password, line length: 21
E20220703 10:02:59.858148 1032 slot_migrate.cc:406] [migrate] Failed to auth destination server with 'pass', stop migrating slot 22
E20220703 10:02:59.858170 1032 slot_migrate.cc:170] [migrate] Failed to start migrating
E20220703 10:02:59.858280 1032 slot_migrate.cc:494] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E20220703 10:02:05.646659 26764 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:14.635977 29873 replication.cc:162] [replication] connection error/eof, reconnect the master
E20220703 10:02:44.471841 31018 slot_migrate.cc:476] [migrate] Failed to read response, Err: Connection reset by peer
E20220703 10:02:44.472028 31018 slot_migrate.cc:[792](https://github.com/apache/incubator-kvrocks/runs/7167947569?check_suite_focus=true#step:9:793)] [migrate] Wrong response
E20220703 10:02:44.472072 31018 slot_migrate.cc:588] [migrate] Failed to migrate complex key: 0QG
E20220703 10:02:44.472093 31018 slot_migrate.cc:310] [migrate] Failed to migrate key: 0QG
E20220703 10:02:44.472122 31018 slot_migrate.cc:181] [migrate] Failed to send snapshot
E20220703 10:02:44.472157 31018 slot_migrate.cc:420] [migrate] Failed to send import command to destination, slot: 8, error: Broken pipe
sanitizer error was reported, exiting...
```
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/710
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org