You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/09/07 06:25:04 UTC
[GitHub] [incubator-doris] dh-cloud opened a new pull request #6582: handle bdb rollbackexception
dh-cloud opened a new pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582
when use 3 fe follow, when restart the fes, and regardless of order，we probability can't start fe success，and bdb throw RollbackException，
In this scenario，the bdb suggest should catch the exception，simply closing all your ReplicatedEnvironment handles, and then reopening.
the error stack is:
`
ERROR (replayer|70) [BDBEnvironment.getDatabaseNames():325] bdb environment failure exception.
com.sleepycat.je.rep.RollbackException: (JE 18.1.11) Environment must be closed, caused by: com.sleepycat.je.rep.RollbackException: Environment invalid because of previous exception: (JE 18.1.11) 10.35.25.207_9010_1625733043423(10):bdb Node 10.35.25.207_9010_1625733043423(10):bdb must rollback 1 total commits(1 of which were durable) to the earliest point indicated by transaction id=-67564
time=2021-07-09 09:59:30.901 vlsn=112,533 lsn=0xe/0xa625 durable=false in order to rejoin the replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated. Log files were truncated to file 0x14, offset 0x42489, vlsn 112,532 HARD_RECOVERY: Rolled back
past transaction commit or abort. Must run recovery by re-opening Environment handles
Environment is invalid and must be closed. Originally thrown by HA thread: REPLICA 10.35.25.207_9010_1625733043423(10) Originally thrown by HA thread: REPLICA
10.35.25.207_9010_1625733043423(10) Environment invalid because of previous exception: (JE 18.1.11) 10.35.25.207_9010_1625733043423(10):bdb Node
10.35.25.207_9010_1625733043423(10):bdb must rollback 1 total commits(1 of which were durable) to the earliest point indicated by transaction id=-67564
time=2021-07-09 09:59:30.901 vlsn=112,533 lsn=0xe/0xa625 durable=false in order to rejoin the
replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated.
Log files were truncated to file 0x14, offset 0x42489, vlsn 112,532 HARD_RECOVERY: Rolled back past transaction commit or abort. Must run recovery by re-opening Environment handles Environment is invalid and must be closed. Originally thrown by HA thread: REPLICA 10.35.25.207_9010_1625733043423(10) Originally thrown by HA thread: REPLICA 10.35.25.207_9010_1625733043423(10)
at com.sleepycat.je.rep.RollbackException.wrapSelf(RollbackException.java:146) ~[je-18.1.11.jar:18.1.11]
at com.sleepycat.je.rep.RollbackException.wrapSelf(RollbackException.java:62) ~[je-18.1.11.jar:18.1.11]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1801) ~[je-18.1.11.jar:18.1.11]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1810) ~[je-18.1.11.jar:18.1.11]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[je-18.1.11.jar:18.1.11]
at com.sleepycat.je.Environment.getDatabaseNames(Environment.java:2455) ~[je-18.1.11.jar:18.1.11]
at org.apache.doris.journal.bdbje.BDBEnvironment.getDatabaseNames(BDBEnvironment.java:318) [palo-fe.jar:3.4.0]
at org.apache.doris.journal.bdbje.BDBJEJournal.getMaxJournalId(BDBJEJournal.java:258) [palo-fe.jar:3.4.0]
at org.apache.doris.persist.EditLog.getMaxJournalId(EditLog.java:95) [palo-fe.jar:3.4.0]
at org.apache.doris.catalog.Catalog.getMaxJournalId(Catalog.java:4808) [palo-fe.jar:3.4.0]
at org.apache.doris.catalog.Catalog.replayJournal(Catalog.java:2422) [palo-fe.jar:3.4.0]
at org.apache.doris.catalog.Catalog$3.runOneCycle(Catalog.java:2227) [palo-fe.jar:3.4.0]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116) [palo-fe.jar:3.4.0]
`
so we catch the RollbackException, and reopen the ReplicatedEnvironment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] dh-cloud commented on pull request #6582: handle bdb rollbackexception
Posted by GitBox <gi...@apache.org>.
dh-cloud commented on pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582#issuecomment-914869176
> Hi @dh-cloud , thanks for your PR.
> Can this be reproduced by some steps?
3 fe，execute many insert into statement，during the executing, close the 2 follow fe, close master fe last.
then start original 2 follow fe，wait elect the master fe，then start original master fe. then the new follow fe log is larger then the new master fe, need rollback and hard recovery.
all call bdbEnvironment.getDatabaseNames will throw error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6582: handle bdb rollbackexception
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582#issuecomment-927148883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] dh-cloud edited a comment on pull request #6582: handle bdb rollbackexception
Posted by GitBox <gi...@apache.org>.
dh-cloud edited a comment on pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582#issuecomment-914869176
> Hi @dh-cloud , thanks for your PR.
> Can this be reproduced by some steps?
3 fe，execute many insert into statement，during the executing, close the 2 follow fe, at last close master fe.
then start original 2 follow fe，wait elect the master fe，then start original master fe. then the new follow fe log is larger then the new master fe, need rollback and hard recovery.
all call bdbEnvironment.getDatabaseNames will throw error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6582: handle bdb rollbackexception
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] dh-cloud edited a comment on pull request #6582: handle bdb rollbackexception
Posted by GitBox <gi...@apache.org>.
dh-cloud edited a comment on pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582#issuecomment-914869176
> Hi @dh-cloud , thanks for your PR.
> Can this be reproduced by some steps?
3 fe，execute many insert into statement，during the executing, close the 2 follow fe, at last close master fe.
then start original 2 follow fe，wait elect the master fe，then start original master fe. then the new follow fe log is larger than the new master fe, need rollback and hard recovery.
all call bdbEnvironment.getDatabaseNames will throw error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6582: handle bdb rollbackexception
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #6582:
URL: https://github.com/apache/incubator-doris/pull/6582#issuecomment-914383149
Hi @dh-cloud , thanks for your PR.
Can this be reproduced by some steps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org