You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Aniruddha (JIRA)" <ji...@apache.org> on 2012/08/19 10:36:37 UTC
[jira] [Created] (BOOKKEEPER-374) Bookkeeper doesn't search all
ledger directories for index files.
Aniruddha created BOOKKEEPER-374:
------------------------------------
Summary: Bookkeeper doesn't search all ledger directories for index files.
Key: BOOKKEEPER-374
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-374
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Affects Versions: 4.2.0
Reporter: Aniruddha
We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to either
1) Return the most recent file
2) Throw an exception if more than one file exists (if that's the expected behavior)
Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-374) Bookkeeper doesn't search all
ledger directories for index files.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437851#comment-13437851 ]
Sijie Guo commented on BOOKKEEPER-374:
--------------------------------------
I am just thinking why there are two index files generated for a same ledger. since the fileInfoCache is protected in synchronized blocks to avoid two FileInfo are put in different directories for same ledger, if there is only one bookie daemon thread, this is case should not happen.
If there are two processes running at the same time, said one process is shutting down, which tries to flush the index to generate an index file; the other process is starting up, which replays journal files, it would find that there is no index file, and generate a new index file in different dir. but this case would only happend before BOOKKEEPER-294, which replays journal files before setting up NIOServer.
@Aniruddha, which version of bookkeeper are you using?
> Bookkeeper doesn't search all ledger directories for index files.
> -----------------------------------------------------------------
>
> Key: BOOKKEEPER-374
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-374
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Aniruddha
>
> We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
> When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
> Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
> It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to
> 1) Throw an exception if more than one file exists (if that's the expected behavior)
> 2) Do something else?
> Any thoughts?
> Edit:
> For another ledger, there were 2 files again, created within a minute of each other, on separate directories and with the same size.
> I ran bin/bookie shell ledger -m <ledger_number>. For the most recent file, all entries were N/A, but the other file had all the actual entries.
> So, returning the most recent file might not be the best approach? Should we perhaps check all files?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (BOOKKEEPER-374) Bookkeeper doesn't search all
ledger directories for index files.
Posted by "Aniruddha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aniruddha resolved BOOKKEEPER-374.
----------------------------------
Resolution: Fixed
> Bookkeeper doesn't search all ledger directories for index files.
> -----------------------------------------------------------------
>
> Key: BOOKKEEPER-374
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-374
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Aniruddha
>
> We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
> When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
> Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
> It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to
> 1) Throw an exception if more than one file exists (if that's the expected behavior)
> 2) Do something else?
> Any thoughts?
> Edit:
> For another ledger, there were 2 files again, created within a minute of each other, on separate directories and with the same size.
> I ran bin/bookie shell ledger -m <ledger_number>. For the most recent file, all entries were N/A, but the other file had all the actual entries.
> So, returning the most recent file might not be the best approach? Should we perhaps check all files?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-374) Bookkeeper doesn't search all
ledger directories for index files.
Posted by "Aniruddha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aniruddha updated BOOKKEEPER-374:
---------------------------------
Description:
We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to
1) Throw an exception if more than one file exists (if that's the expected behavior)
2) Do something else?
Any thoughts?
Edit:
For another ledger, there were 2 files again, created within a minute of each other, on separate directories and with the same size.
I ran bin/bookie shell ledger -m <ledger_number>. For the most recent file, all entries were N/A, but the other file had all the actual entries.
So, returning the most recent file might not be the best approach? Should we perhaps check all files?
was:
We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to either
1) Return the most recent file
2) Throw an exception if more than one file exists (if that's the expected behavior)
Any thoughts?
> Bookkeeper doesn't search all ledger directories for index files.
> -----------------------------------------------------------------
>
> Key: BOOKKEEPER-374
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-374
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Aniruddha
>
> We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
> When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
> Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
> It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to
> 1) Throw an exception if more than one file exists (if that's the expected behavior)
> 2) Do something else?
> Any thoughts?
> Edit:
> For another ledger, there were 2 files again, created within a minute of each other, on separate directories and with the same size.
> I ran bin/bookie shell ledger -m <ledger_number>. For the most recent file, all entries were N/A, but the other file had all the actual entries.
> So, returning the most recent file might not be the best approach? Should we perhaps check all files?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-374) Bookkeeper doesn't search all
ledger directories for index files.
Posted by "Aniruddha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437981#comment-13437981 ]
Aniruddha commented on BOOKKEEPER-374:
--------------------------------------
Yes, that makes sense. We did have two processes running simultaneously. The version we were running was prior to BK-294. I guess upgrading should cause this problem to go away. Thanks.
> Bookkeeper doesn't search all ledger directories for index files.
> -----------------------------------------------------------------
>
> Key: BOOKKEEPER-374
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-374
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Aniruddha
>
> We observed the following behavior when some of the bookkeeper jobs were abruptly restarted a few times.
> When the bookies and hedwig hubs were restarted, the hubs tried to query the entries for a particular topic, but an exception was always returned and the reads never completed successfully.
> Eventually, we found that the ledger from which entries couldn't be read actually had 2 index files on disk. We use multiple ledger directories. The first file was in ledger directory number 2 and had a size of 0. The other file which actually contained all the data was in directory 4.
> It doesn't seem right that one ledger has 2 index files. Also, the findIndexFile() function returns the first found ledger index file. Should this be patched to
> 1) Throw an exception if more than one file exists (if that's the expected behavior)
> 2) Do something else?
> Any thoughts?
> Edit:
> For another ledger, there were 2 files again, created within a minute of each other, on separate directories and with the same size.
> I ran bin/bookie shell ledger -m <ledger_number>. For the most recent file, all entries were N/A, but the other file had all the actual entries.
> So, returning the most recent file might not be the best approach? Should we perhaps check all files?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira