You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Steve Zesch (JIRA)" <ji...@apache.org> on 2014/03/11 20:03:43 UTC
[jira] [Created] (FLUME-2346) idLogFileMap in Log can lose track of
file ids
Steve Zesch created FLUME-2346:
----------------------------------
Summary: idLogFileMap in Log can lose track of file ids
Key: FLUME-2346
URL: https://issues.apache.org/jira/browse/FLUME-2346
Project: Flume
Issue Type: Bug
Components: File Channel
Affects Versions: v1.4.0
Reporter: Steve Zesch
The following code from Log's writeCheckpoint method can lose track of file ids to Reader mappings which will lead to a NPE being thrown in subsequent calls to writeCheckpoint.
{code:title=Log#writeCheckpoint}
Iterator<Integer> idIterator = logFileRefCountsAll.iterator();
while (idIterator.hasNext()) {
int id = idIterator.next();
LogFile.RandomReader reader = idLogFileMap.remove(id);
File file = reader.getFile();
reader.close();
LogFile.MetaDataWriter writer =
LogFileFactory.getMetaDataWriter(file, id);
try {
writer.markCheckpoint(logWriteOrderID);
} finally {
writer.close();
}
reader = LogFileFactory.getRandomReader(file, encryptionKeyProvider);
idLogFileMap.put(id, reader);
LOGGER.debug("Updated checkpoint for file: " + file
+ "logWriteOrderID " + logWriteOrderID);
idIterator.remove();
}
{code}
The problem occurs when writer.markCheckpoint throws an exception and the id -> reader mapping is not added back to idLogFileMap. The next time writeCheckpoint is called logFileRefCountsAll still contains the file id, but idLogFileMap.remove(id) returns null since the id is no longer in the map. Attempting to call reader.getFile() then throws a NPE.
Is there a reason that the initial reader obtained from idLogFileMap is closed and then a new reader for the same file is later created an inserted into the map? If that is not required, then one possible solution would be to remove this logic and not remove the id -> reader mapping in idLogFileMap. If that logic is required, then perhaps the code to insert a new id -> reader mapping into idLogFileMap could be added to the finally block.
--
This message was sent by Atlassian JIRA
(v6.2#6252)