You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bookkeeper.apache.org by Enrico Olivelli - Diennea <en...@diennea.com> on 2015/09/08 14:51:33 UTC
R: Journal file never deleted
I think I go into
https://issues.apache.org/jira/browse/BOOKKEEPER-695
Da: Enrico Olivelli - Diennea [mailto:enrico.olivelli@diennea.com]
Inviato: lunedì 13 luglio 2015 12:58
A: user@bookkeeper.apache.org
Oggetto: R: Journal file never deleted
At time of writing in my test env I have:
- 132 .txn files
- Most of them are 16777216 bytes
- I have 0 ledgers
- journalMaxSizeMB=1048
- logSizeLimit (entryLogSizeLimit)= 2147483648
- size of directory referred by ledgerDirNames is 2.8MB
honestly I can’t estimate the effective size of data which was written to my bookie
I do not have any automatic test case useful to reproduce the problem
The procedure is simple, the are two main use cases:
1) Open ledger and write…..close. drop it , close le client (most common case of course)
2) Open ledger and write, if an error occurs (on other resource or during writes to BK), try to close the ledger, try to close the client, then from another procedure (usually on another machine) open the ledger (with fencing), read all the entries, close the ledger, drop the ledger
a. Important Subcase: errors during the “recovery”…then we need to retry recovery after some time
Da: Sijie Guo [mailto:sijie@apache.org]
Inviato: giovedì 9 luglio 2015 18:30
A: user@bookkeeper.apache.org<ma...@bookkeeper.apache.org>; Enrico Olivelli - Diennea
Oggetto: Re: Journal file never deleted
The org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint are the inner classes 'MAX' and 'MIN'. They are expected if entry log file isn't rotated. As the checkpoint logic in bookie is only checkpoint when entry log file rotated, for performance consideration.
I am not sure if we are on same page. deleting ledgers & stopping writing entries won't help deleting journal files. The way how journal files rotated and deleted works as below:
- all entries are appended in journal
- entries are added into ledger storage and periodic flushed to disk.
- once the ledger storage is flushed, the log mark will be updated, indicating entries added before the mark are persisted. then the journals before the mark is able to be deleted.
- so if there is only one journal, it won't be deleted. if there are more journal files, it would only delete journals that before log mark and keep #getMaxBackupJournals journals before log mark.
so it would be good that you could share more details on the problem you encountered:
- how many journal files you have?
- how many entry log files (.log) you have?
- what is the your journal max size? 'journalMaxSizeMB'
- what is your size of entry log file? 'logSizeLimit'
- how many data you have written?
- how's your procedure on reproducing this behavior?
- Sijie
On Thu, Jul 9, 2015 at 12:13 AM, Enrico Olivelli - Diennea <en...@diennea.com>> wrote:
I reproduced the same problem in a simple staging environment, I deleted every ledger and stopped writing new entries.
I’m trying to do a little debug and I see that in Journal.java there is this line
if (!(checkpoint instanceof LogMarkCheckpoint)) {
return; // we didn't create this checkpoint, so dont do anything with it
}
Actually my test bookie gets a org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint$1@98aabc2<ma...@98aabc2> class instead of a LogMarkCheclpoint, when the
checkpoint is done in the SynchThread#checkpoint method at every FlushInterval
hope this can help
Enrico
Da: Sijie Guo [mailto:sijie@apache.org<ma...@apache.org>]
Inviato: mercoledì 8 luglio 2015 19:06
A: user@bookkeeper.apache.org<ma...@bookkeeper.apache.org>
Oggetto: Re: Journal file never deleted
The journal files are only deleted when its corresponding data in ledger storage is flushed. What is your rate on adding entries? How many .txn files you have?
On Wed, Jul 8, 2015 at 3:27 AM, Enrico Olivelli - Diennea <en...@diennea.com>> wrote:
I’m running all the default values, except from maxbackups which I set to 1
Even if a delete every ledger the logs never get deleted.
I enabled level DEBUG for every logger but I get no useful hint.
Normally at which rate logs file should be deleted ?
Thanks
Enrico
Da: Flavio Junqueira [mailto:fpjunqueira@yahoo.com<ma...@yahoo.com>]
Inviato: mercoledì 8 luglio 2015 12:17
A: user@bookkeeper.apache.org<ma...@bookkeeper.apache.org>
Oggetto: Re: Journal file never deleted
Hi Enrico,
Have you had a look at the journal settings here?
http://bookkeeper.apache.org/docs/r4.3.1/bookieConfigParams.html
-Flavio
On 08 Jul 2015, at 10:37, Enrico Olivelli - Diennea <en...@diennea.com>> wrote:
Hi,
I’m running BookKeeper 4.3.0 (and 4.3.1). I noticed that journal files (*.txn files) never get deleted. I’m using ServerConfiguration.setMaxBackupJournals(1) and default flushinterval of 1 second.
I’m running “embedded” bookies, that is that I launch the bookie inside another JVM, with the following code:
ServerConfiguration conf = new ServerConfiguration();
conf.setBookiePort(port);
conf.setUseHostNameAsBookieID(true);
conf.setJournalDirName(logDir.getAbsolutePath());
conf.setZkServers(zookeeperServer);
conf.setLedgerDirNames(Arrays.stream(dataDirs).map(File::getAbsolutePath).collect(Collectors.toList()).toArray(new String[dataDirs.length]));
conf.setFlushInterval(1000);
conf.setAutoRecoveryDaemonEnabled(false);
…
conf.setMaxBackupJournals(1);
…
this.bookie = new BookieServer(conf);
this.bookee.start();
any suggestion ?
thanks
Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100<tel:%28%2B39%29%C2%A00546%C2%A0066100> - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)
MagNews - E-mail Marketing Solutions
http://www.magnews.it<http://www.magnews.it/>
Diennea - Digital Marketing Solutions
http://www.diennea.com<http://www.diennea.com/>
________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/