Re: Issue with full disk store directories

[Alberto Bustamante Reyes <al...@est.tech>]

Hi,


An update about this issue. I think the problem is in the PersistentOplogSet class, in the following method:


/**
   * Returns the next available DirectoryHolder which has space. If no dir has space then it will
   * return one anyway if compaction is enabled.
   *
   * @param minAvailableSpace the minimum amount of space we need in this directory.
   */
  DirectoryHolder getNextDir(int minAvailableSpace, boolean checkForWarning)




In order to select a directory, this check is performed:


if (dirHolder.getAvailableSpace() >= minAvailableSpace)


But I think it should compare the available space with the size of the oplog files, as they are created with the maximun size.

After changing the check by:


if (dirHolder.getAvailableSpace() >= parent.getMaxOplogSizeInBytes())


then the full folder is skipped, and the next one is selected.



I'll try to write a test to ilustrate this.


BR/


Alberto B.



________________________________
De: Alberto Bustamante Reyes <al...@est.tech>
Enviado: lunes, 6 de mayo de 2019 10:04:34
Para: dev@geode.apache.org
Asunto: RE: Issue with full disk store directories

Not sure if I understand your question: in the test I did, I used one disk store composed by three directories, each one with different size. These directories were in the same disk partition. The issue I saw is that when the log files are initialized, it is not checked if they fit in the directory, so if the maximum directory sized is reached, the server crashes.
________________________________
De: Anthony Baker <ab...@pivotal.io>
Enviado: lunes, 29 de abril de 2019 17:24
Para: dev@geode.apache.org
Asunto: Re: Issue with full disk store directories

Question:  are you using similarly sized disk partitioned for all your disk stores?

> On Apr 24, 2019, at 3:42 AM, Alberto Bustamante Reyes <al...@est.tech> wrote:
>
> Hi all,
>
> I reported an issue in Jira, related with full disk store directories: https://issues.apache.org/jira/browse/GEODE-6652
> As I describe there, the issue is that when using a disk store with directories of different sizes, when oplog files rotate, the available space of the next disk store directory to be used seems not to be checked correctly.
>
> BR/
>
> Alberto
>
>

Re: Issue with full disk store directories

[Anthony Baker <ab...@pivotal.io>]

Thanks for investigating this.  Looking forward to hearing the results of your testing.  Note that when we create an oplog file we allocate all the bytes (including writing to each disk page) to avoid corruption if the disk becomes full while writing records.

Anthony


> On Jun 7, 2019, at 4:26 AM, Alberto Bustamante Reyes <al...@est.tech> wrote:
> 
> Hi,
> 
> 
> An update about this issue. I think the problem is in the PersistentOplogSet class, in the following method:
> 
> 
> /**
>   * Returns the next available DirectoryHolder which has space. If no dir has space then it will
>   * return one anyway if compaction is enabled.
>   *
>   * @param minAvailableSpace the minimum amount of space we need in this directory.
>   */
>  DirectoryHolder getNextDir(int minAvailableSpace, boolean checkForWarning)
> 
> 
> 
> 
> In order to select a directory, this check is performed:
> 
> 
> if (dirHolder.getAvailableSpace() >= minAvailableSpace)
> 
> 
> But I think it should compare the available space with the size of the oplog files, as they are created with the maximun size.
> 
> After changing the check by:
> 
> 
> if (dirHolder.getAvailableSpace() >= parent.getMaxOplogSizeInBytes())
> 
> 
> then the full folder is skipped, and the next one is selected.
> 
> 
> 
> I'll try to write a test to ilustrate this.
> 
> 
> BR/
> 
> 
> Alberto B.
> 
> 
> 
> ________________________________
> De: Alberto Bustamante Reyes <al...@est.tech>
> Enviado: lunes, 6 de mayo de 2019 10:04:34
> Para: dev@geode.apache.org
> Asunto: RE: Issue with full disk store directories
> 
> Not sure if I understand your question: in the test I did, I used one disk store composed by three directories, each one with different size. These directories were in the same disk partition. The issue I saw is that when the log files are initialized, it is not checked if they fit in the directory, so if the maximum directory sized is reached, the server crashes.
> ________________________________
> De: Anthony Baker <ab...@pivotal.io>
> Enviado: lunes, 29 de abril de 2019 17:24
> Para: dev@geode.apache.org
> Asunto: Re: Issue with full disk store directories
> 
> Question:  are you using similarly sized disk partitioned for all your disk stores?
> 
>> On Apr 24, 2019, at 3:42 AM, Alberto Bustamante Reyes <al...@est.tech> wrote:
>> 
>> Hi all,
>> 
>> I reported an issue in Jira, related with full disk store directories: https://issues.apache.org/jira/browse/GEODE-6652
>> As I describe there, the issue is that when using a disk store with directories of different sizes, when oplog files rotate, the available space of the next disk store directory to be used seems not to be checked correctly.
>> 
>> BR/
>> 
>> Alberto
>> 
>> 
>