You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Amit Pandey <am...@gmail.com> on 2020/09/06 18:06:51 UTC
Question on how Geode handles data on Disk
Hi Geode Devs,
How does Geode handle data on disk ? How does Geode mark that some data of
a region is in disk , is it basically what we call as the Tombstone
approach ? Also if I request data for ID 1 will it bring it only from disk
or
Secondly, how does Geode handle updates of data on disk ? SO does it
discard the data on disk when it brings a tuple in memory and writes a new
file to disk ?
Regards
Re: Question on how Geode handles data on Disk
Posted by Anilkumar Gingade <ag...@vmware.com>.
Amit,
You can find high level details at:
https://geode.apache.org/docs/guide/112/developing/storing_data_on_disk/chapter_overview.html
Geode keeps the Key always in memory. Geode creates different Region-Entries (Key-Value pair) based on the region configuration and how/where data is stored.
When application tries to retrieve data for a given key, it knows where to look for it in the disk and loads it to the memory (only that value).
There are certain operation like querying, where getting the data into memory is avoided, as it does region/index scan.
-Anil.
On 9/7/20, 11:57 PM, "Amit Pandey" <am...@gmail.com> wrote:
I agree its kind of academic but any answers will be appreciated, I just
wanted to understand the performance characteristics when using disk
persistence
On Sun, Sep 6, 2020 at 11:54 PM Amit Pandey <am...@gmail.com>
wrote:
> What I meant here "Also if I request data for ID 1 will it bring it only
> from disk or " was that if I request a tuple which is not in memory and is
> in disk,
> 1) How does Geode know its in DISK
> 2) Does it bring only that tuple (value) to Memory or brings the whole
> page
>
> Regards
>
>
> On Sun, Sep 6, 2020 at 11:36 PM Amit Pandey <am...@gmail.com>
> wrote:
>
>> Hi Geode Devs,
>>
>> How does Geode handle data on disk ? How does Geode mark that some data
>> of a region is in disk , is it basically what we call as the Tombstone
>> approach ? Also if I request data for ID 1 will it bring it only from disk
>> or
>>
>> Secondly, how does Geode handle updates of data on disk ? SO does it
>> discard the data on disk when it brings a tuple in memory and writes a new
>> file to disk ?
>>
>> Regards
>>
>>
Re: Question on how Geode handles data on Disk
Posted by Amit Pandey <am...@gmail.com>.
I agree its kind of academic but any answers will be appreciated, I just
wanted to understand the performance characteristics when using disk
persistence
On Sun, Sep 6, 2020 at 11:54 PM Amit Pandey <am...@gmail.com>
wrote:
> What I meant here "Also if I request data for ID 1 will it bring it only
> from disk or " was that if I request a tuple which is not in memory and is
> in disk,
> 1) How does Geode know its in DISK
> 2) Does it bring only that tuple (value) to Memory or brings the whole
> page
>
> Regards
>
>
> On Sun, Sep 6, 2020 at 11:36 PM Amit Pandey <am...@gmail.com>
> wrote:
>
>> Hi Geode Devs,
>>
>> How does Geode handle data on disk ? How does Geode mark that some data
>> of a region is in disk , is it basically what we call as the Tombstone
>> approach ? Also if I request data for ID 1 will it bring it only from disk
>> or
>>
>> Secondly, how does Geode handle updates of data on disk ? SO does it
>> discard the data on disk when it brings a tuple in memory and writes a new
>> file to disk ?
>>
>> Regards
>>
>>
Re: Question on how Geode handles data on Disk
Posted by Udo Kohlmeyer <ud...@vmware.com>.
Hi there Amit,
On Sep 7, 2020, 4:25 AM +1000, Amit Pandey <am...@gmail.com>, wrote:
What I meant here "Also if I request data for ID 1 will it bring it only
from disk or " was that if I request a tuple which is not in memory and is
in disk,
1) How does Geode know its in DISK
2) Does it bring only that tuple (value) to Memory or brings the whole
page
Regards
On Sun, Sep 6, 2020 at 11:36 PM Amit Pandey <am...@gmail.com>
wrote:
Hi Geode Devs,
How does Geode handle data on disk ? How does Geode mark that some data of
a region is in disk , is it basically what we call as the Tombstone
approach ? Also if I request data for ID 1 will it bring it only from disk
or
Secondly, how does Geode handle updates of data on disk ? SO does it
discard the data on disk when it brings a tuple in memory and writes a new
file to disk ?
Regards
Re: Question on how Geode handles data on Disk
Posted by Udo Kohlmeyer <ud...@vmware.com>.
Hi there Amit,
(Attempt 2),
So, create a Region, you configure if the Region is to persist the data to disk or not. There is no selective algorithm that would select which data is written to disk and which is not. If it is configured to be persistent or eviction overflowed, it will be written to disk. Persistence allows you to shutdown your server and restart it with the saved data, and overflowed will be that data is written to disk on LRU eviction when memory pressures are experienced. Overflowed will only allow for the reading of the value from disk whilst the server is running. If you shutdown the server, any overflowed regions (with non-persistence) will loose all region data for that server. You can configure persistence and overflow on the same region.
When Geode starts up, it reads all keys from disk into memory. At that time, no values are read into memory. There is a special object that we use internally to describe that value, in order for Geode to know where to load that value from disk when required.
When you do a ‘get’ operation, and the data is still on disk, Geode will know how (and where) to load the value (single) from disk, into memory. Geode does not disregard the data on disk, just because it is in memory, but it will interact with the object in memory and not from disk. If you make any CUD operations on the object, it will automatically write the data back down to disk. This is ONLY true for regions configured with Persistence. It is important to note, that any updates to the Object need to be written back to the region using the ‘put’ operation. Any changes made to the object without being written back to the region will lead to data being lost.
Note: It is possible to make in-memory changes on an object, when you use the `copy-on-read=false` setting. In this case, the region could return you the direct memory referenced object, and making changes to the object without committing it back to the region using the ‘put’ method will and can result in data being lost. As the whole replication and persistence cycle is activated on the ‘put’ operation.
Can write a for much longer here.. But let me know if this does not explain the behavior well enough.
—Udo
On Sep 7, 2020, 4:25 AM +1000, Amit Pandey <am...@gmail.com>, wrote:
What I meant here "Also if I request data for ID 1 will it bring it only
from disk or " was that if I request a tuple which is not in memory and is
in disk,
1) How does Geode know its in DISK
2) Does it bring only that tuple (value) to Memory or brings the whole
page
Regards
On Sun, Sep 6, 2020 at 11:36 PM Amit Pandey <am...@gmail.com>
wrote:
Hi Geode Devs,
How does Geode handle data on disk ? How does Geode mark that some data of
a region is in disk , is it basically what we call as the Tombstone
approach ? Also if I request data for ID 1 will it bring it only from disk
or
Secondly, how does Geode handle updates of data on disk ? SO does it
discard the data on disk when it brings a tuple in memory and writes a new
file to disk ?
Regards
Re: Question on how Geode handles data on Disk
Posted by Amit Pandey <am...@gmail.com>.
What I meant here "Also if I request data for ID 1 will it bring it only
from disk or " was that if I request a tuple which is not in memory and is
in disk,
1) How does Geode know its in DISK
2) Does it bring only that tuple (value) to Memory or brings the whole
page
Regards
On Sun, Sep 6, 2020 at 11:36 PM Amit Pandey <am...@gmail.com>
wrote:
> Hi Geode Devs,
>
> How does Geode handle data on disk ? How does Geode mark that some data of
> a region is in disk , is it basically what we call as the Tombstone
> approach ? Also if I request data for ID 1 will it bring it only from disk
> or
>
> Secondly, how does Geode handle updates of data on disk ? SO does it
> discard the data on disk when it brings a tuple in memory and writes a new
> file to disk ?
>
> Regards
>
>