You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Dmitriy Govorukhin <dm...@gmail.com> on 2017/05/17 08:16:44 UTC
Persistent Distributed Store Metrics
Folk,
As you know, ignite 2.1 will contain new module (pds), it will be
provide ability to store data on disk. Let's discuss what type of
metrics we need for this?
I think it must be metrics per memory policy, per cache, checkpoint,
and global metrics which will be aggregate all metrics.
I did sketch.
PersistentStoreMetrics.java
public interface PersistentStoreMetrics {
// Global metrics.
public long getMemorySize();
public long getDiskSize();
public long getPagesInMemory();
public long getPagesSizeInMemory();
public long getPagesOnDisk();
public long getPagesSizeOnDisk();
public long getFreePages();
public long getFreePagesSize();
public long getDirtyPages();
public long getDirtyPagesSize();
public long walLog();
public long walLogSize();
// Frequency.
public long getPagesRead();
public long getPagesWrite();
public long getFsync();
public long getWal();
public long getAverageWalFsyncTime();
// Per cache.
public PersistentStoreCacheMetrics cache(String name);
public PersistentStoreCacheMetrics cache(int cacheId);
// For last checkpoint.
public PersistentStoreCheckpointMetrics getLastCheckPoint();
}
>>>>>>>>>>>>>>>>>>>>>>>
PersistentStoreCacheMetrics.java
public interface PersistentStoreCacheMetrics {
public String name();
public double getFillFactor();
public double getFillFactor(int part);
public long getMemorySize();
public long getDiskSize();
public long getPagesInMemory();
public long getPagesSizeInMemory();
public long getPagesOnDisk();
public long getPagesSizeOnDisk();
public long getFreePages();
public long getFreePagesSize();
public long getDirtyPages();
public long getDirtyPagesSize();
public long getPagesRead();
public long getPagesWritten();
}
>>>>>>>>>>>>>>>>>>>>>>>
PersistentStoreCheckpointMetrics.java
public interface PersistentStoreCheckpointMetrics {
public long getTotalPages();
//TODO Page type is internal?
public long[] pagesType();
public long getExecutingTime();
public long getFsyncTime();
public long getPagesCopyOnWrite();
}
Re: Persistent Distributed Store Metrics
Posted by Alexey Goncharuk <al...@gmail.com>.
Denis,
Thanks for the review. I addressed your comments and merged the changes to
ignite-5267 branch.
2017-06-06 22:56 GMT+03:00 Denis Magda <dm...@apache.org>:
> > On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >
> > On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
> >> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>
> >>> Wow, hold on - as far as I remember there was a VOTE to accept the
> >>> contribution of the code into the project _on a branch_. We haven't
> vetted
> >>> its
> >>> inclusion into the next release, We are still at the phase of getting
> >>> familiar
> >>> with the code.
> >>>
> >>
> >> Cos, the community has been vetting the inclusion of the new code for
> over
> >> 3 weeks already (the process and dates are documented here [2]). To be
> >> honest, I am not sure what the appropriate time frame should be, but I
> >> would think that a month would be a good check-in point.
> >
> > I would think it should be as long as we, as the community, are
> comfortable
> > with the massive change coming into the release. We already been through
> the
> > discussion on the timing, etc. and I don't want to harp on that.
>
> Just in case I’m working on the high-level storage documentation. Here is
> a draft for your reference:
> https://apacheignite.readme.io/docs/distributed-persistent-store <
> https://apacheignite.readme.io/docs/distributed-persistent-store>
>
> The page is invisible for wide audience at the moment. Only for Ignite
> committers.
>
> —
> Denis
>
> > On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >
> > On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
> >> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>
> >>> Wow, hold on - as far as I remember there was a VOTE to accept the
> >>> contribution of the code into the project _on a branch_. We haven't
> vetted
> >>> its
> >>> inclusion into the next release, We are still at the phase of getting
> >>> familiar
> >>> with the code.
> >>>
> >>
> >> Cos, the community has been vetting the inclusion of the new code for
> over
> >> 3 weeks already (the process and dates are documented here [2]). To be
> >> honest, I am not sure what the appropriate time frame should be, but I
> >> would think that a month would be a good check-in point.
> >
> > I would think it should be as long as we, as the community, are
> comfortable
> > with the massive change coming into the release. We already been through
> the
> > discussion on the timing, etc. and I don't want to harp on that.
> >
> >> There is also an active stabilization thread for the persistence branch
> >> [3]. I encouraged the community to get involved and post any questions
> or
> >> concerns there as well.
> >>
> >> There is an upcoming in-memory computing conference that is coming up in
> >> June in Amsterdam [4], where there are many Ignite talks scheduled. It
> >> would be great to be able to talk about the persistence features of
> Ignite
> >> there as well. I would like to ask the community to mobilize with
> reviewing
> >> the donated code, so we could have something concrete to tell the
> audience
> >> on the conference.
> >
> > I am sure that having the code on the branch is good enough to be able
> to talk
> > about this. Having this in the release isn't really a contingency to be
> able
> > to make a presentation, right?
> >
> > Cos
> >
> >> [2]
> >> http://incubator.apache.org/ip-clearance/persistent-
> distributed-store-ignite.html
> >> [3]
> >> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-
> Stabilization-for-release-td18288.html
> >> [4] https://imcsummit.org/
> >>
> >>
> >>
> >>> And from what I am seeing in the discussions like this [1], we need to
> be
> >>> extra careful.
> >>>
> >>
> >> I would keep the discussion in [1] separate from the persistence store.
> >> These are 2 unrelated issues. I will respond on [1] either today or
> >> tomorrow, but I agree in general that it should be fixed ASAP.
> >>
> >>
> >>> BTW, you have sent this email 9 days before the vote had happened! A
> bit
> >>> too
> >>> soon, if you ask me.
> >>>
> >>
> >> Cos, this email was sent 1 week after the initial donated code branch
> was
> >> presented to the community (see [2] above). The developers involved were
> >> eager to make it available to the users as soon as possible, but no code
> >> has been merged to the master branch yet.
> >>
> >> I would like to encourage everyone in the community to participate in
> the
> >> persistence branch coding discussions, like the one in this thread, and
> >> familiarize themselves with the code.
> >>
> >>
> >>>
> >>> [1] https://is.gd/UQCr51
> >>>
> >>> Cos
> >>>
> >>> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
> >>>> Folk,
> >>>>
> >>>> As you know, ignite 2.1 will contain new module (pds), it will be
> >>>> provide ability to store data on disk. Let's discuss what type of
> >>>> metrics we need for this?
> >>>> I think it must be metrics per memory policy, per cache, checkpoint,
> >>>> and global metrics which will be aggregate all metrics.
> >>>>
> >>>> I did sketch.
> >>>>
> >>>> PersistentStoreMetrics.java
> >>>>
> >>>> public interface PersistentStoreMetrics {
> >>>>
> >>>> // Global metrics.
> >>>>
> >>>> public long getMemorySize();
> >>>>
> >>>> public long getDiskSize();
> >>>>
> >>>> public long getPagesInMemory();
> >>>>
> >>>> public long getPagesSizeInMemory();
> >>>>
> >>>> public long getPagesOnDisk();
> >>>>
> >>>> public long getPagesSizeOnDisk();
> >>>>
> >>>> public long getFreePages();
> >>>>
> >>>> public long getFreePagesSize();
> >>>>
> >>>> public long getDirtyPages();
> >>>>
> >>>> public long getDirtyPagesSize();
> >>>>
> >>>> public long walLog();
> >>>>
> >>>> public long walLogSize();
> >>>>
> >>>> // Frequency.
> >>>>
> >>>> public long getPagesRead();
> >>>>
> >>>> public long getPagesWrite();
> >>>>
> >>>> public long getFsync();
> >>>>
> >>>> public long getWal();
> >>>>
> >>>> public long getAverageWalFsyncTime();
> >>>>
> >>>> // Per cache.
> >>>>
> >>>> public PersistentStoreCacheMetrics cache(String name);
> >>>>
> >>>> public PersistentStoreCacheMetrics cache(int cacheId);
> >>>>
> >>>> // For last checkpoint.
> >>>>
> >>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCacheMetrics.java
> >>>>
> >>>> public interface PersistentStoreCacheMetrics {
> >>>>
> >>>> public String name();
> >>>>
> >>>> public double getFillFactor();
> >>>>
> >>>> public double getFillFactor(int part);
> >>>>
> >>>> public long getMemorySize();
> >>>>
> >>>> public long getDiskSize();
> >>>>
> >>>> public long getPagesInMemory();
> >>>>
> >>>> public long getPagesSizeInMemory();
> >>>>
> >>>> public long getPagesOnDisk();
> >>>>
> >>>> public long getPagesSizeOnDisk();
> >>>>
> >>>> public long getFreePages();
> >>>>
> >>>> public long getFreePagesSize();
> >>>>
> >>>> public long getDirtyPages();
> >>>>
> >>>> public long getDirtyPagesSize();
> >>>>
> >>>> public long getPagesRead();
> >>>>
> >>>> public long getPagesWritten();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCheckpointMetrics.java
> >>>>
> >>>> public interface PersistentStoreCheckpointMetrics {
> >>>>
> >>>> public long getTotalPages();
> >>>>
> >>>> //TODO Page type is internal?
> >>>> public long[] pagesType();
> >>>>
> >>>> public long getExecutingTime();
> >>>>
> >>>> public long getFsyncTime();
> >>>>
> >>>> public long getPagesCopyOnWrite();
> >>>> }
> >>>
>
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>>
>>
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
>
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.
Just in case I’m working on the high-level storage documentation. Here is a draft for your reference:
https://apacheignite.readme.io/docs/distributed-persistent-store <https://apacheignite.readme.io/docs/distributed-persistent-store>
The page is invisible for wide audience at the moment. Only for Ignite committers.
—
Denis
> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>>
>>
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
>
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.
>
>> There is also an active stabilization thread for the persistence branch
>> [3]. I encouraged the community to get involved and post any questions or
>> concerns there as well.
>>
>> There is an upcoming in-memory computing conference that is coming up in
>> June in Amsterdam [4], where there are many Ignite talks scheduled. It
>> would be great to be able to talk about the persistence features of Ignite
>> there as well. I would like to ask the community to mobilize with reviewing
>> the donated code, so we could have something concrete to tell the audience
>> on the conference.
>
> I am sure that having the code on the branch is good enough to be able to talk
> about this. Having this in the release isn't really a contingency to be able
> to make a presentation, right?
>
> Cos
>
>> [2]
>> http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
>> [3]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
>> [4] https://imcsummit.org/
>>
>>
>>
>>> And from what I am seeing in the discussions like this [1], we need to be
>>> extra careful.
>>>
>>
>> I would keep the discussion in [1] separate from the persistence store.
>> These are 2 unrelated issues. I will respond on [1] either today or
>> tomorrow, but I agree in general that it should be fixed ASAP.
>>
>>
>>> BTW, you have sent this email 9 days before the vote had happened! A bit
>>> too
>>> soon, if you ask me.
>>>
>>
>> Cos, this email was sent 1 week after the initial donated code branch was
>> presented to the community (see [2] above). The developers involved were
>> eager to make it available to the users as soon as possible, but no code
>> has been merged to the master branch yet.
>>
>> I would like to encourage everyone in the community to participate in the
>> persistence branch coding discussions, like the one in this thread, and
>> familiarize themselves with the code.
>>
>>
>>>
>>> [1] https://is.gd/UQCr51
>>>
>>> Cos
>>>
>>> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
>>>> Folk,
>>>>
>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>> provide ability to store data on disk. Let's discuss what type of
>>>> metrics we need for this?
>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>> and global metrics which will be aggregate all metrics.
>>>>
>>>> I did sketch.
>>>>
>>>> PersistentStoreMetrics.java
>>>>
>>>> public interface PersistentStoreMetrics {
>>>>
>>>> // Global metrics.
>>>>
>>>> public long getMemorySize();
>>>>
>>>> public long getDiskSize();
>>>>
>>>> public long getPagesInMemory();
>>>>
>>>> public long getPagesSizeInMemory();
>>>>
>>>> public long getPagesOnDisk();
>>>>
>>>> public long getPagesSizeOnDisk();
>>>>
>>>> public long getFreePages();
>>>>
>>>> public long getFreePagesSize();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getDirtyPagesSize();
>>>>
>>>> public long walLog();
>>>>
>>>> public long walLogSize();
>>>>
>>>> // Frequency.
>>>>
>>>> public long getPagesRead();
>>>>
>>>> public long getPagesWrite();
>>>>
>>>> public long getFsync();
>>>>
>>>> public long getWal();
>>>>
>>>> public long getAverageWalFsyncTime();
>>>>
>>>> // Per cache.
>>>>
>>>> public PersistentStoreCacheMetrics cache(String name);
>>>>
>>>> public PersistentStoreCacheMetrics cache(int cacheId);
>>>>
>>>> // For last checkpoint.
>>>>
>>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCacheMetrics.java
>>>>
>>>> public interface PersistentStoreCacheMetrics {
>>>>
>>>> public String name();
>>>>
>>>> public double getFillFactor();
>>>>
>>>> public double getFillFactor(int part);
>>>>
>>>> public long getMemorySize();
>>>>
>>>> public long getDiskSize();
>>>>
>>>> public long getPagesInMemory();
>>>>
>>>> public long getPagesSizeInMemory();
>>>>
>>>> public long getPagesOnDisk();
>>>>
>>>> public long getPagesSizeOnDisk();
>>>>
>>>> public long getFreePages();
>>>>
>>>> public long getFreePagesSize();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getDirtyPagesSize();
>>>>
>>>> public long getPagesRead();
>>>>
>>>> public long getPagesWritten();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCheckpointMetrics.java
>>>>
>>>> public interface PersistentStoreCheckpointMetrics {
>>>>
>>>> public long getTotalPages();
>>>>
>>>> //TODO Page type is internal?
>>>> public long[] pagesType();
>>>>
>>>> public long getExecutingTime();
>>>>
>>>> public long getFsyncTime();
>>>>
>>>> public long getPagesCopyOnWrite();
>>>> }
>>>
Re: Fwd: Persistent Distributed Store Metrics
Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Wow, hold on - as far as I remember there was a VOTE to accept the
> > contribution of the code into the project _on a branch_. We haven't vetted
> > its
> > inclusion into the next release, We are still at the phase of getting
> > familiar
> > with the code.
> >
>
> Cos, the community has been vetting the inclusion of the new code for over
> 3 weeks already (the process and dates are documented here [2]). To be
> honest, I am not sure what the appropriate time frame should be, but I
> would think that a month would be a good check-in point.
I would think it should be as long as we, as the community, are comfortable
with the massive change coming into the release. We already been through the
discussion on the timing, etc. and I don't want to harp on that.
> There is also an active stabilization thread for the persistence branch
> [3]. I encouraged the community to get involved and post any questions or
> concerns there as well.
>
> There is an upcoming in-memory computing conference that is coming up in
> June in Amsterdam [4], where there are many Ignite talks scheduled. It
> would be great to be able to talk about the persistence features of Ignite
> there as well. I would like to ask the community to mobilize with reviewing
> the donated code, so we could have something concrete to tell the audience
> on the conference.
I am sure that having the code on the branch is good enough to be able to talk
about this. Having this in the release isn't really a contingency to be able
to make a presentation, right?
Cos
> [2]
> http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
> [3]
> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
> [4] https://imcsummit.org/
>
>
>
> > And from what I am seeing in the discussions like this [1], we need to be
> > extra careful.
> >
>
> I would keep the discussion in [1] separate from the persistence store.
> These are 2 unrelated issues. I will respond on [1] either today or
> tomorrow, but I agree in general that it should be fixed ASAP.
>
>
> > BTW, you have sent this email 9 days before the vote had happened! A bit
> > too
> > soon, if you ask me.
> >
>
> Cos, this email was sent 1 week after the initial donated code branch was
> presented to the community (see [2] above). The developers involved were
> eager to make it available to the users as soon as possible, but no code
> has been merged to the master branch yet.
>
> I would like to encourage everyone in the community to participate in the
> persistence branch coding discussions, like the one in this thread, and
> familiarize themselves with the code.
>
>
> >
> > [1] https://is.gd/UQCr51
> >
> > Cos
> >
> > On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
> > > Folk,
> > >
> > > As you know, ignite 2.1 will contain new module (pds), it will be
> > > provide ability to store data on disk. Let's discuss what type of
> > > metrics we need for this?
> > > I think it must be metrics per memory policy, per cache, checkpoint,
> > > and global metrics which will be aggregate all metrics.
> > >
> > > I did sketch.
> > >
> > > PersistentStoreMetrics.java
> > >
> > > public interface PersistentStoreMetrics {
> > >
> > > // Global metrics.
> > >
> > > public long getMemorySize();
> > >
> > > public long getDiskSize();
> > >
> > > public long getPagesInMemory();
> > >
> > > public long getPagesSizeInMemory();
> > >
> > > public long getPagesOnDisk();
> > >
> > > public long getPagesSizeOnDisk();
> > >
> > > public long getFreePages();
> > >
> > > public long getFreePagesSize();
> > >
> > > public long getDirtyPages();
> > >
> > > public long getDirtyPagesSize();
> > >
> > > public long walLog();
> > >
> > > public long walLogSize();
> > >
> > > // Frequency.
> > >
> > > public long getPagesRead();
> > >
> > > public long getPagesWrite();
> > >
> > > public long getFsync();
> > >
> > > public long getWal();
> > >
> > > public long getAverageWalFsyncTime();
> > >
> > > // Per cache.
> > >
> > > public PersistentStoreCacheMetrics cache(String name);
> > >
> > > public PersistentStoreCacheMetrics cache(int cacheId);
> > >
> > > // For last checkpoint.
> > >
> > > public PersistentStoreCheckpointMetrics getLastCheckPoint();
> > > }
> > >
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >
> > > PersistentStoreCacheMetrics.java
> > >
> > > public interface PersistentStoreCacheMetrics {
> > >
> > > public String name();
> > >
> > > public double getFillFactor();
> > >
> > > public double getFillFactor(int part);
> > >
> > > public long getMemorySize();
> > >
> > > public long getDiskSize();
> > >
> > > public long getPagesInMemory();
> > >
> > > public long getPagesSizeInMemory();
> > >
> > > public long getPagesOnDisk();
> > >
> > > public long getPagesSizeOnDisk();
> > >
> > > public long getFreePages();
> > >
> > > public long getFreePagesSize();
> > >
> > > public long getDirtyPages();
> > >
> > > public long getDirtyPagesSize();
> > >
> > > public long getPagesRead();
> > >
> > > public long getPagesWritten();
> > > }
> > >
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >
> > > PersistentStoreCheckpointMetrics.java
> > >
> > > public interface PersistentStoreCheckpointMetrics {
> > >
> > > public long getTotalPages();
> > >
> > > //TODO Page type is internal?
> > > public long[] pagesType();
> > >
> > > public long getExecutingTime();
> > >
> > > public long getFsyncTime();
> > >
> > > public long getPagesCopyOnWrite();
> > > }
> >
Fwd: Persistent Distributed Store Metrics
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Wow, hold on - as far as I remember there was a VOTE to accept the
> contribution of the code into the project _on a branch_. We haven't vetted
> its
> inclusion into the next release, We are still at the phase of getting
> familiar
> with the code.
>
Cos, the community has been vetting the inclusion of the new code for over
3 weeks already (the process and dates are documented here [2]). To be
honest, I am not sure what the appropriate time frame should be, but I
would think that a month would be a good check-in point.
There is also an active stabilization thread for the persistence branch
[3]. I encouraged the community to get involved and post any questions or
concerns there as well.
There is an upcoming in-memory computing conference that is coming up in
June in Amsterdam [4], where there are many Ignite talks scheduled. It
would be great to be able to talk about the persistence features of Ignite
there as well. I would like to ask the community to mobilize with reviewing
the donated code, so we could have something concrete to tell the audience
on the conference.
[2]
http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
[3]
http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
[4] https://imcsummit.org/
> And from what I am seeing in the discussions like this [1], we need to be
> extra careful.
>
I would keep the discussion in [1] separate from the persistence store.
These are 2 unrelated issues. I will respond on [1] either today or
tomorrow, but I agree in general that it should be fixed ASAP.
> BTW, you have sent this email 9 days before the vote had happened! A bit
> too
> soon, if you ask me.
>
Cos, this email was sent 1 week after the initial donated code branch was
presented to the community (see [2] above). The developers involved were
eager to make it available to the users as soon as possible, but no code
has been merged to the master branch yet.
I would like to encourage everyone in the community to participate in the
persistence branch coding discussions, like the one in this thread, and
familiarize themselves with the code.
>
> [1] https://is.gd/UQCr51
>
> Cos
>
> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
> > Folk,
> >
> > As you know, ignite 2.1 will contain new module (pds), it will be
> > provide ability to store data on disk. Let's discuss what type of
> > metrics we need for this?
> > I think it must be metrics per memory policy, per cache, checkpoint,
> > and global metrics which will be aggregate all metrics.
> >
> > I did sketch.
> >
> > PersistentStoreMetrics.java
> >
> > public interface PersistentStoreMetrics {
> >
> > // Global metrics.
> >
> > public long getMemorySize();
> >
> > public long getDiskSize();
> >
> > public long getPagesInMemory();
> >
> > public long getPagesSizeInMemory();
> >
> > public long getPagesOnDisk();
> >
> > public long getPagesSizeOnDisk();
> >
> > public long getFreePages();
> >
> > public long getFreePagesSize();
> >
> > public long getDirtyPages();
> >
> > public long getDirtyPagesSize();
> >
> > public long walLog();
> >
> > public long walLogSize();
> >
> > // Frequency.
> >
> > public long getPagesRead();
> >
> > public long getPagesWrite();
> >
> > public long getFsync();
> >
> > public long getWal();
> >
> > public long getAverageWalFsyncTime();
> >
> > // Per cache.
> >
> > public PersistentStoreCacheMetrics cache(String name);
> >
> > public PersistentStoreCacheMetrics cache(int cacheId);
> >
> > // For last checkpoint.
> >
> > public PersistentStoreCheckpointMetrics getLastCheckPoint();
> > }
> >
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> > PersistentStoreCacheMetrics.java
> >
> > public interface PersistentStoreCacheMetrics {
> >
> > public String name();
> >
> > public double getFillFactor();
> >
> > public double getFillFactor(int part);
> >
> > public long getMemorySize();
> >
> > public long getDiskSize();
> >
> > public long getPagesInMemory();
> >
> > public long getPagesSizeInMemory();
> >
> > public long getPagesOnDisk();
> >
> > public long getPagesSizeOnDisk();
> >
> > public long getFreePages();
> >
> > public long getFreePagesSize();
> >
> > public long getDirtyPages();
> >
> > public long getDirtyPagesSize();
> >
> > public long getPagesRead();
> >
> > public long getPagesWritten();
> > }
> >
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> > PersistentStoreCheckpointMetrics.java
> >
> > public interface PersistentStoreCheckpointMetrics {
> >
> > public long getTotalPages();
> >
> > //TODO Page type is internal?
> > public long[] pagesType();
> >
> > public long getExecutingTime();
> >
> > public long getFsyncTime();
> >
> > public long getPagesCopyOnWrite();
> > }
>
Re: Persistent Distributed Store Metrics
Posted by Konstantin Boudnik <co...@apache.org>.
Wow, hold on - as far as I remember there was a VOTE to accept the
contribution of the code into the project _on a branch_. We haven't vetted its
inclusion into the next release, We are still at the phase of getting familiar
with the code.
And from what I am seeing in the discussions like this [1], we need to be
extra careful.
BTW, you have sent this email 9 days before the vote had happened! A bit too
soon, if you ask me.
[1] https://is.gd/UQCr51
Cos
On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
> Folk,
>
> As you know, ignite 2.1 will contain new module (pds), it will be
> provide ability to store data on disk. Let's discuss what type of
> metrics we need for this?
> I think it must be metrics per memory policy, per cache, checkpoint,
> and global metrics which will be aggregate all metrics.
>
> I did sketch.
>
> PersistentStoreMetrics.java
>
> public interface PersistentStoreMetrics {
>
> // Global metrics.
>
> public long getMemorySize();
>
> public long getDiskSize();
>
> public long getPagesInMemory();
>
> public long getPagesSizeInMemory();
>
> public long getPagesOnDisk();
>
> public long getPagesSizeOnDisk();
>
> public long getFreePages();
>
> public long getFreePagesSize();
>
> public long getDirtyPages();
>
> public long getDirtyPagesSize();
>
> public long walLog();
>
> public long walLogSize();
>
> // Frequency.
>
> public long getPagesRead();
>
> public long getPagesWrite();
>
> public long getFsync();
>
> public long getWal();
>
> public long getAverageWalFsyncTime();
>
> // Per cache.
>
> public PersistentStoreCacheMetrics cache(String name);
>
> public PersistentStoreCacheMetrics cache(int cacheId);
>
> // For last checkpoint.
>
> public PersistentStoreCheckpointMetrics getLastCheckPoint();
> }
>
> >>>>>>>>>>>>>>>>>>>>>>>
>
> PersistentStoreCacheMetrics.java
>
> public interface PersistentStoreCacheMetrics {
>
> public String name();
>
> public double getFillFactor();
>
> public double getFillFactor(int part);
>
> public long getMemorySize();
>
> public long getDiskSize();
>
> public long getPagesInMemory();
>
> public long getPagesSizeInMemory();
>
> public long getPagesOnDisk();
>
> public long getPagesSizeOnDisk();
>
> public long getFreePages();
>
> public long getFreePagesSize();
>
> public long getDirtyPages();
>
> public long getDirtyPagesSize();
>
> public long getPagesRead();
>
> public long getPagesWritten();
> }
>
> >>>>>>>>>>>>>>>>>>>>>>>
>
> PersistentStoreCheckpointMetrics.java
>
> public interface PersistentStoreCheckpointMetrics {
>
> public long getTotalPages();
>
> //TODO Page type is internal?
> public long[] pagesType();
>
> public long getExecutingTime();
>
> public long getFsyncTime();
>
> public long getPagesCopyOnWrite();
> }
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
Dmitriy, please give me a couple of days to look at this.
—
Denis
> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <dm...@gmail.com> wrote:
>
> Folk,
>
> As you know, ignite 2.1 will contain new module (pds), it will be
> provide ability to store data on disk. Let's discuss what type of
> metrics we need for this?
> I think it must be metrics per memory policy, per cache, checkpoint,
> and global metrics which will be aggregate all metrics.
>
> I did sketch.
>
> PersistentStoreMetrics.java
>
> public interface PersistentStoreMetrics {
>
> // Global metrics.
>
> public long getMemorySize();
>
> public long getDiskSize();
>
> public long getPagesInMemory();
>
> public long getPagesSizeInMemory();
>
> public long getPagesOnDisk();
>
> public long getPagesSizeOnDisk();
>
> public long getFreePages();
>
> public long getFreePagesSize();
>
> public long getDirtyPages();
>
> public long getDirtyPagesSize();
>
> public long walLog();
>
> public long walLogSize();
>
> // Frequency.
>
> public long getPagesRead();
>
> public long getPagesWrite();
>
> public long getFsync();
>
> public long getWal();
>
> public long getAverageWalFsyncTime();
>
> // Per cache.
>
> public PersistentStoreCacheMetrics cache(String name);
>
> public PersistentStoreCacheMetrics cache(int cacheId);
>
> // For last checkpoint.
>
> public PersistentStoreCheckpointMetrics getLastCheckPoint();
> }
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
> PersistentStoreCacheMetrics.java
>
> public interface PersistentStoreCacheMetrics {
>
> public String name();
>
> public double getFillFactor();
>
> public double getFillFactor(int part);
>
> public long getMemorySize();
>
> public long getDiskSize();
>
> public long getPagesInMemory();
>
> public long getPagesSizeInMemory();
>
> public long getPagesOnDisk();
>
> public long getPagesSizeOnDisk();
>
> public long getFreePages();
>
> public long getFreePagesSize();
>
> public long getDirtyPages();
>
> public long getDirtyPagesSize();
>
> public long getPagesRead();
>
> public long getPagesWritten();
> }
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
> PersistentStoreCheckpointMetrics.java
>
> public interface PersistentStoreCheckpointMetrics {
>
> public long getTotalPages();
>
> //TODO Page type is internal?
> public long[] pagesType();
>
> public long getExecutingTime();
>
> public long getFsyncTime();
>
> public long getPagesCopyOnWrite();
> }
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
Alex,
Makes sense to me. The only thing that should be fixed is the javadoc of this method:
/**
* Gets a total number of allocated pages in a memory region.
*
* @return Total number of allocated pages.
*/
public long getTotalAllocatedPages();
As far as I understand the metric should represent “all the pages allocated in the persistent store if the one enabled or all the pages we have on disk”. What’d be a right description?
—
Denis
> On Jun 5, 2017, at 9:33 AM, Alexey Goncharuk <al...@gmail.com> wrote:
>
> Guyd,
>
> I've extended the set of metrics a little bit and pushed my changes to
> ignite-5375 branch. Can you please review the metrics and see if we are now
> ok with the interfaces?
>
> 2017-06-01 21:04 GMT+03:00 Denis Magda <dm...@apache.org>:
>
>> Sergey, thanks,
>>
>> You get a green light from my side. Please go ahead and bring this to life
>> unless anyone else has any comments.
>>
>> —
>> Denis
>>
>>> On Jun 1, 2017, at 4:18 AM, Sergey Chugunov <se...@gmail.com>
>> wrote:
>>>
>>> Guys,
>>>
>>> I created a ticket [1] summing up results of our discussion. Please
>> review
>>> and leave comments if something is still unclear there.
>>>
>>> Thanks,
>>> Sergey
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-5375
>>>
>>> On Wed, May 31, 2017 at 11:56 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org>
>>> wrote:
>>>
>>>> Denis, I do agree, the concept of virtual memory may work. I will start
>>>> another discussion thread on that.
>>>>
>>>> We should extensively javadoc all these interfaces. The current
>> one-liner
>>>> javadoc documentation is not enough.
>>>>
>>>> Also, I would like to make a small correction to the MemoryMetrics
>>>> interface, see below...
>>>>
>>>>
>>>> On Wed, May 31, 2017 at 1:35 PM, Denis Magda <dm...@apache.org> wrote:
>>>>
>>>>> Guys,
>>>>> If to assume that the *page memory* is a sort of *virtual* memory that
>>>>> maps a page to RAM or disk then MemoryMetrics interface makes sense if
>> we
>>>>> think of it as {Virtual}MemoryMetrics. It’s late to rename the
>> interface
>>>>> but this design flaw can be handled with a proper documentation.
>>>>> At all, let’s gather all the page memory related metrics (both RAM and
>>>>> disk) in MemoryMetrics interface. Here is it’s final version:
>>>>> public interface MemoryMetrics {
>>>>> public String getName();
>>>>>
>>>>> public long getTotalAllocatedPages();
>>>>
>>>>
>>>> I think this method currently returns only the pages allocated in
>> memory.
>>>> To behave correctly, it now should return the total pages allocated
>> within
>>>> the virtual memory, which should be equal to the total pages on disk.
>>>>
>>>>
>>>>> public long getPagesOnDisk();
>>>>>
>>>>
>>>> This method should be removed as it represents the same value as
>>>> "getTotalAllocatedPages()" method. Instead, we should add the following
>>>> method:
>>>>
>>>> // Returns the number of pages allocated in physical off-heap memory.
>>>> getPhysicalMemoryPages();
>>>>
>>>>
>>>>>
>>>>> public long getDirtyPages();
>>>>>
>>>>> public float getAllocationRate();
>>>>>
>>>>> public float getEvictionRate();
>>>>>
>>>>> public float getPagesFillFactor();
>>>>>
>>>>> public double getPagesMissRate();
>>>>>
>>>>> public float getLargeEntriesPagesPercentage();
>>>>> }
>>>>> I would rename getAllocationRate and getEvitionRate to
>>>>> getPagesAllocationRate and getEvictionRates respectively but then we
>> need
>>>>> to deprecate the existing methods which is not good.
>>>>>
>>>>> As for the persistence metrics related to WAL and checkpointing let’s
>>>>> store them here:
>>>>>
>>>>> public interface PersistentStoreMetrics {
>>>>> public double getWalLoggingRate();
>>>>>
>>>>> public int getWalArchiveSegments();
>>>>>
>>>>> public double getWalFsyncTime();
>>>>>
>>>>> public double getCheckpointingTime();
>>>>>
>>>>> public double getCheckpointingFsyncTime();
>>>>>
>>>>> public long getCheckpointingTotalPagesNumber();
>>>>>
>>>>> public long[] getCheckpointingPagesByTypeNumber();
>>>>>
>>>>> public long getCheckpointingCopiedOnWritePagesNumber();
>>>>> }
>>>>>
>>>>> Comments on the name of non released methods are welcomed.
>>>>>
>>>>> —
>>>>> Denis
>>>>>
>>>>>> On May 30, 2017, at 2:59 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
>>>
>>>>> wrote:
>>>>>>
>>>>>> On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org>
>>>> wrote:
>>>>>>
>>>>>>> I don’t have any extra metrics in mind for now. All I want to say
>> that
>>>>>>> presently the Store metrics interface aggregates WAL and
>> checkpointing
>>>>>>> metrics and in the future we might add additional Store related
>>>> metrics
>>>>>>> there (that has nothing to do with WAL). So, this is why
>>>>> IgniteWalMetrics
>>>>>>> doesn’t sound like a generic interface.
>>>>>>>
>>>>>>
>>>>>> I think it is getting more and more confusing. I, for instance, do not
>>>>>> understand where memory metrics end and storage begins, neither I
>> think
>>>>> it
>>>>>> is important to separate memory policy related metrics from the WAL or
>>>>>> Checkpoint related metrics.
>>>>>>
>>>>>> How about we put them all together and have clear method names?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> —
>>>>>>> Denis
>>>>>>>
>>>>>>>> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <
>>>> dsetrakyan@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> I’m afraid that in the future we might need to add non WAL related
>>>>>>> metrics
>>>>>>>>> under this interface. This is why it might make sense to keep the
>>>> name
>>>>>>>>> generic.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hm... I am not sure we are going to have any other components in
>>>>> addition
>>>>>>>> to WAL and Store here. What do you have in mind?
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: Persistent Distributed Store Metrics
Posted by Alexey Goncharuk <al...@gmail.com>.
Guyd,
I've extended the set of metrics a little bit and pushed my changes to
ignite-5375 branch. Can you please review the metrics and see if we are now
ok with the interfaces?
2017-06-01 21:04 GMT+03:00 Denis Magda <dm...@apache.org>:
> Sergey, thanks,
>
> You get a green light from my side. Please go ahead and bring this to life
> unless anyone else has any comments.
>
> —
> Denis
>
> > On Jun 1, 2017, at 4:18 AM, Sergey Chugunov <se...@gmail.com>
> wrote:
> >
> > Guys,
> >
> > I created a ticket [1] summing up results of our discussion. Please
> review
> > and leave comments if something is still unclear there.
> >
> > Thanks,
> > Sergey
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5375
> >
> > On Wed, May 31, 2017 at 11:56 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> >> Denis, I do agree, the concept of virtual memory may work. I will start
> >> another discussion thread on that.
> >>
> >> We should extensively javadoc all these interfaces. The current
> one-liner
> >> javadoc documentation is not enough.
> >>
> >> Also, I would like to make a small correction to the MemoryMetrics
> >> interface, see below...
> >>
> >>
> >> On Wed, May 31, 2017 at 1:35 PM, Denis Magda <dm...@apache.org> wrote:
> >>
> >>> Guys,
> >>> If to assume that the *page memory* is a sort of *virtual* memory that
> >>> maps a page to RAM or disk then MemoryMetrics interface makes sense if
> we
> >>> think of it as {Virtual}MemoryMetrics. It’s late to rename the
> interface
> >>> but this design flaw can be handled with a proper documentation.
> >>> At all, let’s gather all the page memory related metrics (both RAM and
> >>> disk) in MemoryMetrics interface. Here is it’s final version:
> >>> public interface MemoryMetrics {
> >>> public String getName();
> >>>
> >>> public long getTotalAllocatedPages();
> >>
> >>
> >> I think this method currently returns only the pages allocated in
> memory.
> >> To behave correctly, it now should return the total pages allocated
> within
> >> the virtual memory, which should be equal to the total pages on disk.
> >>
> >>
> >>> public long getPagesOnDisk();
> >>>
> >>
> >> This method should be removed as it represents the same value as
> >> "getTotalAllocatedPages()" method. Instead, we should add the following
> >> method:
> >>
> >> // Returns the number of pages allocated in physical off-heap memory.
> >> getPhysicalMemoryPages();
> >>
> >>
> >>>
> >>> public long getDirtyPages();
> >>>
> >>> public float getAllocationRate();
> >>>
> >>> public float getEvictionRate();
> >>>
> >>> public float getPagesFillFactor();
> >>>
> >>> public double getPagesMissRate();
> >>>
> >>> public float getLargeEntriesPagesPercentage();
> >>> }
> >>> I would rename getAllocationRate and getEvitionRate to
> >>> getPagesAllocationRate and getEvictionRates respectively but then we
> need
> >>> to deprecate the existing methods which is not good.
> >>>
> >>> As for the persistence metrics related to WAL and checkpointing let’s
> >>> store them here:
> >>>
> >>> public interface PersistentStoreMetrics {
> >>> public double getWalLoggingRate();
> >>>
> >>> public int getWalArchiveSegments();
> >>>
> >>> public double getWalFsyncTime();
> >>>
> >>> public double getCheckpointingTime();
> >>>
> >>> public double getCheckpointingFsyncTime();
> >>>
> >>> public long getCheckpointingTotalPagesNumber();
> >>>
> >>> public long[] getCheckpointingPagesByTypeNumber();
> >>>
> >>> public long getCheckpointingCopiedOnWritePagesNumber();
> >>> }
> >>>
> >>> Comments on the name of non released methods are welcomed.
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On May 30, 2017, at 2:59 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> >>> wrote:
> >>>>
> >>>> On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org>
> >> wrote:
> >>>>
> >>>>> I don’t have any extra metrics in mind for now. All I want to say
> that
> >>>>> presently the Store metrics interface aggregates WAL and
> checkpointing
> >>>>> metrics and in the future we might add additional Store related
> >> metrics
> >>>>> there (that has nothing to do with WAL). So, this is why
> >>> IgniteWalMetrics
> >>>>> doesn’t sound like a generic interface.
> >>>>>
> >>>>
> >>>> I think it is getting more and more confusing. I, for instance, do not
> >>>> understand where memory metrics end and storage begins, neither I
> think
> >>> it
> >>>> is important to separate memory policy related metrics from the WAL or
> >>>> Checkpoint related metrics.
> >>>>
> >>>> How about we put them all together and have clear method names?
> >>>>
> >>>>
> >>>>>
> >>>>> —
> >>>>> Denis
> >>>>>
> >>>>>> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <
> >> dsetrakyan@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>>> I’m afraid that in the future we might need to add non WAL related
> >>>>> metrics
> >>>>>>> under this interface. This is why it might make sense to keep the
> >> name
> >>>>>>> generic.
> >>>>>>>
> >>>>>>
> >>>>>> Hm... I am not sure we are going to have any other components in
> >>> addition
> >>>>>> to WAL and Store here. What do you have in mind?
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
Sergey, thanks,
You get a green light from my side. Please go ahead and bring this to life unless anyone else has any comments.
—
Denis
> On Jun 1, 2017, at 4:18 AM, Sergey Chugunov <se...@gmail.com> wrote:
>
> Guys,
>
> I created a ticket [1] summing up results of our discussion. Please review
> and leave comments if something is still unclear there.
>
> Thanks,
> Sergey
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5375
>
> On Wed, May 31, 2017 at 11:56 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> Denis, I do agree, the concept of virtual memory may work. I will start
>> another discussion thread on that.
>>
>> We should extensively javadoc all these interfaces. The current one-liner
>> javadoc documentation is not enough.
>>
>> Also, I would like to make a small correction to the MemoryMetrics
>> interface, see below...
>>
>>
>> On Wed, May 31, 2017 at 1:35 PM, Denis Magda <dm...@apache.org> wrote:
>>
>>> Guys,
>>> If to assume that the *page memory* is a sort of *virtual* memory that
>>> maps a page to RAM or disk then MemoryMetrics interface makes sense if we
>>> think of it as {Virtual}MemoryMetrics. It’s late to rename the interface
>>> but this design flaw can be handled with a proper documentation.
>>> At all, let’s gather all the page memory related metrics (both RAM and
>>> disk) in MemoryMetrics interface. Here is it’s final version:
>>> public interface MemoryMetrics {
>>> public String getName();
>>>
>>> public long getTotalAllocatedPages();
>>
>>
>> I think this method currently returns only the pages allocated in memory.
>> To behave correctly, it now should return the total pages allocated within
>> the virtual memory, which should be equal to the total pages on disk.
>>
>>
>>> public long getPagesOnDisk();
>>>
>>
>> This method should be removed as it represents the same value as
>> "getTotalAllocatedPages()" method. Instead, we should add the following
>> method:
>>
>> // Returns the number of pages allocated in physical off-heap memory.
>> getPhysicalMemoryPages();
>>
>>
>>>
>>> public long getDirtyPages();
>>>
>>> public float getAllocationRate();
>>>
>>> public float getEvictionRate();
>>>
>>> public float getPagesFillFactor();
>>>
>>> public double getPagesMissRate();
>>>
>>> public float getLargeEntriesPagesPercentage();
>>> }
>>> I would rename getAllocationRate and getEvitionRate to
>>> getPagesAllocationRate and getEvictionRates respectively but then we need
>>> to deprecate the existing methods which is not good.
>>>
>>> As for the persistence metrics related to WAL and checkpointing let’s
>>> store them here:
>>>
>>> public interface PersistentStoreMetrics {
>>> public double getWalLoggingRate();
>>>
>>> public int getWalArchiveSegments();
>>>
>>> public double getWalFsyncTime();
>>>
>>> public double getCheckpointingTime();
>>>
>>> public double getCheckpointingFsyncTime();
>>>
>>> public long getCheckpointingTotalPagesNumber();
>>>
>>> public long[] getCheckpointingPagesByTypeNumber();
>>>
>>> public long getCheckpointingCopiedOnWritePagesNumber();
>>> }
>>>
>>> Comments on the name of non released methods are welcomed.
>>>
>>> —
>>> Denis
>>>
>>>> On May 30, 2017, at 2:59 PM, Dmitriy Setrakyan <ds...@apache.org>
>>> wrote:
>>>>
>>>> On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org>
>> wrote:
>>>>
>>>>> I don’t have any extra metrics in mind for now. All I want to say that
>>>>> presently the Store metrics interface aggregates WAL and checkpointing
>>>>> metrics and in the future we might add additional Store related
>> metrics
>>>>> there (that has nothing to do with WAL). So, this is why
>>> IgniteWalMetrics
>>>>> doesn’t sound like a generic interface.
>>>>>
>>>>
>>>> I think it is getting more and more confusing. I, for instance, do not
>>>> understand where memory metrics end and storage begins, neither I think
>>> it
>>>> is important to separate memory policy related metrics from the WAL or
>>>> Checkpoint related metrics.
>>>>
>>>> How about we put them all together and have clear method names?
>>>>
>>>>
>>>>>
>>>>> —
>>>>> Denis
>>>>>
>>>>>> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org>
>>> wrote:
>>>>>>
>>>>>>> I’m afraid that in the future we might need to add non WAL related
>>>>> metrics
>>>>>>> under this interface. This is why it might make sense to keep the
>> name
>>>>>>> generic.
>>>>>>>
>>>>>>
>>>>>> Hm... I am not sure we are going to have any other components in
>>> addition
>>>>>> to WAL and Store here. What do you have in mind?
>>>>>
>>>>>
>>>
>>>
>>
Re: Persistent Distributed Store Metrics
Posted by Sergey Chugunov <se...@gmail.com>.
Guys,
I created a ticket [1] summing up results of our discussion. Please review
and leave comments if something is still unclear there.
Thanks,
Sergey
[1] https://issues.apache.org/jira/browse/IGNITE-5375
On Wed, May 31, 2017 at 11:56 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Denis, I do agree, the concept of virtual memory may work. I will start
> another discussion thread on that.
>
> We should extensively javadoc all these interfaces. The current one-liner
> javadoc documentation is not enough.
>
> Also, I would like to make a small correction to the MemoryMetrics
> interface, see below...
>
>
> On Wed, May 31, 2017 at 1:35 PM, Denis Magda <dm...@apache.org> wrote:
>
> > Guys,
> > If to assume that the *page memory* is a sort of *virtual* memory that
> > maps a page to RAM or disk then MemoryMetrics interface makes sense if we
> > think of it as {Virtual}MemoryMetrics. It’s late to rename the interface
> > but this design flaw can be handled with a proper documentation.
> > At all, let’s gather all the page memory related metrics (both RAM and
> > disk) in MemoryMetrics interface. Here is it’s final version:
> > public interface MemoryMetrics {
> > public String getName();
> >
> > public long getTotalAllocatedPages();
>
>
> I think this method currently returns only the pages allocated in memory.
> To behave correctly, it now should return the total pages allocated within
> the virtual memory, which should be equal to the total pages on disk.
>
>
> > public long getPagesOnDisk();
> >
>
> This method should be removed as it represents the same value as
> "getTotalAllocatedPages()" method. Instead, we should add the following
> method:
>
> // Returns the number of pages allocated in physical off-heap memory.
> getPhysicalMemoryPages();
>
>
> >
> > public long getDirtyPages();
> >
> > public float getAllocationRate();
> >
> > public float getEvictionRate();
> >
> > public float getPagesFillFactor();
> >
> > public double getPagesMissRate();
> >
> > public float getLargeEntriesPagesPercentage();
> > }
> > I would rename getAllocationRate and getEvitionRate to
> > getPagesAllocationRate and getEvictionRates respectively but then we need
> > to deprecate the existing methods which is not good.
> >
> > As for the persistence metrics related to WAL and checkpointing let’s
> > store them here:
> >
> > public interface PersistentStoreMetrics {
> > public double getWalLoggingRate();
> >
> > public int getWalArchiveSegments();
> >
> > public double getWalFsyncTime();
> >
> > public double getCheckpointingTime();
> >
> > public double getCheckpointingFsyncTime();
> >
> > public long getCheckpointingTotalPagesNumber();
> >
> > public long[] getCheckpointingPagesByTypeNumber();
> >
> > public long getCheckpointingCopiedOnWritePagesNumber();
> > }
> >
> > Comments on the name of non released methods are welcomed.
> >
> > —
> > Denis
> >
> > > On May 30, 2017, at 2:59 PM, Dmitriy Setrakyan <ds...@apache.org>
> > wrote:
> > >
> > > On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org>
> wrote:
> > >
> > >> I don’t have any extra metrics in mind for now. All I want to say that
> > >> presently the Store metrics interface aggregates WAL and checkpointing
> > >> metrics and in the future we might add additional Store related
> metrics
> > >> there (that has nothing to do with WAL). So, this is why
> > IgniteWalMetrics
> > >> doesn’t sound like a generic interface.
> > >>
> > >
> > > I think it is getting more and more confusing. I, for instance, do not
> > > understand where memory metrics end and storage begins, neither I think
> > it
> > > is important to separate memory policy related metrics from the WAL or
> > > Checkpoint related metrics.
> > >
> > > How about we put them all together and have clear method names?
> > >
> > >
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > >> wrote:
> > >>>
> > >>> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org>
> > wrote:
> > >>>
> > >>>> I’m afraid that in the future we might need to add non WAL related
> > >> metrics
> > >>>> under this interface. This is why it might make sense to keep the
> name
> > >>>> generic.
> > >>>>
> > >>>
> > >>> Hm... I am not sure we are going to have any other components in
> > addition
> > >>> to WAL and Store here. What do you have in mind?
> > >>
> > >>
> >
> >
>
Re: Persistent Distributed Store Metrics
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Denis, I do agree, the concept of virtual memory may work. I will start
another discussion thread on that.
We should extensively javadoc all these interfaces. The current one-liner
javadoc documentation is not enough.
Also, I would like to make a small correction to the MemoryMetrics
interface, see below...
On Wed, May 31, 2017 at 1:35 PM, Denis Magda <dm...@apache.org> wrote:
> Guys,
> If to assume that the *page memory* is a sort of *virtual* memory that
> maps a page to RAM or disk then MemoryMetrics interface makes sense if we
> think of it as {Virtual}MemoryMetrics. It’s late to rename the interface
> but this design flaw can be handled with a proper documentation.
> At all, let’s gather all the page memory related metrics (both RAM and
> disk) in MemoryMetrics interface. Here is it’s final version:
> public interface MemoryMetrics {
> public String getName();
>
> public long getTotalAllocatedPages();
I think this method currently returns only the pages allocated in memory.
To behave correctly, it now should return the total pages allocated within
the virtual memory, which should be equal to the total pages on disk.
> public long getPagesOnDisk();
>
This method should be removed as it represents the same value as
"getTotalAllocatedPages()" method. Instead, we should add the following
method:
// Returns the number of pages allocated in physical off-heap memory.
getPhysicalMemoryPages();
>
> public long getDirtyPages();
>
> public float getAllocationRate();
>
> public float getEvictionRate();
>
> public float getPagesFillFactor();
>
> public double getPagesMissRate();
>
> public float getLargeEntriesPagesPercentage();
> }
> I would rename getAllocationRate and getEvitionRate to
> getPagesAllocationRate and getEvictionRates respectively but then we need
> to deprecate the existing methods which is not good.
>
> As for the persistence metrics related to WAL and checkpointing let’s
> store them here:
>
> public interface PersistentStoreMetrics {
> public double getWalLoggingRate();
>
> public int getWalArchiveSegments();
>
> public double getWalFsyncTime();
>
> public double getCheckpointingTime();
>
> public double getCheckpointingFsyncTime();
>
> public long getCheckpointingTotalPagesNumber();
>
> public long[] getCheckpointingPagesByTypeNumber();
>
> public long getCheckpointingCopiedOnWritePagesNumber();
> }
>
> Comments on the name of non released methods are welcomed.
>
> —
> Denis
>
> > On May 30, 2017, at 2:59 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> I don’t have any extra metrics in mind for now. All I want to say that
> >> presently the Store metrics interface aggregates WAL and checkpointing
> >> metrics and in the future we might add additional Store related metrics
> >> there (that has nothing to do with WAL). So, this is why
> IgniteWalMetrics
> >> doesn’t sound like a generic interface.
> >>
> >
> > I think it is getting more and more confusing. I, for instance, do not
> > understand where memory metrics end and storage begins, neither I think
> it
> > is important to separate memory policy related metrics from the WAL or
> > Checkpoint related metrics.
> >
> > How about we put them all together and have clear method names?
> >
> >
> >>
> >> —
> >> Denis
> >>
> >>> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <ds...@apache.org>
> >> wrote:
> >>>
> >>> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org>
> wrote:
> >>>
> >>>> I’m afraid that in the future we might need to add non WAL related
> >> metrics
> >>>> under this interface. This is why it might make sense to keep the name
> >>>> generic.
> >>>>
> >>>
> >>> Hm... I am not sure we are going to have any other components in
> addition
> >>> to WAL and Store here. What do you have in mind?
> >>
> >>
>
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
Guys,
If to assume that the *page memory* is a sort of *virtual* memory that maps a page to RAM or disk then MemoryMetrics interface makes sense if we think of it as {Virtual}MemoryMetrics. It’s late to rename the interface but this design flaw can be handled with a proper documentation.
At all, let’s gather all the page memory related metrics (both RAM and disk) in MemoryMetrics interface. Here is it’s final version:
public interface MemoryMetrics {
public String getName();
public long getTotalAllocatedPages();
public long getPagesOnDisk();
public long getDirtyPages();
public float getAllocationRate();
public float getEvictionRate();
public float getPagesFillFactor();
public double getPagesMissRate();
public float getLargeEntriesPagesPercentage();
}
I would rename getAllocationRate and getEvitionRate to getPagesAllocationRate and getEvictionRates respectively but then we need to deprecate the existing methods which is not good.
As for the persistence metrics related to WAL and checkpointing let’s store them here:
public interface PersistentStoreMetrics {
public double getWalLoggingRate();
public int getWalArchiveSegments();
public double getWalFsyncTime();
public double getCheckpointingTime();
public double getCheckpointingFsyncTime();
public long getCheckpointingTotalPagesNumber();
public long[] getCheckpointingPagesByTypeNumber();
public long getCheckpointingCopiedOnWritePagesNumber();
}
Comments on the name of non released methods are welcomed.
—
Denis
> On May 30, 2017, at 2:59 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org> wrote:
>
>> I don’t have any extra metrics in mind for now. All I want to say that
>> presently the Store metrics interface aggregates WAL and checkpointing
>> metrics and in the future we might add additional Store related metrics
>> there (that has nothing to do with WAL). So, this is why IgniteWalMetrics
>> doesn’t sound like a generic interface.
>>
>
> I think it is getting more and more confusing. I, for instance, do not
> understand where memory metrics end and storage begins, neither I think it
> is important to separate memory policy related metrics from the WAL or
> Checkpoint related metrics.
>
> How about we put them all together and have clear method names?
>
>
>>
>> —
>> Denis
>>
>>> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>>>
>>> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org> wrote:
>>>
>>>> I’m afraid that in the future we might need to add non WAL related
>> metrics
>>>> under this interface. This is why it might make sense to keep the name
>>>> generic.
>>>>
>>>
>>> Hm... I am not sure we are going to have any other components in addition
>>> to WAL and Store here. What do you have in mind?
>>
>>
Re: Persistent Distributed Store Metrics
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Tue, May 30, 2017 at 2:46 PM, Denis Magda <dm...@apache.org> wrote:
> I don’t have any extra metrics in mind for now. All I want to say that
> presently the Store metrics interface aggregates WAL and checkpointing
> metrics and in the future we might add additional Store related metrics
> there (that has nothing to do with WAL). So, this is why IgniteWalMetrics
> doesn’t sound like a generic interface.
>
I think it is getting more and more confusing. I, for instance, do not
understand where memory metrics end and storage begins, neither I think it
is important to separate memory policy related metrics from the WAL or
Checkpoint related metrics.
How about we put them all together and have clear method names?
>
> —
> Denis
>
> > On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> I’m afraid that in the future we might need to add non WAL related
> metrics
> >> under this interface. This is why it might make sense to keep the name
> >> generic.
> >>
> >
> > Hm... I am not sure we are going to have any other components in addition
> > to WAL and Store here. What do you have in mind?
>
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
I don’t have any extra metrics in mind for now. All I want to say that presently the Store metrics interface aggregates WAL and checkpointing metrics and in the future we might add additional Store related metrics there (that has nothing to do with WAL). So, this is why IgniteWalMetrics doesn’t sound like a generic interface.
—
Denis
> On May 30, 2017, at 2:39 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org> wrote:
>
>> I’m afraid that in the future we might need to add non WAL related metrics
>> under this interface. This is why it might make sense to keep the name
>> generic.
>>
>
> Hm... I am not sure we are going to have any other components in addition
> to WAL and Store here. What do you have in mind?
Re: Persistent Distributed Store Metrics
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Tue, May 30, 2017 at 1:22 PM, Denis Magda <dm...@apache.org> wrote:
> I’m afraid that in the future we might need to add non WAL related metrics
> under this interface. This is why it might make sense to keep the name
> generic.
>
Hm... I am not sure we are going to have any other components in addition
to WAL and Store here. What do you have in mind?
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
I’m afraid that in the future we might need to add non WAL related metrics under this interface. This is why it might make sense to keep the name generic.
—
Denis
> On May 30, 2017, at 12:29 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> It looks to me like all global metrics have to do with WAL. Why not call
> them IgniteWalMetrics then?
>
> On Tue, May 30, 2017 at 11:58 AM, Denis Magda <dm...@apache.org> wrote:
>
>>> Also, I see your argument about separating memory from disk metrics, but
>> I
>>> think it makes sense to combine them, because it is hard to separate them
>>> from user standpoint. Let's call them IgniteStorageMetrics or something
>> of
>>> the kind.
>>
>> The issue is that we can’t combine all of them because these
>>
>>>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>>>> public double getPageEvictionRate();
>>>>
>>>> public double getPageMissRate();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getPagesOnDisk();
>>>> }
>>
>>
>> are calculated per memory policy (memory region) while these
>>
>>>> public interface PersistenceStoreMetrics {
>>>> //WAL-related metrics
>>>> public double getWalLoggingRate();
>>>>
>>>> public int getWalArchiveSegments();
>>>>
>>>> public double getWalFsyncAverage();
>>>>
>>>> //checkpoint-related metrics
>>>> public double getCpTime();
>>>>
>>>> public double getCpFsyncTime();
>>>>
>>>> public long getCpTotalPages();
>>>>
>>>> public long[] getCpPagesByType();
>>>>
>>>> public long getCpCopiedOnWritePages();
>>>> }
>>
>> are global.
>>
>>
>> —
>> Denis
>>
>>> On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>>>
>>> Several questions:
>>>
>>> 1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should
>> be
>>> consistent with other time metrics, and should be called
>>> "getWalFsyncAverageTime()"
>>> 2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
>>> 3. Also, how about WAL total pages count or total size, whichever one
>> is
>>> easier?
>>> 4. How about the WAL size that has not been checkpointed yet?
>>> 5. How about WAL history size (the number of WAL files that we are
>>> rotating through)?
>>>
>>> Also, I see your argument about separating memory from disk metrics, but
>> I
>>> think it makes sense to combine them, because it is hard to separate them
>>> from user standpoint. Let's call them IgniteStorageMetrics or something
>> of
>>> the kind.
>>>
>>> D.
>>>
>>>
>>> On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <
>> sergey.chugunov@gmail.com>
>>> wrote:
>>>
>>>> Hello Denis,
>>>>
>>>> I have an idea of how we could clarify proposed above approach to
>>>> collecting metrics for PDS.
>>>>
>>>> I suggest to split all metrics into two major groups (disk-related and
>>>> memory-related) and group them in the following interfaces:
>>>>
>>>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>>>> public double getPageEvictionRate();
>>>>
>>>> public double getPageMissRate();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getPagesOnDisk();
>>>> }
>>>>
>>>> public interface PersistenceStoreMetrics {
>>>> //WAL-related metrics
>>>> public double getWalLoggingRate();
>>>>
>>>> public int getWalArchiveSegments();
>>>>
>>>> public double getWalFsyncAverage();
>>>>
>>>> //checkpoint-related metrics
>>>> public double getCpTime();
>>>>
>>>> public double getCpFsyncTime();
>>>>
>>>> public long getCpTotalPages();
>>>>
>>>> public long[] getCpPagesByType();
>>>>
>>>> public long getCpCopiedOnWritePages();
>>>> }
>>>>
>>>> From my standpoint it is important to distinguish between PDS metrics
>> that
>>>> are related mostly to memory management (from the first interface) and
>>>> metrics bound to disk operations (from the second interface).
>>>>
>>>> First group of metrics is useful when collected per Memory Policy. It
>>>> doesn't provide a lot of information when collected across many Memory
>>>> Policies.
>>>>
>>>> Second group is now global (this may change in the future if Persistence
>>>> Policies are introduced) and allows to easily spot any high-level issues
>>>> with durable storage configuration or performance. More specific
>> questions
>>>> like "how this Memory Policy loads a durable storage subsystem?" may be
>>>> deduced from PersistenceMemoryMetrics indicators.
>>>>
>>>> Denis, does it make sense for you?
>>>>
>>>> Any other ideas/questions/improvements or metrics to add to these
>>>> interfaces?
>>>>
>>>> I would like to discuss all the details here before creating any tickets
>>>> for the change.
>>>>
>>>> Thanks,
>>>> Sergey.
>>>>
>>>>
>>>> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dm...@apache.org> wrote:
>>>>
>>>>> Dmitriy,
>>>>>
>>>>> I have some high level concerns.
>>>>>
>>>>> Presently, Ignite already exposes memory/data related metrics via
>>>>> MemoryMetrics and CacheMetrics interface.
>>>>>
>>>>> Looking at PersistentStoreMetrics interface I can’t get why methods
>> like
>>>>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are
>> NOT
>>>>> in MemoryMetrics interface.
>>>>>
>>>>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
>>>>> contains a big subset of the methods from PersistentStoreMetric. There
>>>> just
>>>>> should be some API call that either returns global metrics or cache
>>>>> specific ones. Plus, some of the methods might be added to existing
>>>>> CacheInterface. Why not.
>>>>>
>>>>> I prefer to address this first. Guys, any other thoughts on this?
>>>>>
>>>>> —
>>>>> Denis
>>>>>
>>>>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
>>>>> dmitriy.govorukhin@gmail.com> wrote:
>>>>>>
>>>>>> Folk,
>>>>>>
>>>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>>>> provide ability to store data on disk. Let's discuss what type of
>>>>>> metrics we need for this?
>>>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>>>> and global metrics which will be aggregate all metrics.
>>>>>>
>>>>>> I did sketch.
>>>>>>
>>>>>> PersistentStoreMetrics.java
>>>>>>
>>>>>> public interface PersistentStoreMetrics {
>>>>>>
>>>>>> // Global metrics.
>>>>>>
>>>>>> public long getMemorySize();
>>>>>>
>>>>>> public long getDiskSize();
>>>>>>
>>>>>> public long getPagesInMemory();
>>>>>>
>>>>>> public long getPagesSizeInMemory();
>>>>>>
>>>>>> public long getPagesOnDisk();
>>>>>>
>>>>>> public long getPagesSizeOnDisk();
>>>>>>
>>>>>> public long getFreePages();
>>>>>>
>>>>>> public long getFreePagesSize();
>>>>>>
>>>>>> public long getDirtyPages();
>>>>>>
>>>>>> public long getDirtyPagesSize();
>>>>>>
>>>>>> public long walLog();
>>>>>>
>>>>>> public long walLogSize();
>>>>>>
>>>>>> // Frequency.
>>>>>>
>>>>>> public long getPagesRead();
>>>>>>
>>>>>> public long getPagesWrite();
>>>>>>
>>>>>> public long getFsync();
>>>>>>
>>>>>> public long getWal();
>>>>>>
>>>>>> public long getAverageWalFsyncTime();
>>>>>>
>>>>>> // Per cache.
>>>>>>
>>>>>> public PersistentStoreCacheMetrics cache(String name);
>>>>>>
>>>>>> public PersistentStoreCacheMetrics cache(int cacheId);
>>>>>>
>>>>>> // For last checkpoint.
>>>>>>
>>>>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>>>> }
>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>> PersistentStoreCacheMetrics.java
>>>>>>
>>>>>> public interface PersistentStoreCacheMetrics {
>>>>>>
>>>>>> public String name();
>>>>>>
>>>>>> public double getFillFactor();
>>>>>>
>>>>>> public double getFillFactor(int part);
>>>>>>
>>>>>> public long getMemorySize();
>>>>>>
>>>>>> public long getDiskSize();
>>>>>>
>>>>>> public long getPagesInMemory();
>>>>>>
>>>>>> public long getPagesSizeInMemory();
>>>>>>
>>>>>> public long getPagesOnDisk();
>>>>>>
>>>>>> public long getPagesSizeOnDisk();
>>>>>>
>>>>>> public long getFreePages();
>>>>>>
>>>>>> public long getFreePagesSize();
>>>>>>
>>>>>> public long getDirtyPages();
>>>>>>
>>>>>> public long getDirtyPagesSize();
>>>>>>
>>>>>> public long getPagesRead();
>>>>>>
>>>>>> public long getPagesWritten();
>>>>>> }
>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>> PersistentStoreCheckpointMetrics.java
>>>>>>
>>>>>> public interface PersistentStoreCheckpointMetrics {
>>>>>>
>>>>>> public long getTotalPages();
>>>>>>
>>>>>> //TODO Page type is internal?
>>>>>> public long[] pagesType();
>>>>>>
>>>>>> public long getExecutingTime();
>>>>>>
>>>>>> public long getFsyncTime();
>>>>>>
>>>>>> public long getPagesCopyOnWrite();
>>>>>> }
>>>>>
>>>>>
>>>>
>>
>>
Re: Persistent Distributed Store Metrics
Posted by Dmitriy Setrakyan <ds...@apache.org>.
It looks to me like all global metrics have to do with WAL. Why not call
them IgniteWalMetrics then?
On Tue, May 30, 2017 at 11:58 AM, Denis Magda <dm...@apache.org> wrote:
> > Also, I see your argument about separating memory from disk metrics, but
> I
> > think it makes sense to combine them, because it is hard to separate them
> > from user standpoint. Let's call them IgniteStorageMetrics or something
> of
> > the kind.
>
> The issue is that we can’t combine all of them because these
>
> >> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> >> public double getPageEvictionRate();
> >>
> >> public double getPageMissRate();
> >>
> >> public long getDirtyPages();
> >>
> >> public long getPagesOnDisk();
> >> }
>
>
> are calculated per memory policy (memory region) while these
>
> >> public interface PersistenceStoreMetrics {
> >> //WAL-related metrics
> >> public double getWalLoggingRate();
> >>
> >> public int getWalArchiveSegments();
> >>
> >> public double getWalFsyncAverage();
> >>
> >> //checkpoint-related metrics
> >> public double getCpTime();
> >>
> >> public double getCpFsyncTime();
> >>
> >> public long getCpTotalPages();
> >>
> >> public long[] getCpPagesByType();
> >>
> >> public long getCpCopiedOnWritePages();
> >> }
>
> are global.
>
>
> —
> Denis
>
> > On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > Several questions:
> >
> > 1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should
> be
> > consistent with other time metrics, and should be called
> > "getWalFsyncAverageTime()"
> > 2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
> > 3. Also, how about WAL total pages count or total size, whichever one
> is
> > easier?
> > 4. How about the WAL size that has not been checkpointed yet?
> > 5. How about WAL history size (the number of WAL files that we are
> > rotating through)?
> >
> > Also, I see your argument about separating memory from disk metrics, but
> I
> > think it makes sense to combine them, because it is hard to separate them
> > from user standpoint. Let's call them IgniteStorageMetrics or something
> of
> > the kind.
> >
> > D.
> >
> >
> > On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> > wrote:
> >
> >> Hello Denis,
> >>
> >> I have an idea of how we could clarify proposed above approach to
> >> collecting metrics for PDS.
> >>
> >> I suggest to split all metrics into two major groups (disk-related and
> >> memory-related) and group them in the following interfaces:
> >>
> >> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> >> public double getPageEvictionRate();
> >>
> >> public double getPageMissRate();
> >>
> >> public long getDirtyPages();
> >>
> >> public long getPagesOnDisk();
> >> }
> >>
> >> public interface PersistenceStoreMetrics {
> >> //WAL-related metrics
> >> public double getWalLoggingRate();
> >>
> >> public int getWalArchiveSegments();
> >>
> >> public double getWalFsyncAverage();
> >>
> >> //checkpoint-related metrics
> >> public double getCpTime();
> >>
> >> public double getCpFsyncTime();
> >>
> >> public long getCpTotalPages();
> >>
> >> public long[] getCpPagesByType();
> >>
> >> public long getCpCopiedOnWritePages();
> >> }
> >>
> >> From my standpoint it is important to distinguish between PDS metrics
> that
> >> are related mostly to memory management (from the first interface) and
> >> metrics bound to disk operations (from the second interface).
> >>
> >> First group of metrics is useful when collected per Memory Policy. It
> >> doesn't provide a lot of information when collected across many Memory
> >> Policies.
> >>
> >> Second group is now global (this may change in the future if Persistence
> >> Policies are introduced) and allows to easily spot any high-level issues
> >> with durable storage configuration or performance. More specific
> questions
> >> like "how this Memory Policy loads a durable storage subsystem?" may be
> >> deduced from PersistenceMemoryMetrics indicators.
> >>
> >> Denis, does it make sense for you?
> >>
> >> Any other ideas/questions/improvements or metrics to add to these
> >> interfaces?
> >>
> >> I would like to discuss all the details here before creating any tickets
> >> for the change.
> >>
> >> Thanks,
> >> Sergey.
> >>
> >>
> >> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dm...@apache.org> wrote:
> >>
> >>> Dmitriy,
> >>>
> >>> I have some high level concerns.
> >>>
> >>> Presently, Ignite already exposes memory/data related metrics via
> >>> MemoryMetrics and CacheMetrics interface.
> >>>
> >>> Looking at PersistentStoreMetrics interface I can’t get why methods
> like
> >>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are
> NOT
> >>> in MemoryMetrics interface.
> >>>
> >>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
> >>> contains a big subset of the methods from PersistentStoreMetric. There
> >> just
> >>> should be some API call that either returns global metrics or cache
> >>> specific ones. Plus, some of the methods might be added to existing
> >>> CacheInterface. Why not.
> >>>
> >>> I prefer to address this first. Guys, any other thoughts on this?
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
> >>> dmitriy.govorukhin@gmail.com> wrote:
> >>>>
> >>>> Folk,
> >>>>
> >>>> As you know, ignite 2.1 will contain new module (pds), it will be
> >>>> provide ability to store data on disk. Let's discuss what type of
> >>>> metrics we need for this?
> >>>> I think it must be metrics per memory policy, per cache, checkpoint,
> >>>> and global metrics which will be aggregate all metrics.
> >>>>
> >>>> I did sketch.
> >>>>
> >>>> PersistentStoreMetrics.java
> >>>>
> >>>> public interface PersistentStoreMetrics {
> >>>>
> >>>> // Global metrics.
> >>>>
> >>>> public long getMemorySize();
> >>>>
> >>>> public long getDiskSize();
> >>>>
> >>>> public long getPagesInMemory();
> >>>>
> >>>> public long getPagesSizeInMemory();
> >>>>
> >>>> public long getPagesOnDisk();
> >>>>
> >>>> public long getPagesSizeOnDisk();
> >>>>
> >>>> public long getFreePages();
> >>>>
> >>>> public long getFreePagesSize();
> >>>>
> >>>> public long getDirtyPages();
> >>>>
> >>>> public long getDirtyPagesSize();
> >>>>
> >>>> public long walLog();
> >>>>
> >>>> public long walLogSize();
> >>>>
> >>>> // Frequency.
> >>>>
> >>>> public long getPagesRead();
> >>>>
> >>>> public long getPagesWrite();
> >>>>
> >>>> public long getFsync();
> >>>>
> >>>> public long getWal();
> >>>>
> >>>> public long getAverageWalFsyncTime();
> >>>>
> >>>> // Per cache.
> >>>>
> >>>> public PersistentStoreCacheMetrics cache(String name);
> >>>>
> >>>> public PersistentStoreCacheMetrics cache(int cacheId);
> >>>>
> >>>> // For last checkpoint.
> >>>>
> >>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCacheMetrics.java
> >>>>
> >>>> public interface PersistentStoreCacheMetrics {
> >>>>
> >>>> public String name();
> >>>>
> >>>> public double getFillFactor();
> >>>>
> >>>> public double getFillFactor(int part);
> >>>>
> >>>> public long getMemorySize();
> >>>>
> >>>> public long getDiskSize();
> >>>>
> >>>> public long getPagesInMemory();
> >>>>
> >>>> public long getPagesSizeInMemory();
> >>>>
> >>>> public long getPagesOnDisk();
> >>>>
> >>>> public long getPagesSizeOnDisk();
> >>>>
> >>>> public long getFreePages();
> >>>>
> >>>> public long getFreePagesSize();
> >>>>
> >>>> public long getDirtyPages();
> >>>>
> >>>> public long getDirtyPagesSize();
> >>>>
> >>>> public long getPagesRead();
> >>>>
> >>>> public long getPagesWritten();
> >>>> }
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>> PersistentStoreCheckpointMetrics.java
> >>>>
> >>>> public interface PersistentStoreCheckpointMetrics {
> >>>>
> >>>> public long getTotalPages();
> >>>>
> >>>> //TODO Page type is internal?
> >>>> public long[] pagesType();
> >>>>
> >>>> public long getExecutingTime();
> >>>>
> >>>> public long getFsyncTime();
> >>>>
> >>>> public long getPagesCopyOnWrite();
> >>>> }
> >>>
> >>>
> >>
>
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
> Also, I see your argument about separating memory from disk metrics, but I
> think it makes sense to combine them, because it is hard to separate them
> from user standpoint. Let's call them IgniteStorageMetrics or something of
> the kind.
The issue is that we can’t combine all of them because these
>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>> public double getPageEvictionRate();
>>
>> public double getPageMissRate();
>>
>> public long getDirtyPages();
>>
>> public long getPagesOnDisk();
>> }
are calculated per memory policy (memory region) while these
>> public interface PersistenceStoreMetrics {
>> //WAL-related metrics
>> public double getWalLoggingRate();
>>
>> public int getWalArchiveSegments();
>>
>> public double getWalFsyncAverage();
>>
>> //checkpoint-related metrics
>> public double getCpTime();
>>
>> public double getCpFsyncTime();
>>
>> public long getCpTotalPages();
>>
>> public long[] getCpPagesByType();
>>
>> public long getCpCopiedOnWritePages();
>> }
are global.
—
Denis
> On May 30, 2017, at 11:44 AM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> Several questions:
>
> 1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should be
> consistent with other time metrics, and should be called
> "getWalFsyncAverageTime()"
> 2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
> 3. Also, how about WAL total pages count or total size, whichever one is
> easier?
> 4. How about the WAL size that has not been checkpointed yet?
> 5. How about WAL history size (the number of WAL files that we are
> rotating through)?
>
> Also, I see your argument about separating memory from disk metrics, but I
> think it makes sense to combine them, because it is hard to separate them
> from user standpoint. Let's call them IgniteStorageMetrics or something of
> the kind.
>
> D.
>
>
> On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <se...@gmail.com>
> wrote:
>
>> Hello Denis,
>>
>> I have an idea of how we could clarify proposed above approach to
>> collecting metrics for PDS.
>>
>> I suggest to split all metrics into two major groups (disk-related and
>> memory-related) and group them in the following interfaces:
>>
>> public interface PersistenceMemoryMetrics extends MemoryMetrics {
>> public double getPageEvictionRate();
>>
>> public double getPageMissRate();
>>
>> public long getDirtyPages();
>>
>> public long getPagesOnDisk();
>> }
>>
>> public interface PersistenceStoreMetrics {
>> //WAL-related metrics
>> public double getWalLoggingRate();
>>
>> public int getWalArchiveSegments();
>>
>> public double getWalFsyncAverage();
>>
>> //checkpoint-related metrics
>> public double getCpTime();
>>
>> public double getCpFsyncTime();
>>
>> public long getCpTotalPages();
>>
>> public long[] getCpPagesByType();
>>
>> public long getCpCopiedOnWritePages();
>> }
>>
>> From my standpoint it is important to distinguish between PDS metrics that
>> are related mostly to memory management (from the first interface) and
>> metrics bound to disk operations (from the second interface).
>>
>> First group of metrics is useful when collected per Memory Policy. It
>> doesn't provide a lot of information when collected across many Memory
>> Policies.
>>
>> Second group is now global (this may change in the future if Persistence
>> Policies are introduced) and allows to easily spot any high-level issues
>> with durable storage configuration or performance. More specific questions
>> like "how this Memory Policy loads a durable storage subsystem?" may be
>> deduced from PersistenceMemoryMetrics indicators.
>>
>> Denis, does it make sense for you?
>>
>> Any other ideas/questions/improvements or metrics to add to these
>> interfaces?
>>
>> I would like to discuss all the details here before creating any tickets
>> for the change.
>>
>> Thanks,
>> Sergey.
>>
>>
>> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dm...@apache.org> wrote:
>>
>>> Dmitriy,
>>>
>>> I have some high level concerns.
>>>
>>> Presently, Ignite already exposes memory/data related metrics via
>>> MemoryMetrics and CacheMetrics interface.
>>>
>>> Looking at PersistentStoreMetrics interface I can’t get why methods like
>>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT
>>> in MemoryMetrics interface.
>>>
>>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
>>> contains a big subset of the methods from PersistentStoreMetric. There
>> just
>>> should be some API call that either returns global metrics or cache
>>> specific ones. Plus, some of the methods might be added to existing
>>> CacheInterface. Why not.
>>>
>>> I prefer to address this first. Guys, any other thoughts on this?
>>>
>>> —
>>> Denis
>>>
>>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
>>> dmitriy.govorukhin@gmail.com> wrote:
>>>>
>>>> Folk,
>>>>
>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>> provide ability to store data on disk. Let's discuss what type of
>>>> metrics we need for this?
>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>> and global metrics which will be aggregate all metrics.
>>>>
>>>> I did sketch.
>>>>
>>>> PersistentStoreMetrics.java
>>>>
>>>> public interface PersistentStoreMetrics {
>>>>
>>>> // Global metrics.
>>>>
>>>> public long getMemorySize();
>>>>
>>>> public long getDiskSize();
>>>>
>>>> public long getPagesInMemory();
>>>>
>>>> public long getPagesSizeInMemory();
>>>>
>>>> public long getPagesOnDisk();
>>>>
>>>> public long getPagesSizeOnDisk();
>>>>
>>>> public long getFreePages();
>>>>
>>>> public long getFreePagesSize();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getDirtyPagesSize();
>>>>
>>>> public long walLog();
>>>>
>>>> public long walLogSize();
>>>>
>>>> // Frequency.
>>>>
>>>> public long getPagesRead();
>>>>
>>>> public long getPagesWrite();
>>>>
>>>> public long getFsync();
>>>>
>>>> public long getWal();
>>>>
>>>> public long getAverageWalFsyncTime();
>>>>
>>>> // Per cache.
>>>>
>>>> public PersistentStoreCacheMetrics cache(String name);
>>>>
>>>> public PersistentStoreCacheMetrics cache(int cacheId);
>>>>
>>>> // For last checkpoint.
>>>>
>>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCacheMetrics.java
>>>>
>>>> public interface PersistentStoreCacheMetrics {
>>>>
>>>> public String name();
>>>>
>>>> public double getFillFactor();
>>>>
>>>> public double getFillFactor(int part);
>>>>
>>>> public long getMemorySize();
>>>>
>>>> public long getDiskSize();
>>>>
>>>> public long getPagesInMemory();
>>>>
>>>> public long getPagesSizeInMemory();
>>>>
>>>> public long getPagesOnDisk();
>>>>
>>>> public long getPagesSizeOnDisk();
>>>>
>>>> public long getFreePages();
>>>>
>>>> public long getFreePagesSize();
>>>>
>>>> public long getDirtyPages();
>>>>
>>>> public long getDirtyPagesSize();
>>>>
>>>> public long getPagesRead();
>>>>
>>>> public long getPagesWritten();
>>>> }
>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>> PersistentStoreCheckpointMetrics.java
>>>>
>>>> public interface PersistentStoreCheckpointMetrics {
>>>>
>>>> public long getTotalPages();
>>>>
>>>> //TODO Page type is internal?
>>>> public long[] pagesType();
>>>>
>>>> public long getExecutingTime();
>>>>
>>>> public long getFsyncTime();
>>>>
>>>> public long getPagesCopyOnWrite();
>>>> }
>>>
>>>
>>
Re: Persistent Distributed Store Metrics
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Several questions:
1. What is "getWalFsyncAverage()"? Is it time? If yes, then it should be
consistent with other time metrics, and should be called
"getWalFsyncAverageTime()"
2. I see "getPagesOnDisk()", should we also have "getPagesInMemory()"?
3. Also, how about WAL total pages count or total size, whichever one is
easier?
4. How about the WAL size that has not been checkpointed yet?
5. How about WAL history size (the number of WAL files that we are
rotating through)?
Also, I see your argument about separating memory from disk metrics, but I
think it makes sense to combine them, because it is hard to separate them
from user standpoint. Let's call them IgniteStorageMetrics or something of
the kind.
D.
On Mon, May 29, 2017 at 9:24 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Hello Denis,
>
> I have an idea of how we could clarify proposed above approach to
> collecting metrics for PDS.
>
> I suggest to split all metrics into two major groups (disk-related and
> memory-related) and group them in the following interfaces:
>
> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> public double getPageEvictionRate();
>
> public double getPageMissRate();
>
> public long getDirtyPages();
>
> public long getPagesOnDisk();
> }
>
> public interface PersistenceStoreMetrics {
> //WAL-related metrics
> public double getWalLoggingRate();
>
> public int getWalArchiveSegments();
>
> public double getWalFsyncAverage();
>
> //checkpoint-related metrics
> public double getCpTime();
>
> public double getCpFsyncTime();
>
> public long getCpTotalPages();
>
> public long[] getCpPagesByType();
>
> public long getCpCopiedOnWritePages();
> }
>
> From my standpoint it is important to distinguish between PDS metrics that
> are related mostly to memory management (from the first interface) and
> metrics bound to disk operations (from the second interface).
>
> First group of metrics is useful when collected per Memory Policy. It
> doesn't provide a lot of information when collected across many Memory
> Policies.
>
> Second group is now global (this may change in the future if Persistence
> Policies are introduced) and allows to easily spot any high-level issues
> with durable storage configuration or performance. More specific questions
> like "how this Memory Policy loads a durable storage subsystem?" may be
> deduced from PersistenceMemoryMetrics indicators.
>
> Denis, does it make sense for you?
>
> Any other ideas/questions/improvements or metrics to add to these
> interfaces?
>
> I would like to discuss all the details here before creating any tickets
> for the change.
>
> Thanks,
> Sergey.
>
>
> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dm...@apache.org> wrote:
>
> > Dmitriy,
> >
> > I have some high level concerns.
> >
> > Presently, Ignite already exposes memory/data related metrics via
> > MemoryMetrics and CacheMetrics interface.
> >
> > Looking at PersistentStoreMetrics interface I can’t get why methods like
> > getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT
> > in MemoryMetrics interface.
> >
> > Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
> > contains a big subset of the methods from PersistentStoreMetric. There
> just
> > should be some API call that either returns global metrics or cache
> > specific ones. Plus, some of the methods might be added to existing
> > CacheInterface. Why not.
> >
> > I prefer to address this first. Guys, any other thoughts on this?
> >
> > —
> > Denis
> >
> > > On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
> > dmitriy.govorukhin@gmail.com> wrote:
> > >
> > > Folk,
> > >
> > > As you know, ignite 2.1 will contain new module (pds), it will be
> > > provide ability to store data on disk. Let's discuss what type of
> > > metrics we need for this?
> > > I think it must be metrics per memory policy, per cache, checkpoint,
> > > and global metrics which will be aggregate all metrics.
> > >
> > > I did sketch.
> > >
> > > PersistentStoreMetrics.java
> > >
> > > public interface PersistentStoreMetrics {
> > >
> > > // Global metrics.
> > >
> > > public long getMemorySize();
> > >
> > > public long getDiskSize();
> > >
> > > public long getPagesInMemory();
> > >
> > > public long getPagesSizeInMemory();
> > >
> > > public long getPagesOnDisk();
> > >
> > > public long getPagesSizeOnDisk();
> > >
> > > public long getFreePages();
> > >
> > > public long getFreePagesSize();
> > >
> > > public long getDirtyPages();
> > >
> > > public long getDirtyPagesSize();
> > >
> > > public long walLog();
> > >
> > > public long walLogSize();
> > >
> > > // Frequency.
> > >
> > > public long getPagesRead();
> > >
> > > public long getPagesWrite();
> > >
> > > public long getFsync();
> > >
> > > public long getWal();
> > >
> > > public long getAverageWalFsyncTime();
> > >
> > > // Per cache.
> > >
> > > public PersistentStoreCacheMetrics cache(String name);
> > >
> > > public PersistentStoreCacheMetrics cache(int cacheId);
> > >
> > > // For last checkpoint.
> > >
> > > public PersistentStoreCheckpointMetrics getLastCheckPoint();
> > > }
> > >
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >
> > > PersistentStoreCacheMetrics.java
> > >
> > > public interface PersistentStoreCacheMetrics {
> > >
> > > public String name();
> > >
> > > public double getFillFactor();
> > >
> > > public double getFillFactor(int part);
> > >
> > > public long getMemorySize();
> > >
> > > public long getDiskSize();
> > >
> > > public long getPagesInMemory();
> > >
> > > public long getPagesSizeInMemory();
> > >
> > > public long getPagesOnDisk();
> > >
> > > public long getPagesSizeOnDisk();
> > >
> > > public long getFreePages();
> > >
> > > public long getFreePagesSize();
> > >
> > > public long getDirtyPages();
> > >
> > > public long getDirtyPagesSize();
> > >
> > > public long getPagesRead();
> > >
> > > public long getPagesWritten();
> > > }
> > >
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >
> > > PersistentStoreCheckpointMetrics.java
> > >
> > > public interface PersistentStoreCheckpointMetrics {
> > >
> > > public long getTotalPages();
> > >
> > > //TODO Page type is internal?
> > > public long[] pagesType();
> > >
> > > public long getExecutingTime();
> > >
> > > public long getFsyncTime();
> > >
> > > public long getPagesCopyOnWrite();
> > > }
> >
> >
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
Sergey, thanks, now it looks more vivid.
> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> public double getPageEvictionRate();
>
> public double getPageMissRate();
>
> public long getDirtyPages();
>
> public long getPagesOnDisk();
> }
What if we just add these 4 methods directly to MemoryMetrics interface? Otherwise, we’ll have hard times describing the difference between all the variety of metrics interfaces we have.
If the store is turned off the methods can throw a meaningful IllegalStateException or return a negative value. I’m voting for the exception.
> public interface PersistenceStoreMetrics {
> //WAL-related metrics
> public double getWalLoggingRate();
>
> public int getWalArchiveSegments();
>
> public double getWalFsyncAverage();
>
> //checkpoint-related metrics
> public double getCpTime();
>
> public double getCpFsyncTime();
>
> public long getCpTotalPages();
>
> public long[] getCpPagesByType();
>
> public long getCpCopiedOnWritePages();
> }
I’m fine with these interface. Please do minor changes:
- rename Persistence to Persistent so that the final name is “PersistentStoreMetrics”.
- I wouldn’t shorten the checkpointing related methods. Let’s write “getCheckpointing…”
—
Denis
> On May 29, 2017, at 9:24 AM, Sergey Chugunov <se...@gmail.com> wrote:
>
> Hello Denis,
>
> I have an idea of how we could clarify proposed above approach to
> collecting metrics for PDS.
>
> I suggest to split all metrics into two major groups (disk-related and
> memory-related) and group them in the following interfaces:
>
> public interface PersistenceMemoryMetrics extends MemoryMetrics {
> public double getPageEvictionRate();
>
> public double getPageMissRate();
>
> public long getDirtyPages();
>
> public long getPagesOnDisk();
> }
>
> public interface PersistenceStoreMetrics {
> //WAL-related metrics
> public double getWalLoggingRate();
>
> public int getWalArchiveSegments();
>
> public double getWalFsyncAverage();
>
> //checkpoint-related metrics
> public double getCpTime();
>
> public double getCpFsyncTime();
>
> public long getCpTotalPages();
>
> public long[] getCpPagesByType();
>
> public long getCpCopiedOnWritePages();
> }
>
> From my standpoint it is important to distinguish between PDS metrics that
> are related mostly to memory management (from the first interface) and
> metrics bound to disk operations (from the second interface).
>
> First group of metrics is useful when collected per Memory Policy. It
> doesn't provide a lot of information when collected across many Memory
> Policies.
>
> Second group is now global (this may change in the future if Persistence
> Policies are introduced) and allows to easily spot any high-level issues
> with durable storage configuration or performance. More specific questions
> like "how this Memory Policy loads a durable storage subsystem?" may be
> deduced from PersistenceMemoryMetrics indicators.
>
> Denis, does it make sense for you?
>
> Any other ideas/questions/improvements or metrics to add to these
> interfaces?
>
> I would like to discuss all the details here before creating any tickets
> for the change.
>
> Thanks,
> Sergey.
>
>
> On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Dmitriy,
>>
>> I have some high level concerns.
>>
>> Presently, Ignite already exposes memory/data related metrics via
>> MemoryMetrics and CacheMetrics interface.
>>
>> Looking at PersistentStoreMetrics interface I can’t get why methods like
>> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT
>> in MemoryMetrics interface.
>>
>> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
>> contains a big subset of the methods from PersistentStoreMetric. There just
>> should be some API call that either returns global metrics or cache
>> specific ones. Plus, some of the methods might be added to existing
>> CacheInterface. Why not.
>>
>> I prefer to address this first. Guys, any other thoughts on this?
>>
>> —
>> Denis
>>
>>> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
>> dmitriy.govorukhin@gmail.com> wrote:
>>>
>>> Folk,
>>>
>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>> provide ability to store data on disk. Let's discuss what type of
>>> metrics we need for this?
>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>> and global metrics which will be aggregate all metrics.
>>>
>>> I did sketch.
>>>
>>> PersistentStoreMetrics.java
>>>
>>> public interface PersistentStoreMetrics {
>>>
>>> // Global metrics.
>>>
>>> public long getMemorySize();
>>>
>>> public long getDiskSize();
>>>
>>> public long getPagesInMemory();
>>>
>>> public long getPagesSizeInMemory();
>>>
>>> public long getPagesOnDisk();
>>>
>>> public long getPagesSizeOnDisk();
>>>
>>> public long getFreePages();
>>>
>>> public long getFreePagesSize();
>>>
>>> public long getDirtyPages();
>>>
>>> public long getDirtyPagesSize();
>>>
>>> public long walLog();
>>>
>>> public long walLogSize();
>>>
>>> // Frequency.
>>>
>>> public long getPagesRead();
>>>
>>> public long getPagesWrite();
>>>
>>> public long getFsync();
>>>
>>> public long getWal();
>>>
>>> public long getAverageWalFsyncTime();
>>>
>>> // Per cache.
>>>
>>> public PersistentStoreCacheMetrics cache(String name);
>>>
>>> public PersistentStoreCacheMetrics cache(int cacheId);
>>>
>>> // For last checkpoint.
>>>
>>> public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>> }
>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>
>>> PersistentStoreCacheMetrics.java
>>>
>>> public interface PersistentStoreCacheMetrics {
>>>
>>> public String name();
>>>
>>> public double getFillFactor();
>>>
>>> public double getFillFactor(int part);
>>>
>>> public long getMemorySize();
>>>
>>> public long getDiskSize();
>>>
>>> public long getPagesInMemory();
>>>
>>> public long getPagesSizeInMemory();
>>>
>>> public long getPagesOnDisk();
>>>
>>> public long getPagesSizeOnDisk();
>>>
>>> public long getFreePages();
>>>
>>> public long getFreePagesSize();
>>>
>>> public long getDirtyPages();
>>>
>>> public long getDirtyPagesSize();
>>>
>>> public long getPagesRead();
>>>
>>> public long getPagesWritten();
>>> }
>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>
>>> PersistentStoreCheckpointMetrics.java
>>>
>>> public interface PersistentStoreCheckpointMetrics {
>>>
>>> public long getTotalPages();
>>>
>>> //TODO Page type is internal?
>>> public long[] pagesType();
>>>
>>> public long getExecutingTime();
>>>
>>> public long getFsyncTime();
>>>
>>> public long getPagesCopyOnWrite();
>>> }
>>
>>
Re: Persistent Distributed Store Metrics
Posted by Sergey Chugunov <se...@gmail.com>.
Hello Denis,
I have an idea of how we could clarify proposed above approach to
collecting metrics for PDS.
I suggest to split all metrics into two major groups (disk-related and
memory-related) and group them in the following interfaces:
public interface PersistenceMemoryMetrics extends MemoryMetrics {
public double getPageEvictionRate();
public double getPageMissRate();
public long getDirtyPages();
public long getPagesOnDisk();
}
public interface PersistenceStoreMetrics {
//WAL-related metrics
public double getWalLoggingRate();
public int getWalArchiveSegments();
public double getWalFsyncAverage();
//checkpoint-related metrics
public double getCpTime();
public double getCpFsyncTime();
public long getCpTotalPages();
public long[] getCpPagesByType();
public long getCpCopiedOnWritePages();
}
From my standpoint it is important to distinguish between PDS metrics that
are related mostly to memory management (from the first interface) and
metrics bound to disk operations (from the second interface).
First group of metrics is useful when collected per Memory Policy. It
doesn't provide a lot of information when collected across many Memory
Policies.
Second group is now global (this may change in the future if Persistence
Policies are introduced) and allows to easily spot any high-level issues
with durable storage configuration or performance. More specific questions
like "how this Memory Policy loads a durable storage subsystem?" may be
deduced from PersistenceMemoryMetrics indicators.
Denis, does it make sense for you?
Any other ideas/questions/improvements or metrics to add to these
interfaces?
I would like to discuss all the details here before creating any tickets
for the change.
Thanks,
Sergey.
On Fri, May 19, 2017 at 9:26 PM, Denis Magda <dm...@apache.org> wrote:
> Dmitriy,
>
> I have some high level concerns.
>
> Presently, Ignite already exposes memory/data related metrics via
> MemoryMetrics and CacheMetrics interface.
>
> Looking at PersistentStoreMetrics interface I can’t get why methods like
> getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT
> in MemoryMetrics interface.
>
> Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that
> contains a big subset of the methods from PersistentStoreMetric. There just
> should be some API call that either returns global metrics or cache
> specific ones. Plus, some of the methods might be added to existing
> CacheInterface. Why not.
>
> I prefer to address this first. Guys, any other thoughts on this?
>
> —
> Denis
>
> > On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
> >
> > Folk,
> >
> > As you know, ignite 2.1 will contain new module (pds), it will be
> > provide ability to store data on disk. Let's discuss what type of
> > metrics we need for this?
> > I think it must be metrics per memory policy, per cache, checkpoint,
> > and global metrics which will be aggregate all metrics.
> >
> > I did sketch.
> >
> > PersistentStoreMetrics.java
> >
> > public interface PersistentStoreMetrics {
> >
> > // Global metrics.
> >
> > public long getMemorySize();
> >
> > public long getDiskSize();
> >
> > public long getPagesInMemory();
> >
> > public long getPagesSizeInMemory();
> >
> > public long getPagesOnDisk();
> >
> > public long getPagesSizeOnDisk();
> >
> > public long getFreePages();
> >
> > public long getFreePagesSize();
> >
> > public long getDirtyPages();
> >
> > public long getDirtyPagesSize();
> >
> > public long walLog();
> >
> > public long walLogSize();
> >
> > // Frequency.
> >
> > public long getPagesRead();
> >
> > public long getPagesWrite();
> >
> > public long getFsync();
> >
> > public long getWal();
> >
> > public long getAverageWalFsyncTime();
> >
> > // Per cache.
> >
> > public PersistentStoreCacheMetrics cache(String name);
> >
> > public PersistentStoreCacheMetrics cache(int cacheId);
> >
> > // For last checkpoint.
> >
> > public PersistentStoreCheckpointMetrics getLastCheckPoint();
> > }
> >
> >>>>>>>>>>>>>>>>>>>>>>>>
> >
> > PersistentStoreCacheMetrics.java
> >
> > public interface PersistentStoreCacheMetrics {
> >
> > public String name();
> >
> > public double getFillFactor();
> >
> > public double getFillFactor(int part);
> >
> > public long getMemorySize();
> >
> > public long getDiskSize();
> >
> > public long getPagesInMemory();
> >
> > public long getPagesSizeInMemory();
> >
> > public long getPagesOnDisk();
> >
> > public long getPagesSizeOnDisk();
> >
> > public long getFreePages();
> >
> > public long getFreePagesSize();
> >
> > public long getDirtyPages();
> >
> > public long getDirtyPagesSize();
> >
> > public long getPagesRead();
> >
> > public long getPagesWritten();
> > }
> >
> >>>>>>>>>>>>>>>>>>>>>>>>
> >
> > PersistentStoreCheckpointMetrics.java
> >
> > public interface PersistentStoreCheckpointMetrics {
> >
> > public long getTotalPages();
> >
> > //TODO Page type is internal?
> > public long[] pagesType();
> >
> > public long getExecutingTime();
> >
> > public long getFsyncTime();
> >
> > public long getPagesCopyOnWrite();
> > }
>
>
Re: Persistent Distributed Store Metrics
Posted by Denis Magda <dm...@apache.org>.
Dmitriy,
I have some high level concerns.
Presently, Ignite already exposes memory/data related metrics via MemoryMetrics and CacheMetrics interface.
Looking at PersistentStoreMetrics interface I can’t get why methods like getMemorySize(), getPagesInMemory(), getPagesSizeInMemory(), etc. are NOT in MemoryMetrics interface.
Next, I wouldn’t make out PersistentStore*Cache*Metrics interface that contains a big subset of the methods from PersistentStoreMetric. There just should be some API call that either returns global metrics or cache specific ones. Plus, some of the methods might be added to existing CacheInterface. Why not.
I prefer to address this first. Guys, any other thoughts on this?
—
Denis
> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin <dm...@gmail.com> wrote:
>
> Folk,
>
> As you know, ignite 2.1 will contain new module (pds), it will be
> provide ability to store data on disk. Let's discuss what type of
> metrics we need for this?
> I think it must be metrics per memory policy, per cache, checkpoint,
> and global metrics which will be aggregate all metrics.
>
> I did sketch.
>
> PersistentStoreMetrics.java
>
> public interface PersistentStoreMetrics {
>
> // Global metrics.
>
> public long getMemorySize();
>
> public long getDiskSize();
>
> public long getPagesInMemory();
>
> public long getPagesSizeInMemory();
>
> public long getPagesOnDisk();
>
> public long getPagesSizeOnDisk();
>
> public long getFreePages();
>
> public long getFreePagesSize();
>
> public long getDirtyPages();
>
> public long getDirtyPagesSize();
>
> public long walLog();
>
> public long walLogSize();
>
> // Frequency.
>
> public long getPagesRead();
>
> public long getPagesWrite();
>
> public long getFsync();
>
> public long getWal();
>
> public long getAverageWalFsyncTime();
>
> // Per cache.
>
> public PersistentStoreCacheMetrics cache(String name);
>
> public PersistentStoreCacheMetrics cache(int cacheId);
>
> // For last checkpoint.
>
> public PersistentStoreCheckpointMetrics getLastCheckPoint();
> }
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
> PersistentStoreCacheMetrics.java
>
> public interface PersistentStoreCacheMetrics {
>
> public String name();
>
> public double getFillFactor();
>
> public double getFillFactor(int part);
>
> public long getMemorySize();
>
> public long getDiskSize();
>
> public long getPagesInMemory();
>
> public long getPagesSizeInMemory();
>
> public long getPagesOnDisk();
>
> public long getPagesSizeOnDisk();
>
> public long getFreePages();
>
> public long getFreePagesSize();
>
> public long getDirtyPages();
>
> public long getDirtyPagesSize();
>
> public long getPagesRead();
>
> public long getPagesWritten();
> }
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
> PersistentStoreCheckpointMetrics.java
>
> public interface PersistentStoreCheckpointMetrics {
>
> public long getTotalPages();
>
> //TODO Page type is internal?
> public long[] pagesType();
>
> public long getExecutingTime();
>
> public long getFsyncTime();
>
> public long getPagesCopyOnWrite();
> }