You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Dmitriy Govorukhin <dm...@gmail.com> on 2018/05/15 15:37:16 UTC
work with files and directories
Hi Ignite,
Do we have a general approach to work with a file and directories?
I see many duplication logic for write through .tmp file.
For example,
GridCacheDatabaseSharedManager.writeCheckpointEntry();
GridCacheDatabaseSharedManage.nodeStart();
FileWriteAheadLogManager.FileArchiver.archiveSegment();
All of these methods implement the same logic, write to tmp file and rename
to normal name.
I guess, will be better if we stopping write duplication logic code and
start to consolidate all in one place.
Also, I think that current approach to creating files is not quite right
faithful. Each internal component
create/delete files inside himself, and nobody knows where which files
located.
I suggest refactoring code and create something (maybe new manager) that
will know about all files inside the node. All internal components must
create files only through this one.
It makes help to write tests for persistence easier and reduce duplication
code in working with files.
Re: work with files and directories
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
One more thing, If we implement this component and will aggregate
information about all files, we can
create failure handler which provided detail information about node
persistence structure. It will be helpful for debugging node crash.
On Wed, May 16, 2018 at 12:46 PM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> Andrey Gura expressed in words what I also thought.
>
> I agree if we extract common code to some kind of file utils.
>
> In the same time creating common file operation framework is not possible
> because of different nature of WAL, Page Store and other components using
> files.
>
> ср, 16 мая 2018 г. в 11:27, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com
> >:
>
> > Andrey,
> >
> > My point is, it will be very cool if there is some component who will
> know
> > about persistence folder and files,
> > and we can move all generic code, for work with files, to this
> component.
> >
> > On Wed, May 16, 2018 at 12:59 AM, Pavel Kovalenko <jo...@gmail.com>
> > wrote:
> >
> > > Andrey,
> > >
> > > I think the main idea of that not about how to unify writing content to
> > > file. It's about creating peristence files and folders. For other
> > > persistence components (WAL, Checkpoint, etc.) it should be like
> > > FileFactory object with methods like "public File
> > > getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So,
> > all
> > > logic about how files should look will be moved to centralized point
> and
> > it
> > > will be more easier to understand our persistence structure.
> > >
> > > 2018-05-16 0:42 GMT+03:00 Andrey Gura <ag...@apache.org>:
> > >
> > > > Hi,
> > > >
> > > > I understand you idea but it just increases dependencies of different
> > > > component from one that is in general bad practice.
> > > >
> > > > We have different components where each one can use different
> approach
> > > for
> > > > file management. For example page store and WAL have different file
> IO
> > > > implementations due to performance reasons and we have to provide
> > > different
> > > > mechanics for work with files.
> > > >
> > > > Of course we can refactor mentioned components in more structured
> > manner
> > > > but we should not strongly link it with one implementation.
> > > >
> > > > вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dp...@gmail.com>:
> > > >
> > > > > Hi Maxim,
> > > > >
> > > > > FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager
> were
> > > > > intentionally copy-pasted in hope we will soon delete FsyncManager.
> > > > >
> > > > > But it is still shows this tool works good. Probably we could
> > integrate
> > > > > this tool to our processes.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > >
> > > > > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <ma...@gmail.com>:
> > > > >
> > > > > > +1 also for something like "resource manager".
> > > > > >
> > > > > > Recently, I've found for myself sonarcloud.io tool for code
> > > analisys.
> > > > > It's
> > > > > > free for open source project and I've made Ignite project initial
> > run
> > > > > [1].
> > > > > >
> > > > > > I've prepeared analisys for mysefl and found a lot of duplicated
> > code
> > > > > > blocks [1]. Of course it's not the ideal tool but gave us
> direction
> > > of
> > > > > > thoughts. E.g. these classes [3]:
> > > > > > 1) FileWriteAheadLogManager.java
> > > > > > 2) FsyncModeFileWriteAheadLogManager.java
> > > > > >
> > > > > >
> > > > > > [1] https://sonarcloud.io/dashboard?id=org.apache.
> > > > ignite%3Aapache-ignite
> > > > > > [2]
> > > > > >
> > > > > >
> > > > > https://sonarcloud.io/component_measures?id=org.
> > > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > > > > > [3]
> > > > > >
> > > > > >
> > > > > https://sonarcloud.io/component_measures?id=org.
> > > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks&
> > > > selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%
> > > > 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%
> > > > 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> > > > > >
> > > > > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jokserfn@gmail.com
> >:
> > > > > >
> > > > > > > +1 to this approach,
> > > > > > >
> > > > > > > It can be also very helpful in failover scenarios when
> something
> > > > wrong
> > > > > > > happened with disk. In this case we're reducing the number of
> > > points
> > > > of
> > > > > > > failure.
> > > > > > >
> > > > > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > > > > > dmitriy.govorukhin@gmail.com>
> > > > > > > :
> > > > > > >
> > > > > > > > Hi Ignite,
> > > > > > > >
> > > > > > > > Do we have a general approach to work with a file and
> > > directories?
> > > > > > > > I see many duplication logic for write through .tmp file.
> > > > > > > >
> > > > > > > > For example,
> > > > > > > >
> > > > > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > > > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > > > > > >
> > > > > > > > All of these methods implement the same logic, write to tmp
> > file
> > > > and
> > > > > > > rename
> > > > > > > > to normal name.
> > > > > > > >
> > > > > > > > I guess, will be better if we stopping write duplication
> logic
> > > code
> > > > > and
> > > > > > > > start to consolidate all in one place.
> > > > > > > >
> > > > > > > > Also, I think that current approach to creating files is not
> > > quite
> > > > > > right
> > > > > > > > faithful. Each internal component
> > > > > > > > create/delete files inside himself, and nobody knows where
> > which
> > > > > files
> > > > > > > > located.
> > > > > > > >
> > > > > > > > I suggest refactoring code and create something (maybe new
> > > manager)
> > > > > > that
> > > > > > > > will know about all files inside the node. All internal
> > > components
> > > > > must
> > > > > > > > create files only through this one.
> > > > > > > >
> > > > > > > > It makes help to write tests for persistence easier and
> reduce
> > > > > > > duplication
> > > > > > > > code in working with files.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: work with files and directories
Posted by Dmitry Pavlov <dp...@gmail.com>.
Andrey Gura expressed in words what I also thought.
I agree if we extract common code to some kind of file utils.
In the same time creating common file operation framework is not possible
because of different nature of WAL, Page Store and other components using
files.
ср, 16 мая 2018 г. в 11:27, Dmitriy Govorukhin <dmitriy.govorukhin@gmail.com
>:
> Andrey,
>
> My point is, it will be very cool if there is some component who will know
> about persistence folder and files,
> and we can move all generic code, for work with files, to this component.
>
> On Wed, May 16, 2018 at 12:59 AM, Pavel Kovalenko <jo...@gmail.com>
> wrote:
>
> > Andrey,
> >
> > I think the main idea of that not about how to unify writing content to
> > file. It's about creating peristence files and folders. For other
> > persistence components (WAL, Checkpoint, etc.) it should be like
> > FileFactory object with methods like "public File
> > getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So,
> all
> > logic about how files should look will be moved to centralized point and
> it
> > will be more easier to understand our persistence structure.
> >
> > 2018-05-16 0:42 GMT+03:00 Andrey Gura <ag...@apache.org>:
> >
> > > Hi,
> > >
> > > I understand you idea but it just increases dependencies of different
> > > component from one that is in general bad practice.
> > >
> > > We have different components where each one can use different approach
> > for
> > > file management. For example page store and WAL have different file IO
> > > implementations due to performance reasons and we have to provide
> > different
> > > mechanics for work with files.
> > >
> > > Of course we can refactor mentioned components in more structured
> manner
> > > but we should not strongly link it with one implementation.
> > >
> > > вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dp...@gmail.com>:
> > >
> > > > Hi Maxim,
> > > >
> > > > FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager were
> > > > intentionally copy-pasted in hope we will soon delete FsyncManager.
> > > >
> > > > But it is still shows this tool works good. Probably we could
> integrate
> > > > this tool to our processes.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > >
> > > > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <ma...@gmail.com>:
> > > >
> > > > > +1 also for something like "resource manager".
> > > > >
> > > > > Recently, I've found for myself sonarcloud.io tool for code
> > analisys.
> > > > It's
> > > > > free for open source project and I've made Ignite project initial
> run
> > > > [1].
> > > > >
> > > > > I've prepeared analisys for mysefl and found a lot of duplicated
> code
> > > > > blocks [1]. Of course it's not the ideal tool but gave us direction
> > of
> > > > > thoughts. E.g. these classes [3]:
> > > > > 1) FileWriteAheadLogManager.java
> > > > > 2) FsyncModeFileWriteAheadLogManager.java
> > > > >
> > > > >
> > > > > [1] https://sonarcloud.io/dashboard?id=org.apache.
> > > ignite%3Aapache-ignite
> > > > > [2]
> > > > >
> > > > >
> > > > https://sonarcloud.io/component_measures?id=org.
> > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > > > > [3]
> > > > >
> > > > >
> > > > https://sonarcloud.io/component_measures?id=org.
> > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks&
> > > selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%
> > > 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%
> > > 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> > > > >
> > > > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jo...@gmail.com>:
> > > > >
> > > > > > +1 to this approach,
> > > > > >
> > > > > > It can be also very helpful in failover scenarios when something
> > > wrong
> > > > > > happened with disk. In this case we're reducing the number of
> > points
> > > of
> > > > > > failure.
> > > > > >
> > > > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > > > > dmitriy.govorukhin@gmail.com>
> > > > > > :
> > > > > >
> > > > > > > Hi Ignite,
> > > > > > >
> > > > > > > Do we have a general approach to work with a file and
> > directories?
> > > > > > > I see many duplication logic for write through .tmp file.
> > > > > > >
> > > > > > > For example,
> > > > > > >
> > > > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > > > > >
> > > > > > > All of these methods implement the same logic, write to tmp
> file
> > > and
> > > > > > rename
> > > > > > > to normal name.
> > > > > > >
> > > > > > > I guess, will be better if we stopping write duplication logic
> > code
> > > > and
> > > > > > > start to consolidate all in one place.
> > > > > > >
> > > > > > > Also, I think that current approach to creating files is not
> > quite
> > > > > right
> > > > > > > faithful. Each internal component
> > > > > > > create/delete files inside himself, and nobody knows where
> which
> > > > files
> > > > > > > located.
> > > > > > >
> > > > > > > I suggest refactoring code and create something (maybe new
> > manager)
> > > > > that
> > > > > > > will know about all files inside the node. All internal
> > components
> > > > must
> > > > > > > create files only through this one.
> > > > > > >
> > > > > > > It makes help to write tests for persistence easier and reduce
> > > > > > duplication
> > > > > > > code in working with files.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: work with files and directories
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Andrey,
My point is, it will be very cool if there is some component who will know
about persistence folder and files,
and we can move all generic code, for work with files, to this component.
On Wed, May 16, 2018 at 12:59 AM, Pavel Kovalenko <jo...@gmail.com>
wrote:
> Andrey,
>
> I think the main idea of that not about how to unify writing content to
> file. It's about creating peristence files and folders. For other
> persistence components (WAL, Checkpoint, etc.) it should be like
> FileFactory object with methods like "public File
> getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So, all
> logic about how files should look will be moved to centralized point and it
> will be more easier to understand our persistence structure.
>
> 2018-05-16 0:42 GMT+03:00 Andrey Gura <ag...@apache.org>:
>
> > Hi,
> >
> > I understand you idea but it just increases dependencies of different
> > component from one that is in general bad practice.
> >
> > We have different components where each one can use different approach
> for
> > file management. For example page store and WAL have different file IO
> > implementations due to performance reasons and we have to provide
> different
> > mechanics for work with files.
> >
> > Of course we can refactor mentioned components in more structured manner
> > but we should not strongly link it with one implementation.
> >
> > вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dp...@gmail.com>:
> >
> > > Hi Maxim,
> > >
> > > FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager were
> > > intentionally copy-pasted in hope we will soon delete FsyncManager.
> > >
> > > But it is still shows this tool works good. Probably we could integrate
> > > this tool to our processes.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > >
> > > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <ma...@gmail.com>:
> > >
> > > > +1 also for something like "resource manager".
> > > >
> > > > Recently, I've found for myself sonarcloud.io tool for code
> analisys.
> > > It's
> > > > free for open source project and I've made Ignite project initial run
> > > [1].
> > > >
> > > > I've prepeared analisys for mysefl and found a lot of duplicated code
> > > > blocks [1]. Of course it's not the ideal tool but gave us direction
> of
> > > > thoughts. E.g. these classes [3]:
> > > > 1) FileWriteAheadLogManager.java
> > > > 2) FsyncModeFileWriteAheadLogManager.java
> > > >
> > > >
> > > > [1] https://sonarcloud.io/dashboard?id=org.apache.
> > ignite%3Aapache-ignite
> > > > [2]
> > > >
> > > >
> > > https://sonarcloud.io/component_measures?id=org.
> > apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > > > [3]
> > > >
> > > >
> > > https://sonarcloud.io/component_measures?id=org.
> > apache.ignite%3Aapache-ignite&metric=duplicated_blocks&
> > selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%
> > 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%
> > 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> > > >
> > > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jo...@gmail.com>:
> > > >
> > > > > +1 to this approach,
> > > > >
> > > > > It can be also very helpful in failover scenarios when something
> > wrong
> > > > > happened with disk. In this case we're reducing the number of
> points
> > of
> > > > > failure.
> > > > >
> > > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > > > dmitriy.govorukhin@gmail.com>
> > > > > :
> > > > >
> > > > > > Hi Ignite,
> > > > > >
> > > > > > Do we have a general approach to work with a file and
> directories?
> > > > > > I see many duplication logic for write through .tmp file.
> > > > > >
> > > > > > For example,
> > > > > >
> > > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > > > >
> > > > > > All of these methods implement the same logic, write to tmp file
> > and
> > > > > rename
> > > > > > to normal name.
> > > > > >
> > > > > > I guess, will be better if we stopping write duplication logic
> code
> > > and
> > > > > > start to consolidate all in one place.
> > > > > >
> > > > > > Also, I think that current approach to creating files is not
> quite
> > > > right
> > > > > > faithful. Each internal component
> > > > > > create/delete files inside himself, and nobody knows where which
> > > files
> > > > > > located.
> > > > > >
> > > > > > I suggest refactoring code and create something (maybe new
> manager)
> > > > that
> > > > > > will know about all files inside the node. All internal
> components
> > > must
> > > > > > create files only through this one.
> > > > > >
> > > > > > It makes help to write tests for persistence easier and reduce
> > > > > duplication
> > > > > > code in working with files.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: work with files and directories
Posted by Pavel Kovalenko <jo...@gmail.com>.
Andrey,
I think the main idea of that not about how to unify writing content to
file. It's about creating peristence files and folders. For other
persistence components (WAL, Checkpoint, etc.) it should be like
FileFactory object with methods like "public File
getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So, all
logic about how files should look will be moved to centralized point and it
will be more easier to understand our persistence structure.
2018-05-16 0:42 GMT+03:00 Andrey Gura <ag...@apache.org>:
> Hi,
>
> I understand you idea but it just increases dependencies of different
> component from one that is in general bad practice.
>
> We have different components where each one can use different approach for
> file management. For example page store and WAL have different file IO
> implementations due to performance reasons and we have to provide different
> mechanics for work with files.
>
> Of course we can refactor mentioned components in more structured manner
> but we should not strongly link it with one implementation.
>
> вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dp...@gmail.com>:
>
> > Hi Maxim,
> >
> > FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager were
> > intentionally copy-pasted in hope we will soon delete FsyncManager.
> >
> > But it is still shows this tool works good. Probably we could integrate
> > this tool to our processes.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <ma...@gmail.com>:
> >
> > > +1 also for something like "resource manager".
> > >
> > > Recently, I've found for myself sonarcloud.io tool for code analisys.
> > It's
> > > free for open source project and I've made Ignite project initial run
> > [1].
> > >
> > > I've prepeared analisys for mysefl and found a lot of duplicated code
> > > blocks [1]. Of course it's not the ideal tool but gave us direction of
> > > thoughts. E.g. these classes [3]:
> > > 1) FileWriteAheadLogManager.java
> > > 2) FsyncModeFileWriteAheadLogManager.java
> > >
> > >
> > > [1] https://sonarcloud.io/dashboard?id=org.apache.
> ignite%3Aapache-ignite
> > > [2]
> > >
> > >
> > https://sonarcloud.io/component_measures?id=org.
> apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > > [3]
> > >
> > >
> > https://sonarcloud.io/component_measures?id=org.
> apache.ignite%3Aapache-ignite&metric=duplicated_blocks&
> selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%
> 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%
> 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> > >
> > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jo...@gmail.com>:
> > >
> > > > +1 to this approach,
> > > >
> > > > It can be also very helpful in failover scenarios when something
> wrong
> > > > happened with disk. In this case we're reducing the number of points
> of
> > > > failure.
> > > >
> > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > > dmitriy.govorukhin@gmail.com>
> > > > :
> > > >
> > > > > Hi Ignite,
> > > > >
> > > > > Do we have a general approach to work with a file and directories?
> > > > > I see many duplication logic for write through .tmp file.
> > > > >
> > > > > For example,
> > > > >
> > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > > >
> > > > > All of these methods implement the same logic, write to tmp file
> and
> > > > rename
> > > > > to normal name.
> > > > >
> > > > > I guess, will be better if we stopping write duplication logic code
> > and
> > > > > start to consolidate all in one place.
> > > > >
> > > > > Also, I think that current approach to creating files is not quite
> > > right
> > > > > faithful. Each internal component
> > > > > create/delete files inside himself, and nobody knows where which
> > files
> > > > > located.
> > > > >
> > > > > I suggest refactoring code and create something (maybe new manager)
> > > that
> > > > > will know about all files inside the node. All internal components
> > must
> > > > > create files only through this one.
> > > > >
> > > > > It makes help to write tests for persistence easier and reduce
> > > > duplication
> > > > > code in working with files.
> > > > >
> > > >
> > >
> >
>
Re: work with files and directories
Posted by Andrey Gura <ag...@apache.org>.
Hi,
I understand you idea but it just increases dependencies of different
component from one that is in general bad practice.
We have different components where each one can use different approach for
file management. For example page store and WAL have different file IO
implementations due to performance reasons and we have to provide different
mechanics for work with files.
Of course we can refactor mentioned components in more structured manner
but we should not strongly link it with one implementation.
вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dp...@gmail.com>:
> Hi Maxim,
>
> FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager were
> intentionally copy-pasted in hope we will soon delete FsyncManager.
>
> But it is still shows this tool works good. Probably we could integrate
> this tool to our processes.
>
> Sincerely,
> Dmitriy Pavlov
>
>
> вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <ma...@gmail.com>:
>
> > +1 also for something like "resource manager".
> >
> > Recently, I've found for myself sonarcloud.io tool for code analisys.
> It's
> > free for open source project and I've made Ignite project initial run
> [1].
> >
> > I've prepeared analisys for mysefl and found a lot of duplicated code
> > blocks [1]. Of course it's not the ideal tool but gave us direction of
> > thoughts. E.g. these classes [3]:
> > 1) FileWriteAheadLogManager.java
> > 2) FsyncModeFileWriteAheadLogManager.java
> >
> >
> > [1] https://sonarcloud.io/dashboard?id=org.apache.ignite%3Aapache-ignite
> > [2]
> >
> >
> https://sonarcloud.io/component_measures?id=org.apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > [3]
> >
> >
> https://sonarcloud.io/component_measures?id=org.apache.ignite%3Aapache-ignite&metric=duplicated_blocks&selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> >
> > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jo...@gmail.com>:
> >
> > > +1 to this approach,
> > >
> > > It can be also very helpful in failover scenarios when something wrong
> > > happened with disk. In this case we're reducing the number of points of
> > > failure.
> > >
> > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > dmitriy.govorukhin@gmail.com>
> > > :
> > >
> > > > Hi Ignite,
> > > >
> > > > Do we have a general approach to work with a file and directories?
> > > > I see many duplication logic for write through .tmp file.
> > > >
> > > > For example,
> > > >
> > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > >
> > > > All of these methods implement the same logic, write to tmp file and
> > > rename
> > > > to normal name.
> > > >
> > > > I guess, will be better if we stopping write duplication logic code
> and
> > > > start to consolidate all in one place.
> > > >
> > > > Also, I think that current approach to creating files is not quite
> > right
> > > > faithful. Each internal component
> > > > create/delete files inside himself, and nobody knows where which
> files
> > > > located.
> > > >
> > > > I suggest refactoring code and create something (maybe new manager)
> > that
> > > > will know about all files inside the node. All internal components
> must
> > > > create files only through this one.
> > > >
> > > > It makes help to write tests for persistence easier and reduce
> > > duplication
> > > > code in working with files.
> > > >
> > >
> >
>
Re: work with files and directories
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi Maxim,
FileWriteAheadLogManager & FsyncModeFileWriteAheadLogManager were
intentionally copy-pasted in hope we will soon delete FsyncManager.
But it is still shows this tool works good. Probably we could integrate
this tool to our processes.
Sincerely,
Dmitriy Pavlov
вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <ma...@gmail.com>:
> +1 also for something like "resource manager".
>
> Recently, I've found for myself sonarcloud.io tool for code analisys. It's
> free for open source project and I've made Ignite project initial run [1].
>
> I've prepeared analisys for mysefl and found a lot of duplicated code
> blocks [1]. Of course it's not the ideal tool but gave us direction of
> thoughts. E.g. these classes [3]:
> 1) FileWriteAheadLogManager.java
> 2) FsyncModeFileWriteAheadLogManager.java
>
>
> [1] https://sonarcloud.io/dashboard?id=org.apache.ignite%3Aapache-ignite
> [2]
>
> https://sonarcloud.io/component_measures?id=org.apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> [3]
>
> https://sonarcloud.io/component_measures?id=org.apache.ignite%3Aapache-ignite&metric=duplicated_blocks&selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
>
> вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jo...@gmail.com>:
>
> > +1 to this approach,
> >
> > It can be also very helpful in failover scenarios when something wrong
> > happened with disk. In this case we're reducing the number of points of
> > failure.
> >
> > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > dmitriy.govorukhin@gmail.com>
> > :
> >
> > > Hi Ignite,
> > >
> > > Do we have a general approach to work with a file and directories?
> > > I see many duplication logic for write through .tmp file.
> > >
> > > For example,
> > >
> > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > GridCacheDatabaseSharedManage.nodeStart();
> > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > >
> > > All of these methods implement the same logic, write to tmp file and
> > rename
> > > to normal name.
> > >
> > > I guess, will be better if we stopping write duplication logic code and
> > > start to consolidate all in one place.
> > >
> > > Also, I think that current approach to creating files is not quite
> right
> > > faithful. Each internal component
> > > create/delete files inside himself, and nobody knows where which files
> > > located.
> > >
> > > I suggest refactoring code and create something (maybe new manager)
> that
> > > will know about all files inside the node. All internal components must
> > > create files only through this one.
> > >
> > > It makes help to write tests for persistence easier and reduce
> > duplication
> > > code in working with files.
> > >
> >
>
Re: work with files and directories
Posted by Maxim Muzafarov <ma...@gmail.com>.
+1 also for something like "resource manager".
Recently, I've found for myself sonarcloud.io tool for code analisys. It's
free for open source project and I've made Ignite project initial run [1].
I've prepeared analisys for mysefl and found a lot of duplicated code
blocks [1]. Of course it's not the ideal tool but gave us direction of
thoughts. E.g. these classes [3]:
1) FileWriteAheadLogManager.java
2) FsyncModeFileWriteAheadLogManager.java
[1] https://sonarcloud.io/dashboard?id=org.apache.ignite%3Aapache-ignite
[2]
https://sonarcloud.io/component_measures?id=org.apache.ignite%3Aapache-ignite&metric=duplicated_blocks
[3]
https://sonarcloud.io/component_measures?id=org.apache.ignite%3Aapache-ignite&metric=duplicated_blocks&selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jo...@gmail.com>:
> +1 to this approach,
>
> It can be also very helpful in failover scenarios when something wrong
> happened with disk. In this case we're reducing the number of points of
> failure.
>
> 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com>
> :
>
> > Hi Ignite,
> >
> > Do we have a general approach to work with a file and directories?
> > I see many duplication logic for write through .tmp file.
> >
> > For example,
> >
> > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > GridCacheDatabaseSharedManage.nodeStart();
> > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> >
> > All of these methods implement the same logic, write to tmp file and
> rename
> > to normal name.
> >
> > I guess, will be better if we stopping write duplication logic code and
> > start to consolidate all in one place.
> >
> > Also, I think that current approach to creating files is not quite right
> > faithful. Each internal component
> > create/delete files inside himself, and nobody knows where which files
> > located.
> >
> > I suggest refactoring code and create something (maybe new manager) that
> > will know about all files inside the node. All internal components must
> > create files only through this one.
> >
> > It makes help to write tests for persistence easier and reduce
> duplication
> > code in working with files.
> >
>
Re: work with files and directories
Posted by Pavel Kovalenko <jo...@gmail.com>.
+1 to this approach,
It can be also very helpful in failover scenarios when something wrong
happened with disk. In this case we're reducing the number of points of
failure.
2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <dm...@gmail.com>
:
> Hi Ignite,
>
> Do we have a general approach to work with a file and directories?
> I see many duplication logic for write through .tmp file.
>
> For example,
>
> GridCacheDatabaseSharedManager.writeCheckpointEntry();
> GridCacheDatabaseSharedManage.nodeStart();
> FileWriteAheadLogManager.FileArchiver.archiveSegment();
>
> All of these methods implement the same logic, write to tmp file and rename
> to normal name.
>
> I guess, will be better if we stopping write duplication logic code and
> start to consolidate all in one place.
>
> Also, I think that current approach to creating files is not quite right
> faithful. Each internal component
> create/delete files inside himself, and nobody knows where which files
> located.
>
> I suggest refactoring code and create something (maybe new manager) that
> will know about all files inside the node. All internal components must
> create files only through this one.
>
> It makes help to write tests for persistence easier and reduce duplication
> code in working with files.
>