Refactor FileNodeManager and FileNodeProcessor

[Xiangdong Huang <sa...@gmail.com>]

Hi,

The two class, FileNodeManager and FileNodeProcessor, are too large (more
than 3000 lines totally), and the logic is complex. But these two classes
play very very important roles in IoTDB.

Now, I decide to refactor the two classes to try to make it more clear and
guarantee the logic is optimized. It may spend about 1 ~ 2 weeks.

So,

1) I will close PR #77 (rename FileNodeManager to StorageGroupManager)
without merge.
2) If some new coming PR will modify the logic of the two classes, please
let me know.  I can merge your PR to avoid heavy conflict of codes...

Best,

-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Re: Refactor FileNodeManager and FileNodeProcessor

[Xiangdong Huang <sa...@gmail.com>]

Hi,

I have almost finished refactor the TsFileProcessor module. (
https://issues.apache.org/jira/browse/IOTDB-58)

TsFileProcessor is aim to replace BufferWriteProcessor and some functions
of FileNodeProcessor.

TsFileProcessor manages the ordered data in memtable and all complete
TsFiles. it only provide insert/update/delete/close/flush and query APIs.
- For Insert, if it returns false, then it means that the inserted data
should not be managed by TsFileProcessor, which means it is an overflowed
data.
- For query, the function will query data not only from the memtable but
also from TsFiles.
- The parameter of Insert() is InsertPlan, rather than (deviceID, time,
measurements, values). It is to reduce the class transformation... (Another
way to optimize it is do not use InsertPlan in WAL..., which may be better)

WAL is not enabled in this module. I will add the function when all related
codes is refactored.

Now the codes are not used in IoTDB's process, you can try it by run
TsFileProcessorTest.java. When the overflowProcessor and FileNodeProcessor
is refactored, we can enable this part of codes.

The branch is: refactor_bufferwrite, I will submit a PR later to let
interested guys review it.

Then I will refactor the module of OverflowProcessor.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Willem Jiang <wi...@gmail.com> 于2019年3月9日周六 上午10:31写道：

> Hi Xiangdong,
>
> I think you can keep everyone posted by creating a refactoring breach
> in the repo.
> In this way it will save you some energy when you merge the PR from
> other developer, as other developer can create their PR which is based
> on your latest change.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Sat, Mar 9, 2019 at 12:09 AM Xiangdong Huang <sa...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > The two class, FileNodeManager and FileNodeProcessor, are too large (more
> > than 3000 lines totally), and the logic is complex. But these two classes
> > play very very important roles in IoTDB.
> >
> > Now, I decide to refactor the two classes to try to make it more clear
> and
> > guarantee the logic is optimized. It may spend about 1 ~ 2 weeks.
> >
> > So,
> >
> > 1) I will close PR #77 (rename FileNodeManager to StorageGroupManager)
> > without merge.
> > 2) If some new coming PR will modify the logic of the two classes, please
> > let me know.  I can merge your PR to avoid heavy conflict of codes...
> >
> > Best,
> >
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
>

Re: Refactor FileNodeManager and FileNodeProcessor

[Willem Jiang <wi...@gmail.com>]

Hi Xiangdong,

I think you can keep everyone posted by creating a refactoring breach
in the repo.
In this way it will save you some energy when you merge the PR from
other developer, as other developer can create their PR which is based
on your latest change.

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Sat, Mar 9, 2019 at 12:09 AM Xiangdong Huang <sa...@gmail.com> wrote:
>
> Hi,
>
> The two class, FileNodeManager and FileNodeProcessor, are too large (more
> than 3000 lines totally), and the logic is complex. But these two classes
> play very very important roles in IoTDB.
>
> Now, I decide to refactor the two classes to try to make it more clear and
> guarantee the logic is optimized. It may spend about 1 ~ 2 weeks.
>
> So,
>
> 1) I will close PR #77 (rename FileNodeManager to StorageGroupManager)
> without merge.
> 2) If some new coming PR will modify the logic of the two classes, please
> let me know.  I can merge your PR to avoid heavy conflict of codes...
>
> Best,
>
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院