You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Xiangdong Huang <sa...@gmail.com> on 2019/08/31 14:50:04 UTC
[IoTDB-189] question about the feature of compatibility of HDFS
Hi,
This issue only has a title.. without any description...
I think it is for letting IoTDB write TsFiles on HDFS directly.
My question is, is only TsFile written on HDFS? or all files (e.g., wal,
system data file)?
Besides, if all files are on HDFS, do we need to avoid using File name as a
mark or a lock (i.e., create a file and then rename it) for performance
consideration?
(I am on my trip and then I realize clearly that current info on issue is
not enough for guys who are on the internet)
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Re: [IoTDB-189] question about the feature of compatibility of HDFS
Posted by Jialin Qiao <qj...@mails.tsinghua.edu.cn>.
Hi,
I remember that the sync module will rename the file after merge. More details about sync could be given by @Tianan.
Best,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -----原始邮件-----
> 发件人: "Xiangdong Huang" <sa...@gmail.com>
> 发送时间: 2019-09-02 13:28:50 (星期一)
> 收件人: dev@iotdb.apache.org
> 抄送:
> 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS
>
> Hi,
>
> Look forward to the performance comparison.
>
> > Currently, renaming only occurs in two case
> > (1) updating the version files for each storage group periodically in
> system/storage_group_x/version-xxx
> > (2) updating the compression-ratio file when dynamically adjusting
> parameters
>
> How about the sync module? Seems there is a file for recording the file
> list (well, the sync module execution is not frequent)
>
> Best,
>
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao <qj...@mails.tsinghua.edu.cn> 于2019年9月1日周日 下午8:00写道：
>
> > Hi,
> >
> > > Ah... I remember that there is a file whose name is "version-xxx", in
> > which
> > the "xxx" is a counter, and will be updated if the version difference
> > between the disk and memory >= 50.
> >
> > Yes, each storage group has a file named version-xxx under
> > system/storage_group_xxx folder.
> >
> > > I am not sure whether it is correct that "renaming is not very frequent"
> > because each ChunkGroup has a version (am I right?)..
> >
> > Right, each ChunkGroup will increase this version, and each new data file
> > will increase this version.
> >
> > Currently, renaming only occurs in two case
> > (1) updating the version files for each storage group periodically in
> > system/storage_group_x/version-xxx
> > (2) updating the compression-ratio file when dynamically adjusting
> > parameters
> >
> > It's hard to say whether it is frequent or not, we can compare the
> > performance between local file system and HDFS after supporting HDFS.
> >
> > Best,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> > > -----原始邮件-----
> > > 发件人: "Xiangdong Huang" <sa...@gmail.com>
> > > 发送时间: 2019-09-01 23:10:08 (星期日)
> > > 收件人: dev@iotdb.apache.org
> > > 抄送:
> > > 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS
> > >
> > > Hi,
> > >
> > > > since renaming is not very frequent, maybe we do not need to worry
> > about
> > > performance.
> > >
> > > Ah... I remember that there is a file whose name is "version-xxx", in
> > which
> > > the "xxx" is a counter, and will be updated if the version difference
> > > between the disk and memory >= 50.
> > >
> > > I am not sure whether it is correct that "renaming is not very frequent"
> > > because each ChunkGroup has a version (am I right?)..
> > >
> > > But, I am not saying that writing on HDFS is bad, I just want to remind
> > you
> > > that reducing renaming operations, or proving that renaming on HDFS is
> > not
> > > a time consuming operation.
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > > 黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > Zesong Sun <sa...@foxmail.com> 于2019年8月31日周六 下午8:28写道：
> > >
> > > > Hi,
> > > >
> > > >
> > > > This issue is a sub-task of IoTDB-187 [1], which enables users to
> > choose
> > > > whether files are storied in local file system or HDFS.
> > > >
> > > >
> > > > In this sub-task, I intend to use an encapsulated "File" to replace
> > > > existed Java File in our project.
> > > >
> > > >
> > > > Currently in our design, all files could be written on HDFS.
> > > >
> > > >
> > > > HDFS supports truncating files, and since renaming is not very
> > frequent,
> > > > maybe we do not need to worry about performance.
> > > >
> > > >
> > > > If anyone has any other ideas, please discuss with me.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IOTDB-187
> > > >
> > > >
> > > > ------------------
> > > > Zesong Sun
> > > > School of Software, Tsinghua University
> > > >
> > > > 孙泽嵩
> > > > 清华大学 软件学院
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ------------------ 原始邮件 ------------------
> > > > 发件人: "Xiangdong Huang"<sa...@gmail.com>;
> > > > 发送时间: 2019年8月31日(星期六) 晚上10:50
> > > > 收件人: "dev"<de...@iotdb.apache.org>;
> > > >
> > > > 主题: [IoTDB-189] question about the feature of compatibility of HDFS
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > This issue only has a title.. without any description...
> > > >
> > > > I think it is for letting IoTDB write TsFiles on HDFS directly.
> > > >
> > > > My question is, is only TsFile written on HDFS? or all files (e.g.,
> > wal,
> > > > system data file)?
> > > >
> > > > Besides, if all files are on HDFS, do we need to avoid using File name
> > as a
> > > > mark or a lock (i.e., create a file and then rename it) for performance
> > > > consideration?
> > > >
> > > > (I am on my trip and then I realize clearly that current info on issue
> > is
> > > > not enough for guys who are on the internet)
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > > School of Software, Tsinghua University
> > > >
> > > > 黄向东
> > > > 清华大学 软件学院
> >
Re: [IoTDB-189] question about the feature of compatibility of HDFS
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
Look forward to the performance comparison.
> Currently, renaming only occurs in two case
> (1) updating the version files for each storage group periodically in
system/storage_group_x/version-xxx
> (2) updating the compression-ratio file when dynamically adjusting
parameters
How about the sync module? Seems there is a file for recording the file
list (well, the sync module execution is not frequent)
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Jialin Qiao <qj...@mails.tsinghua.edu.cn> 于2019年9月1日周日 下午8:00写道：
> Hi,
>
> > Ah... I remember that there is a file whose name is "version-xxx", in
> which
> the "xxx" is a counter, and will be updated if the version difference
> between the disk and memory >= 50.
>
> Yes, each storage group has a file named version-xxx under
> system/storage_group_xxx folder.
>
> > I am not sure whether it is correct that "renaming is not very frequent"
> because each ChunkGroup has a version (am I right?)..
>
> Right, each ChunkGroup will increase this version, and each new data file
> will increase this version.
>
> Currently, renaming only occurs in two case
> (1) updating the version files for each storage group periodically in
> system/storage_group_x/version-xxx
> (2) updating the compression-ratio file when dynamically adjusting
> parameters
>
> It's hard to say whether it is frequent or not, we can compare the
> performance between local file system and HDFS after supporting HDFS.
>
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -----原始邮件-----
> > 发件人: "Xiangdong Huang" <sa...@gmail.com>
> > 发送时间: 2019-09-01 23:10:08 (星期日)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS
> >
> > Hi,
> >
> > > since renaming is not very frequent, maybe we do not need to worry
> about
> > performance.
> >
> > Ah... I remember that there is a file whose name is "version-xxx", in
> which
> > the "xxx" is a counter, and will be updated if the version difference
> > between the disk and memory >= 50.
> >
> > I am not sure whether it is correct that "renaming is not very frequent"
> > because each ChunkGroup has a version (am I right?)..
> >
> > But, I am not saying that writing on HDFS is bad, I just want to remind
> you
> > that reducing renaming operations, or proving that renaming on HDFS is
> not
> > a time consuming operation.
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Zesong Sun <sa...@foxmail.com> 于2019年8月31日周六 下午8:28写道：
> >
> > > Hi,
> > >
> > >
> > > This issue is a sub-task of IoTDB-187 [1], which enables users to
> choose
> > > whether files are storied in local file system or HDFS.
> > >
> > >
> > > In this sub-task, I intend to use an encapsulated "File" to replace
> > > existed Java File in our project.
> > >
> > >
> > > Currently in our design, all files could be written on HDFS.
> > >
> > >
> > > HDFS supports truncating files, and since renaming is not very
> frequent,
> > > maybe we do not need to worry about performance.
> > >
> > >
> > > If anyone has any other ideas, please discuss with me.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IOTDB-187
> > >
> > >
> > > ------------------
> > > Zesong Sun
> > > School of Software, Tsinghua University
> > >
> > > 孙泽嵩
> > > 清华大学 软件学院
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "Xiangdong Huang"<sa...@gmail.com>;
> > > 发送时间: 2019年8月31日(星期六) 晚上10:50
> > > 收件人: "dev"<de...@iotdb.apache.org>;
> > >
> > > 主题: [IoTDB-189] question about the feature of compatibility of HDFS
> > >
> > >
> > >
> > > Hi,
> > >
> > > This issue only has a title.. without any description...
> > >
> > > I think it is for letting IoTDB write TsFiles on HDFS directly.
> > >
> > > My question is, is only TsFile written on HDFS? or all files (e.g.,
> wal,
> > > system data file)?
> > >
> > > Besides, if all files are on HDFS, do we need to avoid using File name
> as a
> > > mark or a lock (i.e., create a file and then rename it) for performance
> > > consideration?
> > >
> > > (I am on my trip and then I realize clearly that current info on issue
> is
> > > not enough for guys who are on the internet)
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > > 黄向东
> > > 清华大学 软件学院
>
Re: [IoTDB-189] question about the feature of compatibility of HDFS
Posted by Jialin Qiao <qj...@mails.tsinghua.edu.cn>.
Hi,
> Ah... I remember that there is a file whose name is "version-xxx", in which
the "xxx" is a counter, and will be updated if the version difference
between the disk and memory >= 50.
Yes, each storage group has a file named version-xxx under system/storage_group_xxx folder.
> I am not sure whether it is correct that "renaming is not very frequent"
because each ChunkGroup has a version (am I right?)..
Right, each ChunkGroup will increase this version, and each new data file will increase this version.
Currently, renaming only occurs in two case
(1) updating the version files for each storage group periodically in system/storage_group_x/version-xxx
(2) updating the compression-ratio file when dynamically adjusting parameters
It's hard to say whether it is frequent or not, we can compare the performance between local file system and HDFS after supporting HDFS.
Best,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -----原始邮件-----
> 发件人: "Xiangdong Huang" <sa...@gmail.com>
> 发送时间: 2019-09-01 23:10:08 (星期日)
> 收件人: dev@iotdb.apache.org
> 抄送:
> 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS
>
> Hi,
>
> > since renaming is not very frequent, maybe we do not need to worry about
> performance.
>
> Ah... I remember that there is a file whose name is "version-xxx", in which
> the "xxx" is a counter, and will be updated if the version difference
> between the disk and memory >= 50.
>
> I am not sure whether it is correct that "renaming is not very frequent"
> because each ChunkGroup has a version (am I right?)..
>
> But, I am not saying that writing on HDFS is bad, I just want to remind you
> that reducing renaming operations, or proving that renaming on HDFS is not
> a time consuming operation.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Zesong Sun <sa...@foxmail.com> 于2019年8月31日周六 下午8:28写道：
>
> > Hi,
> >
> >
> > This issue is a sub-task of IoTDB-187 [1], which enables users to choose
> > whether files are storied in local file system or HDFS.
> >
> >
> > In this sub-task, I intend to use an encapsulated "File" to replace
> > existed Java File in our project.
> >
> >
> > Currently in our design, all files could be written on HDFS.
> >
> >
> > HDFS supports truncating files, and since renaming is not very frequent,
> > maybe we do not need to worry about performance.
> >
> >
> > If anyone has any other ideas, please discuss with me.
> >
> >
> > [1] https://issues.apache.org/jira/browse/IOTDB-187
> >
> >
> > ------------------
> > Zesong Sun
> > School of Software, Tsinghua University
> >
> > 孙泽嵩
> > 清华大学 软件学院
> >
> >
> >
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "Xiangdong Huang"<sa...@gmail.com>;
> > 发送时间: 2019年8月31日(星期六) 晚上10:50
> > 收件人: "dev"<de...@iotdb.apache.org>;
> >
> > 主题: [IoTDB-189] question about the feature of compatibility of HDFS
> >
> >
> >
> > Hi,
> >
> > This issue only has a title.. without any description...
> >
> > I think it is for letting IoTDB write TsFiles on HDFS directly.
> >
> > My question is, is only TsFile written on HDFS? or all files (e.g., wal,
> > system data file)?
> >
> > Besides, if all files are on HDFS, do we need to avoid using File name as a
> > mark or a lock (i.e., create a file and then rename it) for performance
> > consideration?
> >
> > (I am on my trip and then I realize clearly that current info on issue is
> > not enough for guys who are on the internet)
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
Re: [IoTDB-189] question about the feature of compatibility of HDFS
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
> since renaming is not very frequent, maybe we do not need to worry about
performance.
Ah... I remember that there is a file whose name is "version-xxx", in which
the "xxx" is a counter, and will be updated if the version difference
between the disk and memory >= 50.
I am not sure whether it is correct that "renaming is not very frequent"
because each ChunkGroup has a version (am I right?)..
But, I am not saying that writing on HDFS is bad, I just want to remind you
that reducing renaming operations, or proving that renaming on HDFS is not
a time consuming operation.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Zesong Sun <sa...@foxmail.com> 于2019年8月31日周六 下午8:28写道：
> Hi,
>
>
> This issue is a sub-task of IoTDB-187 [1], which enables users to choose
> whether files are storied in local file system or HDFS.
>
>
> In this sub-task, I intend to use an encapsulated "File" to replace
> existed Java File in our project.
>
>
> Currently in our design, all files could be written on HDFS.
>
>
> HDFS supports truncating files, and since renaming is not very frequent,
> maybe we do not need to worry about performance.
>
>
> If anyone has any other ideas, please discuss with me.
>
>
> [1] https://issues.apache.org/jira/browse/IOTDB-187
>
>
> ------------------
> Zesong Sun
> School of Software, Tsinghua University
>
> 孙泽嵩
> 清华大学 软件学院
>
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Xiangdong Huang"<sa...@gmail.com>;
> 发送时间: 2019年8月31日(星期六) 晚上10:50
> 收件人: "dev"<de...@iotdb.apache.org>;
>
> 主题: [IoTDB-189] question about the feature of compatibility of HDFS
>
>
>
> Hi,
>
> This issue only has a title.. without any description...
>
> I think it is for letting IoTDB write TsFiles on HDFS directly.
>
> My question is, is only TsFile written on HDFS? or all files (e.g., wal,
> system data file)?
>
> Besides, if all files are on HDFS, do we need to avoid using File name as a
> mark or a lock (i.e., create a file and then rename it) for performance
> consideration?
>
> (I am on my trip and then I realize clearly that current info on issue is
> not enough for guys who are on the internet)
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
回复：[IoTDB-189] question about the feature of compatibility of HDFS
Posted by Zesong Sun <sa...@foxmail.com>.
Hi,
This issue is a sub-task of IoTDB-187 [1], which enables users to choose whether files are storied in local file system or HDFS.
In this sub-task, I intend to use an encapsulated "File" to replace existed Java File in our project.
Currently in our design, all files could be written on HDFS.
HDFS supports truncating files, and since renaming is not very frequent, maybe we do not need to worry about performance.
If anyone has any other ideas, please discuss with me.
[1] https://issues.apache.org/jira/browse/IOTDB-187
------------------
Zesong Sun
School of Software, Tsinghua University
孙泽嵩
清华大学 软件学院
------------------ 原始邮件 ------------------
发件人: "Xiangdong Huang"<sa...@gmail.com>;
发送时间: 2019年8月31日(星期六) 晚上10:50
收件人: "dev"<de...@iotdb.apache.org>;
主题: [IoTDB-189] question about the feature of compatibility of HDFS
Hi,
This issue only has a title.. without any description...
I think it is for letting IoTDB write TsFiles on HDFS directly.
My question is, is only TsFile written on HDFS? or all files (e.g., wal,
system data file)?
Besides, if all files are on HDFS, do we need to avoid using File name as a
mark or a lock (i.e., create a file and then rename it) for performance
consideration?
(I am on my trip and then I realize clearly that current info on issue is
not enough for guys who are on the internet)
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院