You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by atoiLiu <at...@163.com> on 2020/02/11 09:35:09 UTC
Suggestions for new TsFile
Hi,
I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad design.
TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record startIndex , endIndex of TsOffsetArray, it’s looks like :
TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] }, {deviceId(d1), [3,5] }, …. } }
We can delete TsOffsetArray and store the offsets directly in the deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>> to record . This change will save 4 bytes per device on disk, because every device just need record the number of offsets and offsets. it’s looks like：
TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, … }
In addition, TimeSeriesMetaData is an ordered structure on the hard disk, and the TimeSeriesMetaData for each device is linked together, so TsFileMetaData does not need to store all offset information, so there two optimization directions:
1. Save startTime , endTime and offset for each TimeSeriesMetaData in TsFileMetaData. The nice thing about this is that when you read TsFileMetaData from your hard drive, you can directly do a filter to filter which TimeSeriesMetaData is not necessary to read.
2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so that you can loop through it and just need once to seek, it’s looks like :
TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
[1] https://github.com/apache/incubator-iotdb/pull/736 <https://github.com/apache/incubator-iotdb/pull/736>
Thanks
Dawei Liu
Re: Suggestions for new TsFile
Posted by Haonan Hou <hh...@outlook.com>.
+1
It’s a good idea.
Thanks,
Haonan Hou
> On Feb 13, 2020, at 5:20 PM, Dawei Liu <at...@163.com> wrote:
>
> Hi，
>
> Sorry，i overlooked that the first step in server was to filter files through startTime/endTime Map.
>
> +1 for add a Statistics in TimeseriesMetadata，
>
> For example:
> Device Shadow (设备影子) , it is often necessary to find the last information about a device
>
> Thanks
>
> Dawei Liu
>
>> 2020年2月13日 下午4:58，Jialin Qiao <qj...@mails.tsinghua.edu.cn> 写道：
>>
>> Hi,
>>
>> I have a suggestion.
>> We could add a Statistics in TimeseriesMetadata to support fast aggregations.
>>
>> Thanks,
>> --
>> Jialin Qiao
>> School of Software, Tsinghua University
>>
>> 乔嘉林
>> 清华大学 软件学院
>>
>>> -----原始邮件-----
>>> 发件人: "Jialin Qiao" <qj...@mails.tsinghua.edu.cn>
>>> 发送时间: 2020-02-13 16:16:54 (星期四)
>>> 收件人: dev@iotdb.apache.org
>>> 抄送:
>>> 主题: Re: Suggestions for new TsFile
>>>
>>> Hi,
>>>
>>> +1 for most queries contains a time filter
>>>
>>> But I don't know what do you mean by "add a time attribute", add to where?
>>>
>>> Thanks,
>>> --
>>> Jialin Qiao
>>> School of Software, Tsinghua University
>>>
>>> 乔嘉林
>>> 清华大学 软件学院
>>>
>>>> -----原始邮件-----
>>>> 发件人: "Dawei Liu" <at...@163.com>
>>>> 发送时间: 2020-02-13 15:55:48 (星期四)
>>>> 收件人: dev@iotdb.apache.org
>>>> 抄送:
>>>> 主题: Re: Suggestions for new TsFile
>>>>
>>>> Hi,
>>>>
>>>> I found another problem, when I execute : ` SELECT s1 FROM xx WHERE time = 1`
>>>>
>>>> In the new TsFile, need to read the hard drive 3 times,
>>>>
>>>> 1. Read TsFileMetaData
>>>>
>>>> 2. Read the MetaData of all measurement of the device ( TimeSeriesMetaData )
>>>>
>>>> 3. Read the required measurement of the ChunkMetaData and then the time filter( time = 1 ) can be filter which Chunk can be used
>>>>
>>>>
>>>>
>>>> In the current server, most of the time is used for TimeFilter, we read a lot of metadata information, if the end can not be used, this is a very big loss.
>>>>
>>>> So I think we should add a time attribute so that we can know if the file is can’t to use when we first read the hard drive
>>>>
>>>> Thanks
>>>>
>>>> Dawei Liu
>>>>
>>>>
>>>>> 2020年2月12日 下午8:03，Dawei Liu <at...@163.com> 写道：
>>>>>
>>>>> Hi,
>>>>>
>>>>> I see it. It looks very comfortable.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Dawei Liu
>>>>>
>>>>>> 2020年2月12日 下午6:52，Haonan Hou <hh...@outlook.com> 写道：
>>>>>>
>>>>>> Sure, already added.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Haonan Hou
>>>>>>
>>>>>>> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
>>>>>>>
>>>>>>> Hi Haonan,
>>>>>>>
>>>>>>>
>>>>>>> I can not see the picture, could you please put it in the PR?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --
>>>>>>> Jialin Qiao
>>>>>>> School of Software, Tsinghua University
>>>>>>>
>>>>>>> 乔嘉林
>>>>>>> 清华大学 软件学院
>>>>>>>
>>>>>>> -----原始邮件-----
>>>>>>> 发件人:"Haonan Hou" <hh...@outlook.com>
>>>>>>> 发送时间:2020-02-12 16:27:22 (星期三)
>>>>>>> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
>>>>>>> 抄送:
>>>>>>> 主题: Re: Suggestions for new TsFile
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
>>>>>>>
>>>>>>>
>>>>>>> The mean differences is as below:
>>>>>>>
>>>>>>>
>>>>>>> 1. Remove TsOffsetArray.
>>>>>>> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The newer TsFile structure should be looked like:
>>>>>>>
>>>>>>> Here is an example of how the new structure works.
>>>>>>>
>>>>>>>
>>>>>>> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
>>>>>>> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
>>>>>>>
>>>>>>>
>>>>>>> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
>>>>>>>
>>>>>>>
>>>>>>> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>> Haonan Hou
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> If each device only stores each offset of TimeseriesMetadata like this:
>>>>>>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
>>>>>>> }
>>>>>>>
>>>>>>> It could be simplified to recording the start offset and end offset:
>>>>>>> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
>>>>>>>
>>>>>>> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
>>>>>>> 0 }, {deviceId(d1), 3 }, … }
>>>>>>>
>>>>>>> Thanks,
>>>>>>> —————————————————
>>>>>>> Jialin Qiao
>>>>>>> School of Software, Tsinghua University
>>>>>>>
>>>>>>> 乔嘉林
>>>>>>> 清华大学 软件学院
>>>>>>>
>>>>>>>
>>>>>>> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thank you for your reply.
>>>>>>> I am very happy that you can take my suggestion.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Dawei Liu
>>>>>>>
>>>>>>>
>>>>>>> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
>>>>>>>
>>>>>>> Hi Dawei,
>>>>>>>
>>>>>>> Thank you so much that you share your opinion about new TsFile!
>>>>>>> I am very happy to take your suggestions.
>>>>>>>
>>>>>>> You said we can remove TsOffsetArray and directly store the offset of
>>>>>>> TimeseriesMetaData. I agree with you. It is better than my version.
>>>>>>> Besides, for the optimization of TimeserieMetaData, I would like to
>>>>>>> discuss with other people to determine which way is better.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Haonan Hou
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
>>>>>>> design.
>>>>>>>
>>>>>>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
>>>>>>> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
>>>>>>> startIndex , endIndex of TsOffsetArray, it’s looks like :
>>>>>>>
>>>>>>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
>>>>>>> {deviceId(d1), [3,5] }, …. } }
>>>>>>>
>>>>>>> We can delete TsOffsetArray and store the offsets directly in the
>>>>>>> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
>>>>>>> to record . This change will save 4 bytes per device on disk, because every
>>>>>>> device just need record the number of offsets and offsets. it’s looks like：
>>>>>>>
>>>>>>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
>>>>>>> }, … }
>>>>>>>
>>>>>>>
>>>>>>> In addition, TimeSeriesMetaData is an ordered structure on the hard
>>>>>>> disk, and the TimeSeriesMetaData for each device is linked together, so
>>>>>>> TsFileMetaData does not need to store all offset information, so there two
>>>>>>> optimization directions:
>>>>>>>
>>>>>>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
>>>>>>> TsFileMetaData. The nice thing about this is that when you read
>>>>>>> TsFileMetaData from your hard drive, you can directly do a filter to filter
>>>>>>> which TimeSeriesMetaData is not necessary to read.
>>>>>>>
>>>>>>>
>>>>>>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
>>>>>>> that you can loop through it and just need once to seek, it’s looks like :
>>>>>>>
>>>>>>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [1] https://github.com/apache/incubator-iotdb/pull/736 <
>>>>>>> https://github.com/apache/incubator-iotdb/pull/736>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Dawei Liu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>
Re: Suggestions for new TsFile
Posted by Dawei Liu <at...@163.com>.
Hi，
Sorry，i overlooked that the first step in server was to filter files through startTime/endTime Map.
+1 for add a Statistics in TimeseriesMetadata，
For example:
Device Shadow (设备影子) , it is often necessary to find the last information about a device
Thanks
Dawei Liu
> 2020年2月13日 下午4:58，Jialin Qiao <qj...@mails.tsinghua.edu.cn> 写道：
>
> Hi,
>
> I have a suggestion.
> We could add a Statistics in TimeseriesMetadata to support fast aggregations.
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>> -----原始邮件-----
>> 发件人: "Jialin Qiao" <qj...@mails.tsinghua.edu.cn>
>> 发送时间: 2020-02-13 16:16:54 (星期四)
>> 收件人: dev@iotdb.apache.org
>> 抄送:
>> 主题: Re: Suggestions for new TsFile
>>
>> Hi,
>>
>> +1 for most queries contains a time filter
>>
>> But I don't know what do you mean by "add a time attribute", add to where?
>>
>> Thanks,
>> --
>> Jialin Qiao
>> School of Software, Tsinghua University
>>
>> 乔嘉林
>> 清华大学 软件学院
>>
>>> -----原始邮件-----
>>> 发件人: "Dawei Liu" <at...@163.com>
>>> 发送时间: 2020-02-13 15:55:48 (星期四)
>>> 收件人: dev@iotdb.apache.org
>>> 抄送:
>>> 主题: Re: Suggestions for new TsFile
>>>
>>> Hi,
>>>
>>> I found another problem, when I execute : ` SELECT s1 FROM xx WHERE time = 1`
>>>
>>> In the new TsFile, need to read the hard drive 3 times,
>>>
>>> 1. Read TsFileMetaData
>>>
>>> 2. Read the MetaData of all measurement of the device ( TimeSeriesMetaData )
>>>
>>> 3. Read the required measurement of the ChunkMetaData and then the time filter( time = 1 ) can be filter which Chunk can be used
>>>
>>>
>>>
>>> In the current server, most of the time is used for TimeFilter, we read a lot of metadata information, if the end can not be used, this is a very big loss.
>>>
>>> So I think we should add a time attribute so that we can know if the file is can’t to use when we first read the hard drive
>>>
>>> Thanks
>>>
>>> Dawei Liu
>>>
>>>
>>>> 2020年2月12日 下午8:03，Dawei Liu <at...@163.com> 写道：
>>>>
>>>> Hi,
>>>>
>>>> I see it. It looks very comfortable.
>>>>
>>>> Thanks
>>>>
>>>> Dawei Liu
>>>>
>>>>> 2020年2月12日 下午6:52，Haonan Hou <hh...@outlook.com> 写道：
>>>>>
>>>>> Sure, already added.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Haonan Hou
>>>>>
>>>>>> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
>>>>>>
>>>>>> Hi Haonan,
>>>>>>
>>>>>>
>>>>>> I can not see the picture, could you please put it in the PR?
>>>>>>
>>>>>> Thanks,
>>>>>> --
>>>>>> Jialin Qiao
>>>>>> School of Software, Tsinghua University
>>>>>>
>>>>>> 乔嘉林
>>>>>> 清华大学 软件学院
>>>>>>
>>>>>> -----原始邮件-----
>>>>>> 发件人:"Haonan Hou" <hh...@outlook.com>
>>>>>> 发送时间:2020-02-12 16:27:22 (星期三)
>>>>>> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
>>>>>> 抄送:
>>>>>> 主题: Re: Suggestions for new TsFile
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
>>>>>>
>>>>>>
>>>>>> The mean differences is as below:
>>>>>>
>>>>>>
>>>>>> 1. Remove TsOffsetArray.
>>>>>> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The newer TsFile structure should be looked like:
>>>>>>
>>>>>> Here is an example of how the new structure works.
>>>>>>
>>>>>>
>>>>>> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
>>>>>> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
>>>>>>
>>>>>>
>>>>>> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
>>>>>>
>>>>>>
>>>>>> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> Haonan Hou
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> If each device only stores each offset of TimeseriesMetadata like this:
>>>>>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
>>>>>> }
>>>>>>
>>>>>> It could be simplified to recording the start offset and end offset:
>>>>>> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
>>>>>>
>>>>>> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
>>>>>> 0 }, {deviceId(d1), 3 }, … }
>>>>>>
>>>>>> Thanks,
>>>>>> —————————————————
>>>>>> Jialin Qiao
>>>>>> School of Software, Tsinghua University
>>>>>>
>>>>>> 乔嘉林
>>>>>> 清华大学 软件学院
>>>>>>
>>>>>>
>>>>>> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thank you for your reply.
>>>>>> I am very happy that you can take my suggestion.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Dawei Liu
>>>>>>
>>>>>>
>>>>>> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
>>>>>>
>>>>>> Hi Dawei,
>>>>>>
>>>>>> Thank you so much that you share your opinion about new TsFile!
>>>>>> I am very happy to take your suggestions.
>>>>>>
>>>>>> You said we can remove TsOffsetArray and directly store the offset of
>>>>>> TimeseriesMetaData. I agree with you. It is better than my version.
>>>>>> Besides, for the optimization of TimeserieMetaData, I would like to
>>>>>> discuss with other people to determine which way is better.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Haonan Hou
>>>>>>
>>>>>>
>>>>>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
>>>>>> design.
>>>>>>
>>>>>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
>>>>>> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
>>>>>> startIndex , endIndex of TsOffsetArray, it’s looks like :
>>>>>>
>>>>>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
>>>>>> {deviceId(d1), [3,5] }, …. } }
>>>>>>
>>>>>> We can delete TsOffsetArray and store the offsets directly in the
>>>>>> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
>>>>>> to record . This change will save 4 bytes per device on disk, because every
>>>>>> device just need record the number of offsets and offsets. it’s looks like：
>>>>>>
>>>>>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
>>>>>> }, … }
>>>>>>
>>>>>>
>>>>>> In addition, TimeSeriesMetaData is an ordered structure on the hard
>>>>>> disk, and the TimeSeriesMetaData for each device is linked together, so
>>>>>> TsFileMetaData does not need to store all offset information, so there two
>>>>>> optimization directions:
>>>>>>
>>>>>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
>>>>>> TsFileMetaData. The nice thing about this is that when you read
>>>>>> TsFileMetaData from your hard drive, you can directly do a filter to filter
>>>>>> which TimeSeriesMetaData is not necessary to read.
>>>>>>
>>>>>>
>>>>>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
>>>>>> that you can loop through it and just need once to seek, it’s looks like :
>>>>>>
>>>>>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>>>>>>
>>>>>>
>>>>>>
>>>>>> [1] https://github.com/apache/incubator-iotdb/pull/736 <
>>>>>> https://github.com/apache/incubator-iotdb/pull/736>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Dawei Liu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
Re: Suggestions for new TsFile
Posted by Jialin Qiao <qj...@mails.tsinghua.edu.cn>.
Hi,
I have a suggestion.
We could add a Statistics in TimeseriesMetadata to support fast aggregations.
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -----原始邮件-----
> 发件人: "Jialin Qiao" <qj...@mails.tsinghua.edu.cn>
> 发送时间: 2020-02-13 16:16:54 (星期四)
> 收件人: dev@iotdb.apache.org
> 抄送:
> 主题: Re: Suggestions for new TsFile
>
> Hi,
>
> +1 for most queries contains a time filter
>
> But I don't know what do you mean by "add a time attribute", add to where?
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -----原始邮件-----
> > 发件人: "Dawei Liu" <at...@163.com>
> > 发送时间: 2020-02-13 15:55:48 (星期四)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: Re: Suggestions for new TsFile
> >
> > Hi,
> >
> > I found another problem, when I execute : ` SELECT s1 FROM xx WHERE time = 1`
> >
> > In the new TsFile, need to read the hard drive 3 times,
> >
> > 1. Read TsFileMetaData
> >
> > 2. Read the MetaData of all measurement of the device ( TimeSeriesMetaData )
> >
> > 3. Read the required measurement of the ChunkMetaData and then the time filter( time = 1 ) can be filter which Chunk can be used
> >
> >
> >
> > In the current server, most of the time is used for TimeFilter, we read a lot of metadata information, if the end can not be used, this is a very big loss.
> >
> > So I think we should add a time attribute so that we can know if the file is can’t to use when we first read the hard drive
> >
> > Thanks
> >
> > Dawei Liu
> >
> >
> > > 2020年2月12日 下午8:03，Dawei Liu <at...@163.com> 写道：
> > >
> > > Hi,
> > >
> > > I see it. It looks very comfortable.
> > >
> > > Thanks
> > >
> > > Dawei Liu
> > >
> > >> 2020年2月12日 下午6:52，Haonan Hou <hh...@outlook.com> 写道：
> > >>
> > >> Sure, already added.
> > >>
> > >> Thanks,
> > >>
> > >> Haonan Hou
> > >>
> > >>> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
> > >>>
> > >>> Hi Haonan,
> > >>>
> > >>>
> > >>> I can not see the picture, could you please put it in the PR?
> > >>>
> > >>> Thanks,
> > >>> --
> > >>> Jialin Qiao
> > >>> School of Software, Tsinghua University
> > >>>
> > >>> 乔嘉林
> > >>> 清华大学 软件学院
> > >>>
> > >>> -----原始邮件-----
> > >>> 发件人:"Haonan Hou" <hh...@outlook.com>
> > >>> 发送时间:2020-02-12 16:27:22 (星期三)
> > >>> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
> > >>> 抄送:
> > >>> 主题: Re: Suggestions for new TsFile
> > >>>
> > >>> Hi,
> > >>>
> > >>>
> > >>> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
> > >>>
> > >>>
> > >>> The mean differences is as below:
> > >>>
> > >>>
> > >>> 1. Remove TsOffsetArray.
> > >>> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> The newer TsFile structure should be looked like:
> > >>>
> > >>> Here is an example of how the new structure works.
> > >>>
> > >>>
> > >>> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
> > >>> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
> > >>>
> > >>>
> > >>> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
> > >>>
> > >>>
> > >>> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>>
> > >>> Haonan Hou
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
> > >>>
> > >>>
> > >>> Hi,
> > >>>
> > >>> If each device only stores each offset of TimeseriesMetadata like this:
> > >>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
> > >>> }
> > >>>
> > >>> It could be simplified to recording the start offset and end offset:
> > >>> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
> > >>>
> > >>> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
> > >>> 0 }, {deviceId(d1), 3 }, … }
> > >>>
> > >>> Thanks,
> > >>> —————————————————
> > >>> Jialin Qiao
> > >>> School of Software, Tsinghua University
> > >>>
> > >>> 乔嘉林
> > >>> 清华大学 软件学院
> > >>>
> > >>>
> > >>> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
> > >>>
> > >>> Hi,
> > >>>
> > >>> Thank you for your reply.
> > >>> I am very happy that you can take my suggestion.
> > >>>
> > >>>
> > >>> Thanks
> > >>>
> > >>> Dawei Liu
> > >>>
> > >>>
> > >>> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
> > >>>
> > >>> Hi Dawei,
> > >>>
> > >>> Thank you so much that you share your opinion about new TsFile!
> > >>> I am very happy to take your suggestions.
> > >>>
> > >>> You said we can remove TsOffsetArray and directly store the offset of
> > >>> TimeseriesMetaData. I agree with you. It is better than my version.
> > >>> Besides, for the optimization of TimeserieMetaData, I would like to
> > >>> discuss with other people to determine which way is better.
> > >>>
> > >>> Best,
> > >>>
> > >>> Haonan Hou
> > >>>
> > >>>
> > >>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
> > >>> design.
> > >>>
> > >>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
> > >>> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
> > >>> startIndex , endIndex of TsOffsetArray, it’s looks like :
> > >>>
> > >>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
> > >>> {deviceId(d1), [3,5] }, …. } }
> > >>>
> > >>> We can delete TsOffsetArray and store the offsets directly in the
> > >>> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
> > >>> to record . This change will save 4 bytes per device on disk, because every
> > >>> device just need record the number of offsets and offsets. it’s looks like：
> > >>>
> > >>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
> > >>> }, … }
> > >>>
> > >>>
> > >>> In addition, TimeSeriesMetaData is an ordered structure on the hard
> > >>> disk, and the TimeSeriesMetaData for each device is linked together, so
> > >>> TsFileMetaData does not need to store all offset information, so there two
> > >>> optimization directions:
> > >>>
> > >>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
> > >>> TsFileMetaData. The nice thing about this is that when you read
> > >>> TsFileMetaData from your hard drive, you can directly do a filter to filter
> > >>> which TimeSeriesMetaData is not necessary to read.
> > >>>
> > >>>
> > >>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
> > >>> that you can loop through it and just need once to seek, it’s looks like :
> > >>>
> > >>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
> > >>>
> > >>>
> > >>>
> > >>> [1] https://github.com/apache/incubator-iotdb/pull/736 <
> > >>> https://github.com/apache/incubator-iotdb/pull/736>
> > >>>
> > >>> Thanks
> > >>>
> > >>> Dawei Liu
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
Re: Suggestions for new TsFile
Posted by Jialin Qiao <qj...@mails.tsinghua.edu.cn>.
Hi,
+1 for most queries contains a time filter
But I don't know what do you mean by "add a time attribute", add to where?
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -----原始邮件-----
> 发件人: "Dawei Liu" <at...@163.com>
> 发送时间: 2020-02-13 15:55:48 (星期四)
> 收件人: dev@iotdb.apache.org
> 抄送:
> 主题: Re: Suggestions for new TsFile
>
> Hi,
>
> I found another problem, when I execute : ` SELECT s1 FROM xx WHERE time = 1`
>
> In the new TsFile, need to read the hard drive 3 times,
>
> 1. Read TsFileMetaData
>
> 2. Read the MetaData of all measurement of the device ( TimeSeriesMetaData )
>
> 3. Read the required measurement of the ChunkMetaData and then the time filter( time = 1 ) can be filter which Chunk can be used
>
>
>
> In the current server, most of the time is used for TimeFilter, we read a lot of metadata information, if the end can not be used, this is a very big loss.
>
> So I think we should add a time attribute so that we can know if the file is can’t to use when we first read the hard drive
>
> Thanks
>
> Dawei Liu
>
>
> > 2020年2月12日 下午8:03，Dawei Liu <at...@163.com> 写道：
> >
> > Hi,
> >
> > I see it. It looks very comfortable.
> >
> > Thanks
> >
> > Dawei Liu
> >
> >> 2020年2月12日 下午6:52，Haonan Hou <hh...@outlook.com> 写道：
> >>
> >> Sure, already added.
> >>
> >> Thanks,
> >>
> >> Haonan Hou
> >>
> >>> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
> >>>
> >>> Hi Haonan,
> >>>
> >>>
> >>> I can not see the picture, could you please put it in the PR?
> >>>
> >>> Thanks,
> >>> --
> >>> Jialin Qiao
> >>> School of Software, Tsinghua University
> >>>
> >>> 乔嘉林
> >>> 清华大学 软件学院
> >>>
> >>> -----原始邮件-----
> >>> 发件人:"Haonan Hou" <hh...@outlook.com>
> >>> 发送时间:2020-02-12 16:27:22 (星期三)
> >>> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
> >>> 抄送:
> >>> 主题: Re: Suggestions for new TsFile
> >>>
> >>> Hi,
> >>>
> >>>
> >>> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
> >>>
> >>>
> >>> The mean differences is as below:
> >>>
> >>>
> >>> 1. Remove TsOffsetArray.
> >>> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
> >>>
> >>>
> >>>
> >>>
> >>> The newer TsFile structure should be looked like:
> >>>
> >>> Here is an example of how the new structure works.
> >>>
> >>>
> >>> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
> >>> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
> >>>
> >>>
> >>> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
> >>>
> >>>
> >>> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
> >>>
> >>>
> >>> Thanks,
> >>>
> >>>
> >>> Haonan Hou
> >>>
> >>>
> >>>
> >>>
> >>> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
> >>>
> >>>
> >>> Hi,
> >>>
> >>> If each device only stores each offset of TimeseriesMetadata like this:
> >>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
> >>> }
> >>>
> >>> It could be simplified to recording the start offset and end offset:
> >>> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
> >>>
> >>> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
> >>> 0 }, {deviceId(d1), 3 }, … }
> >>>
> >>> Thanks,
> >>> —————————————————
> >>> Jialin Qiao
> >>> School of Software, Tsinghua University
> >>>
> >>> 乔嘉林
> >>> 清华大学 软件学院
> >>>
> >>>
> >>> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
> >>>
> >>> Hi,
> >>>
> >>> Thank you for your reply.
> >>> I am very happy that you can take my suggestion.
> >>>
> >>>
> >>> Thanks
> >>>
> >>> Dawei Liu
> >>>
> >>>
> >>> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
> >>>
> >>> Hi Dawei,
> >>>
> >>> Thank you so much that you share your opinion about new TsFile!
> >>> I am very happy to take your suggestions.
> >>>
> >>> You said we can remove TsOffsetArray and directly store the offset of
> >>> TimeseriesMetaData. I agree with you. It is better than my version.
> >>> Besides, for the optimization of TimeserieMetaData, I would like to
> >>> discuss with other people to determine which way is better.
> >>>
> >>> Best,
> >>>
> >>> Haonan Hou
> >>>
> >>>
> >>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
> >>> design.
> >>>
> >>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
> >>> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
> >>> startIndex , endIndex of TsOffsetArray, it’s looks like :
> >>>
> >>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
> >>> {deviceId(d1), [3,5] }, …. } }
> >>>
> >>> We can delete TsOffsetArray and store the offsets directly in the
> >>> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
> >>> to record . This change will save 4 bytes per device on disk, because every
> >>> device just need record the number of offsets and offsets. it’s looks like：
> >>>
> >>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
> >>> }, … }
> >>>
> >>>
> >>> In addition, TimeSeriesMetaData is an ordered structure on the hard
> >>> disk, and the TimeSeriesMetaData for each device is linked together, so
> >>> TsFileMetaData does not need to store all offset information, so there two
> >>> optimization directions:
> >>>
> >>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
> >>> TsFileMetaData. The nice thing about this is that when you read
> >>> TsFileMetaData from your hard drive, you can directly do a filter to filter
> >>> which TimeSeriesMetaData is not necessary to read.
> >>>
> >>>
> >>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
> >>> that you can loop through it and just need once to seek, it’s looks like :
> >>>
> >>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
> >>>
> >>>
> >>>
> >>> [1] https://github.com/apache/incubator-iotdb/pull/736 <
> >>> https://github.com/apache/incubator-iotdb/pull/736>
> >>>
> >>> Thanks
> >>>
> >>> Dawei Liu
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
Re: Suggestions for new TsFile
Posted by Dawei Liu <at...@163.com>.
Hi,
I found another problem, when I execute : ` SELECT s1 FROM xx WHERE time = 1`
In the new TsFile, need to read the hard drive 3 times,
1. Read TsFileMetaData
2. Read the MetaData of all measurement of the device ( TimeSeriesMetaData )
3. Read the required measurement of the ChunkMetaData and then the time filter( time = 1 ) can be filter which Chunk can be used
In the current server, most of the time is used for TimeFilter, we read a lot of metadata information, if the end can not be used, this is a very big loss.
So I think we should add a time attribute so that we can know if the file is can’t to use when we first read the hard drive
Thanks
Dawei Liu
> 2020年2月12日 下午8:03，Dawei Liu <at...@163.com> 写道：
>
> Hi,
>
> I see it. It looks very comfortable.
>
> Thanks
>
> Dawei Liu
>
>> 2020年2月12日 下午6:52，Haonan Hou <hh...@outlook.com> 写道：
>>
>> Sure, already added.
>>
>> Thanks,
>>
>> Haonan Hou
>>
>>> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
>>>
>>> Hi Haonan,
>>>
>>>
>>> I can not see the picture, could you please put it in the PR?
>>>
>>> Thanks,
>>> --
>>> Jialin Qiao
>>> School of Software, Tsinghua University
>>>
>>> 乔嘉林
>>> 清华大学 软件学院
>>>
>>> -----原始邮件-----
>>> 发件人:"Haonan Hou" <hh...@outlook.com>
>>> 发送时间:2020-02-12 16:27:22 (星期三)
>>> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
>>> 抄送:
>>> 主题: Re: Suggestions for new TsFile
>>>
>>> Hi,
>>>
>>>
>>> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
>>>
>>>
>>> The mean differences is as below:
>>>
>>>
>>> 1. Remove TsOffsetArray.
>>> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
>>>
>>>
>>>
>>>
>>> The newer TsFile structure should be looked like:
>>>
>>> Here is an example of how the new structure works.
>>>
>>>
>>> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
>>> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
>>>
>>>
>>> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
>>>
>>>
>>> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Haonan Hou
>>>
>>>
>>>
>>>
>>> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
>>>
>>>
>>> Hi,
>>>
>>> If each device only stores each offset of TimeseriesMetadata like this:
>>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
>>> }
>>>
>>> It could be simplified to recording the start offset and end offset:
>>> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
>>>
>>> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
>>> 0 }, {deviceId(d1), 3 }, … }
>>>
>>> Thanks,
>>> —————————————————
>>> Jialin Qiao
>>> School of Software, Tsinghua University
>>>
>>> 乔嘉林
>>> 清华大学 软件学院
>>>
>>>
>>> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
>>>
>>> Hi,
>>>
>>> Thank you for your reply.
>>> I am very happy that you can take my suggestion.
>>>
>>>
>>> Thanks
>>>
>>> Dawei Liu
>>>
>>>
>>> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
>>>
>>> Hi Dawei,
>>>
>>> Thank you so much that you share your opinion about new TsFile!
>>> I am very happy to take your suggestions.
>>>
>>> You said we can remove TsOffsetArray and directly store the offset of
>>> TimeseriesMetaData. I agree with you. It is better than my version.
>>> Besides, for the optimization of TimeserieMetaData, I would like to
>>> discuss with other people to determine which way is better.
>>>
>>> Best,
>>>
>>> Haonan Hou
>>>
>>>
>>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>>>
>>> Hi,
>>>
>>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
>>> design.
>>>
>>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
>>> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
>>> startIndex , endIndex of TsOffsetArray, it’s looks like :
>>>
>>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
>>> {deviceId(d1), [3,5] }, …. } }
>>>
>>> We can delete TsOffsetArray and store the offsets directly in the
>>> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
>>> to record . This change will save 4 bytes per device on disk, because every
>>> device just need record the number of offsets and offsets. it’s looks like：
>>>
>>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
>>> }, … }
>>>
>>>
>>> In addition, TimeSeriesMetaData is an ordered structure on the hard
>>> disk, and the TimeSeriesMetaData for each device is linked together, so
>>> TsFileMetaData does not need to store all offset information, so there two
>>> optimization directions:
>>>
>>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
>>> TsFileMetaData. The nice thing about this is that when you read
>>> TsFileMetaData from your hard drive, you can directly do a filter to filter
>>> which TimeSeriesMetaData is not necessary to read.
>>>
>>>
>>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
>>> that you can loop through it and just need once to seek, it’s looks like :
>>>
>>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>>>
>>>
>>>
>>> [1] https://github.com/apache/incubator-iotdb/pull/736 <
>>> https://github.com/apache/incubator-iotdb/pull/736>
>>>
>>> Thanks
>>>
>>> Dawei Liu
>>>
>>>
>>>
>>>
>>>
>>
Re: Suggestions for new TsFile
Posted by Dawei Liu <at...@163.com>.
Hi,
I see it. It looks very comfortable.
Thanks
Dawei Liu
> 2020年2月12日 下午6:52，Haonan Hou <hh...@outlook.com> 写道：
>
> Sure, already added.
>
> Thanks,
>
> Haonan Hou
>
>> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
>>
>> Hi Haonan,
>>
>>
>> I can not see the picture, could you please put it in the PR?
>>
>> Thanks,
>> --
>> Jialin Qiao
>> School of Software, Tsinghua University
>>
>> 乔嘉林
>> 清华大学 软件学院
>>
>> -----原始邮件-----
>> 发件人:"Haonan Hou" <hh...@outlook.com>
>> 发送时间:2020-02-12 16:27:22 (星期三)
>> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
>> 抄送:
>> 主题: Re: Suggestions for new TsFile
>>
>> Hi,
>>
>>
>> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
>>
>>
>> The mean differences is as below:
>>
>>
>> 1. Remove TsOffsetArray.
>> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
>>
>>
>>
>>
>> The newer TsFile structure should be looked like:
>>
>> Here is an example of how the new structure works.
>>
>>
>> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
>> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
>>
>>
>> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
>>
>>
>> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
>>
>>
>> Thanks,
>>
>>
>> Haonan Hou
>>
>>
>>
>>
>> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
>>
>>
>> Hi,
>>
>> If each device only stores each offset of TimeseriesMetadata like this:
>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
>> }
>>
>> It could be simplified to recording the start offset and end offset:
>> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
>>
>> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
>> 0 }, {deviceId(d1), 3 }, … }
>>
>> Thanks,
>> —————————————————
>> Jialin Qiao
>> School of Software, Tsinghua University
>>
>> 乔嘉林
>> 清华大学 软件学院
>>
>>
>> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
>>
>> Hi,
>>
>> Thank you for your reply.
>> I am very happy that you can take my suggestion.
>>
>>
>> Thanks
>>
>> Dawei Liu
>>
>>
>> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
>>
>> Hi Dawei,
>>
>> Thank you so much that you share your opinion about new TsFile!
>> I am very happy to take your suggestions.
>>
>> You said we can remove TsOffsetArray and directly store the offset of
>> TimeseriesMetaData. I agree with you. It is better than my version.
>> Besides, for the optimization of TimeserieMetaData, I would like to
>> discuss with other people to determine which way is better.
>>
>> Best,
>>
>> Haonan Hou
>>
>>
>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>>
>> Hi,
>>
>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
>> design.
>>
>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
>> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
>> startIndex , endIndex of TsOffsetArray, it’s looks like :
>>
>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
>> {deviceId(d1), [3,5] }, …. } }
>>
>> We can delete TsOffsetArray and store the offsets directly in the
>> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
>> to record . This change will save 4 bytes per device on disk, because every
>> device just need record the number of offsets and offsets. it’s looks like：
>>
>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
>> }, … }
>>
>>
>> In addition, TimeSeriesMetaData is an ordered structure on the hard
>> disk, and the TimeSeriesMetaData for each device is linked together, so
>> TsFileMetaData does not need to store all offset information, so there two
>> optimization directions:
>>
>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
>> TsFileMetaData. The nice thing about this is that when you read
>> TsFileMetaData from your hard drive, you can directly do a filter to filter
>> which TimeSeriesMetaData is not necessary to read.
>>
>>
>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
>> that you can loop through it and just need once to seek, it’s looks like :
>>
>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>>
>>
>>
>> [1] https://github.com/apache/incubator-iotdb/pull/736 <
>> https://github.com/apache/incubator-iotdb/pull/736>
>>
>> Thanks
>>
>> Dawei Liu
>>
>>
>>
>>
>>
>
Re: Suggestions for new TsFile
Posted by Haonan Hou <hh...@outlook.com>.
Sure, already added.
Thanks,
Haonan Hou
> On Feb 12, 2020, at 5:57 PM, Jialin Qiao <qj...@mails.tsinghua.edu.cn> wrote:
>
> Hi Haonan,
>
>
> I can not see the picture, could you please put it in the PR?
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> -----原始邮件-----
> 发件人:"Haonan Hou" <hh...@outlook.com>
> 发送时间:2020-02-12 16:27:22 (星期三)
> 收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
> 抄送:
> 主题: Re: Suggestions for new TsFile
>
> Hi,
>
>
> We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
>
>
> The mean differences is as below:
>
>
> 1. Remove TsOffsetArray.
> 2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
>
>
>
>
> The newer TsFile structure should be looked like:
>
> Here is an example of how the new structure works.
>
>
> When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
> the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
>
>
> After that, we are able to deserialize all TimeseriesMetadata in “d0”.
>
>
> Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
>
>
> Thanks,
>
>
> Haonan Hou
>
>
>
>
> On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
>
>
> Hi,
>
> If each device only stores each offset of TimeseriesMetadata like this:
> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
> }
>
> It could be simplified to recording the start offset and end offset:
> TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
>
> And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
> 0 }, {deviceId(d1), 3 }, … }
>
> Thanks,
> —————————————————
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>
> atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
>
> Hi,
>
> Thank you for your reply.
> I am very happy that you can take my suggestion.
>
>
> Thanks
>
> Dawei Liu
>
>
> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
>
> Hi Dawei,
>
> Thank you so much that you share your opinion about new TsFile!
> I am very happy to take your suggestions.
>
> You said we can remove TsOffsetArray and directly store the offset of
> TimeseriesMetaData. I agree with you. It is better than my version.
> Besides, for the optimization of TimeserieMetaData, I would like to
> discuss with other people to determine which way is better.
>
> Best,
>
> Haonan Hou
>
>
> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>
> Hi,
>
> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
> design.
>
> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
> startIndex , endIndex of TsOffsetArray, it’s looks like :
>
> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
> {deviceId(d1), [3,5] }, …. } }
>
> We can delete TsOffsetArray and store the offsets directly in the
> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
> to record . This change will save 4 bytes per device on disk, because every
> device just need record the number of offsets and offsets. it’s looks like：
>
> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
> }, … }
>
>
> In addition, TimeSeriesMetaData is an ordered structure on the hard
> disk, and the TimeSeriesMetaData for each device is linked together, so
> TsFileMetaData does not need to store all offset information, so there two
> optimization directions:
>
> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
> TsFileMetaData. The nice thing about this is that when you read
> TsFileMetaData from your hard drive, you can directly do a filter to filter
> which TimeSeriesMetaData is not necessary to read.
>
>
> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
> that you can loop through it and just need once to seek, it’s looks like :
>
> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>
>
>
> [1] https://github.com/apache/incubator-iotdb/pull/736 <
> https://github.com/apache/incubator-iotdb/pull/736>
>
> Thanks
>
> Dawei Liu
>
>
>
>
>
Re: Suggestions for new TsFile
Posted by Jialin Qiao <qj...@mails.tsinghua.edu.cn>.
Hi Haonan,
I can not see the picture, could you please put it in the PR?
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
-----原始邮件-----
发件人:"Haonan Hou" <hh...@outlook.com>
发送时间:2020-02-12 16:27:22 (星期三)
收件人: "dev@iotdb.apache.org" <de...@iotdb.apache.org>
抄送:
主题: Re: Suggestions for new TsFile
Hi,
We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
The mean differences is as below:
1. Remove TsOffsetArray.
2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
The newer TsFile structure should be looked like:
Here is an example of how the new structure works.
When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
After that, we are able to deserialize all TimeseriesMetadata in “d0”.
Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
Thanks,
Haonan Hou
On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org> wrote:
Hi,
If each device only stores each offset of TimeseriesMetadata like this:
TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
}
It could be simplified to recording the start offset and end offset:
TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
0 }, {deviceId(d1), 3 }, … }
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
Hi,
Thank you for your reply.
I am very happy that you can take my suggestion.
Thanks
Dawei Liu
2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
Hi Dawei,
Thank you so much that you share your opinion about new TsFile!
I am very happy to take your suggestions.
You said we can remove TsOffsetArray and directly store the offset of
TimeseriesMetaData. I agree with you. It is better than my version.
Besides, for the optimization of TimeserieMetaData, I would like to
discuss with other people to determine which way is better.
Best,
Haonan Hou
On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
Hi,
I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
design.
TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
startIndex , endIndex of TsOffsetArray, it’s looks like :
TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
{deviceId(d1), [3,5] }, …. } }
We can delete TsOffsetArray and store the offsets directly in the
deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
to record . This change will save 4 bytes per device on disk, because every
device just need record the number of offsets and offsets. it’s looks like：
TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
}, … }
In addition, TimeSeriesMetaData is an ordered structure on the hard
disk, and the TimeSeriesMetaData for each device is linked together, so
TsFileMetaData does not need to store all offset information, so there two
optimization directions:
1. Save startTime , endTime and offset for each TimeSeriesMetaData in
TsFileMetaData. The nice thing about this is that when you read
TsFileMetaData from your hard drive, you can directly do a filter to filter
which TimeSeriesMetaData is not necessary to read.
2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
that you can loop through it and just need once to seek, it’s looks like :
TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
[1] https://github.com/apache/incubator-iotdb/pull/736 <
https://github.com/apache/incubator-iotdb/pull/736>
Thanks
Dawei Liu
Re: Suggestions for new TsFile
Posted by Haonan Hou <hh...@outlook.com>.
Hi,
We have a newer design of TsFile, which combines the suggestions from Jialin and Dawei.
The mean differences is as below:
1. Remove TsOffsetArray.
2. Modify the device map in TsFileMetaData to store the start offset of first TimeseriesMetadata and total data size of all TimeseriesMetadatas in each device.
The newer TsFile structure should be looked like:
[cid:6CECB108-BC0C-4EA1-B9D5-FC8A4885D520]
Here is an example of how the new structure works.
When we try to get List<ChunkMetadata> of Timeseries "d0.s1", first we deserialize the map in TsFileMetadata, and we have the startOffset of TimseriesMetadata “s0",
the first TimeseiresMetadata of “d0", and data size of all TimeseriesMetadatas in “d0".
After that, we are able to deserialize all TimeseriesMetadata in “d0”.
Finally we have the TimeseriesMetadata "d0.s1" and can get the ChunkMetadata List of "d0.s1".
Thanks,
Haonan Hou
On Feb 11, 2020, at 8:08 PM, Jialin Qiao <qi...@apache.org>> wrote:
Hi,
If each device only stores each offset of TimeseriesMetadata like this:
TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
}
It could be simplified to recording the start offset and end offset:
TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
0 }, {deviceId(d1), 3 }, … }
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
atoiLiu <at...@163.com>> 于2020年2月11日周二 下午7:59写道：
Hi,
Thank you for your reply.
I am very happy that you can take my suggestion.
Thanks
Dawei Liu
2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com>> 写道：
Hi Dawei,
Thank you so much that you share your opinion about new TsFile!
I am very happy to take your suggestions.
You said we can remove TsOffsetArray and directly store the offset of
TimeseriesMetaData. I agree with you. It is better than my version.
Besides, for the optimization of TimeserieMetaData, I would like to
discuss with other people to determine which way is better.
Best,
Haonan Hou
On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com>> wrote:
Hi,
I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
design.
TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
startIndex , endIndex of TsOffsetArray, it’s looks like :
TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
{deviceId(d1), [3,5] }, …. } }
We can delete TsOffsetArray and store the offsets directly in the
deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
to record . This change will save 4 bytes per device on disk, because every
device just need record the number of offsets and offsets. it’s looks like：
TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
}, … }
In addition, TimeSeriesMetaData is an ordered structure on the hard
disk, and the TimeSeriesMetaData for each device is linked together, so
TsFileMetaData does not need to store all offset information, so there two
optimization directions:
1. Save startTime , endTime and offset for each TimeSeriesMetaData in
TsFileMetaData. The nice thing about this is that when you read
TsFileMetaData from your hard drive, you can directly do a filter to filter
which TimeSeriesMetaData is not necessary to read.
2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
that you can loop through it and just need once to seek, it’s looks like :
TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
[1] https://github.com/apache/incubator-iotdb/pull/736 <
https://github.com/apache/incubator-iotdb/pull/736>
Thanks
Dawei Liu
Re: Suggestions for new TsFile
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
If each device only stores each offset of TimeseriesMetadata like this:
TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, …
}
It could be simplified to recording the start offset and end offset:
TsFileMetaData ---> [ {deviceId(d0), [0, 2] }, {deviceId(d1), [3,5] }, … }
And finally, it could be replaced by: TsFileMetaData ---> [ {deviceId(d0),
0 }, {deviceId(d1), 3 }, … }
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
atoiLiu <at...@163.com> 于2020年2月11日周二 下午7:59写道：
> Hi,
>
> Thank you for your reply.
> I am very happy that you can take my suggestion.
>
>
> Thanks
>
> Dawei Liu
>
>
> > 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
> >
> > Hi Dawei,
> >
> > Thank you so much that you share your opinion about new TsFile!
> > I am very happy to take your suggestions.
> >
> > You said we can remove TsOffsetArray and directly store the offset of
> TimeseriesMetaData. I agree with you. It is better than my version.
> > Besides, for the optimization of TimeserieMetaData, I would like to
> discuss with other people to determine which way is better.
> >
> > Best,
> >
> > Haonan Hou
> >
> >
> >> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
> >>
> >> Hi,
> >>
> >> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad
> design.
> >>
> >> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every
> offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record
> startIndex , endIndex of TsOffsetArray, it’s looks like :
> >>
> >> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] },
> {deviceId(d1), [3,5] }, …. } }
> >>
> >> We can delete TsOffsetArray and store the offsets directly in the
> deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>>
> to record . This change will save 4 bytes per device on disk, because every
> device just need record the number of offsets and offsets. it’s looks like：
> >>
> >> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5]
> }, … }
> >>
> >>
> >> In addition, TimeSeriesMetaData is an ordered structure on the hard
> disk, and the TimeSeriesMetaData for each device is linked together, so
> TsFileMetaData does not need to store all offset information, so there two
> optimization directions:
> >>
> >> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in
> TsFileMetaData. The nice thing about this is that when you read
> TsFileMetaData from your hard drive, you can directly do a filter to filter
> which TimeSeriesMetaData is not necessary to read.
> >>
> >>
> >> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so
> that you can loop through it and just need once to seek, it’s looks like :
> >>
> >> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
> >>
> >>
> >>
> >> [1] https://github.com/apache/incubator-iotdb/pull/736 <
> https://github.com/apache/incubator-iotdb/pull/736>
> >>
> >> Thanks
> >>
> >> Dawei Liu
> >
>
>
Re: Suggestions for new TsFile
Posted by atoiLiu <at...@163.com>.
Hi,
Thank you for your reply.
I am very happy that you can take my suggestion.
Thanks
Dawei Liu
> 2020年2月11日 下午6:04，Haonan Hou <hh...@outlook.com> 写道：
>
> Hi Dawei,
>
> Thank you so much that you share your opinion about new TsFile!
> I am very happy to take your suggestions.
>
> You said we can remove TsOffsetArray and directly store the offset of TimeseriesMetaData. I agree with you. It is better than my version.
> Besides, for the optimization of TimeserieMetaData, I would like to discuss with other people to determine which way is better.
>
> Best,
>
> Haonan Hou
>
>
>> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>>
>> Hi,
>>
>> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad design.
>>
>> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record startIndex , endIndex of TsOffsetArray, it’s looks like :
>>
>> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] }, {deviceId(d1), [3,5] }, …. } }
>>
>> We can delete TsOffsetArray and store the offsets directly in the deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>> to record . This change will save 4 bytes per device on disk, because every device just need record the number of offsets and offsets. it’s looks like：
>>
>> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, … }
>>
>>
>> In addition, TimeSeriesMetaData is an ordered structure on the hard disk, and the TimeSeriesMetaData for each device is linked together, so TsFileMetaData does not need to store all offset information, so there two optimization directions:
>>
>> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in TsFileMetaData. The nice thing about this is that when you read TsFileMetaData from your hard drive, you can directly do a filter to filter which TimeSeriesMetaData is not necessary to read.
>>
>>
>> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so that you can loop through it and just need once to seek, it’s looks like :
>>
>> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>>
>>
>>
>> [1] https://github.com/apache/incubator-iotdb/pull/736 <https://github.com/apache/incubator-iotdb/pull/736>
>>
>> Thanks
>>
>> Dawei Liu
>
Re: Suggestions for new TsFile
Posted by Haonan Hou <hh...@outlook.com>.
Hi Dawei,
Thank you so much that you share your opinion about new TsFile!
I am very happy to take your suggestions.
You said we can remove TsOffsetArray and directly store the offset of TimeseriesMetaData. I agree with you. It is better than my version.
Besides, for the optimization of TimeserieMetaData, I would like to discuss with other people to determine which way is better.
Best,
Haonan Hou
> On Feb 11, 2020, at 5:35 PM, atoiLiu <at...@163.com> wrote:
>
> Hi,
>
> I’m learning new TsFile in PR [1], but I think TsFileMetaData has a bad design.
>
> TsFileMetaData has a TsOffsetArray, TsOffsetArray is record every offset of TimeseriesMetaData, and use Map<deviceId, int[]> to record startIndex , endIndex of TsOffsetArray, it’s looks like :
>
> TsFileMetaData —>{ [0,1,2,3,4,5, ….] [ {deviceId(d0), [0,2] }, {deviceId(d1), [3,5] }, …. } }
>
> We can delete TsOffsetArray and store the offsets directly in the deviceIndexArray, then TsFileMatadata will has a Map<deviceId, List<Long>> to record . This change will save 4 bytes per device on disk, because every device just need record the number of offsets and offsets. it’s looks like：
>
> TsFileMetaData ---> [ {deviceId(d0), [0,1,2] }, {deviceId(d1), [3,4,5] }, … }
>
>
> In addition, TimeSeriesMetaData is an ordered structure on the hard disk, and the TimeSeriesMetaData for each device is linked together, so TsFileMetaData does not need to store all offset information, so there two optimization directions:
>
> 1. Save startTime , endTime and offset for each TimeSeriesMetaData in TsFileMetaData. The nice thing about this is that when you read TsFileMetaData from your hard drive, you can directly do a filter to filter which TimeSeriesMetaData is not necessary to read.
>
>
> 2. Only save the start TimeSeriesMetaData offset in TsFileMetaData so that you can loop through it and just need once to seek, it’s looks like :
>
> TsFileMetaData ---> [ {deviceId(d0), 0 }, {deviceId(d1), 3 }, … }
>
>
>
> [1] https://github.com/apache/incubator-iotdb/pull/736 <https://github.com/apache/incubator-iotdb/pull/736>
>
> Thanks
>
> Dawei Liu