You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Xiangdong Huang <sa...@gmail.com> on 2019/06/26 08:07:32 UTC
Re: Operation and robustness of iotDB
Hi,
If someone uses NativeRestorableIOWriter, please notice this PR [1].
The behavior of the class changes a little:
1. if append == false, the writer will be closed after it is created.
a. if the tsfile is broken, the behavior of the class is fixing the
broken tsfile and make it sealed.
b. if the tsfile is completed, the class does nothing.
2. if append == true, the writer will truncate some data:
a. if the tsfile is broken, then the broken data will be truncated;
(And, this PR changes the strategy: the class will try its best to save
correct data)
b. if the tsfile is completed, then the tail of the file (FileMetadata)
will be truncated.
Therefore, if append == true, you must call close() in the end of your
program; If append == false, you can call close() or not.
[1] https://github.com/apache/incubator-iotdb/pull/188
(the PR does not pass the UT now..)
best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Xiangdong Huang <sa...@gmail.com> 于2019年3月9日周六 下午1:07写道：
> Hi,
>
> This PR has opened for several days. I will merge this PR into the master.
>
> By the way, I notice that there is no document about TsFile API. Actually
> we have previous versions about the API introduction and example on
> https://github.com/thulab/tsfile/wiki , but I am not sure whether they
> catch up with the latest version of API...
>
> I open a JIRA ticket, https://issues.apache.org/jira/browse/IOTDB-40 .
>
> Best,
>
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Xiangdong Huang <sa...@gmail.com> 于2019年3月6日周三 上午12:53写道：
>
>> Hi,
>>
>> I have added a new TsFileIOWriter, which supports to recover data from a
>> broken TsFile (I mean, a TsFile that is not closed correctly). The PR is
>> https://github.com/apache/incubator-iotdb/pull/87.
>>
>> You can use the new feature by:
>> ```
>> NativeRestorableIOWriter rWriter = new NativeRestorableIOWriter(file);
>> TsFileWriter writer = new TsFileWriter(rWriter);
>> ```
>> 1. If the file is a complete TsFile (e.g., the FileMetadata has been
>> persistent into the file and the tail magic string is correct), then you
>> can not write data into this TsFile anymore (actually I can remove all
>> FileMetadata and then let you continue to write, but it needs more coding
>> work...), `TsFileWriter writer = new TsFileWriter(rWriter);` will throw a
>> IoException. You can check it before you create TsFileWriter by using
>> `rWriter.canWrite()`.
>>
>> 2. ChunkGroup is the basic unit that can be recover. That is, if a
>> ChunkGroup is complete, i.e., its chunkGroupFooter is complete, I will
>> remain it, otherwise I will truncate the chunk group. However, I can not
>> know whether a chunkGroupFooter is complete (if only one byte lose, the
>> deserialize method does not throw exception, because something like
>> `deserializeToInt(inputstream)` does not throw exception (it will use 3
>> bytes to convert them to a integer). So, when I say "if a ChunkGroup is
>> complete", I mean that not only its chunkGroupFooter is serialized
>> successfully, but also there is at least one more byte is serialized into
>> the file followed the footer.
>>
>> 3. I have tested many cases, such only a magic head string is persisted,
>> only a ChunkHeader is persisted, the first ChunkHeader is not persisted
>> completely, only some Chunks are persisted, a ChunkGroupFooter is
>> persisted, Two ChunkGroupFooter are persisted, and a complete TsFile is
>> persisted, etc.. (`NativeRestorableIOWriterTest.java`, hope I have cover
>> all cases)
>>
>> Hope TsFile can be used in edge device management applications ASAP!
>>
>> Now please enjoy the new feature.
>>
>> -----------------------------------
>> Xiangdong Huang
>> School of Software, Tsinghua University
>>
>> 黄向东
>> 清华大学 软件学院
>>
>>
>> Julian Feinauer <j....@pragmaticminds.de> 于2019年3月5日周二 下午8:10写道：
>>
>>> Hey,
>>>
>>> thank you fort he link... I did not know of this.. this is exactly what
>>> I was looking for!
>>>
>>> Julian
>>>
>>> PS.: Looking forward to your PR : )
>>>
>>> Am 05.03.19, 12:26 schrieb "Xiangdong Huang" <sa...@gmail.com>:
>>>
>>> Hi,
>>>
>>> 1. We have a document to introduce that:
>>> https://cwiki.apache.org/confluence/display/IOTDB/TsFile+Format
>>>
>>> 2. The new API for recovering data is almost done. I am writing the
>>> UTs
>>> now. Maybe I can submit a PR tonight (if everything is fine...)
>>>
>>> Best,
>>> -----------------------------------
>>> Xiangdong Huang
>>> School of Software, Tsinghua University
>>>
>>> 黄向东
>>> 清华大学 软件学院
>>>
>>>
>>> Julian Feinauer <j....@pragmaticminds.de> 于2019年3月5日周二
>>> 下午6:00写道：
>>>
>>> > Hi Xiangdong,
>>> >
>>> > that sounds excellent.
>>> > Do you have a short overview of how the file format is designed on
>>> disk?
>>> > I know that its somewhat similar to parquet but I did not find more
>>> > details.
>>> > Basically what would suffice for us would be something like
>>> skipping an
>>> > invalid column group (or how you name it) and go on with the next,
>>> or so.
>>> >
>>> > Julian
>>> >
>>> > Am 04.03.19, 13:21 schrieb "Xiangdong Huang" <sa...@gmail.com>:
>>> >
>>> > Hi,
>>> >
>>> > If so, I think I need to add a new API to allow you continue
>>> to write
>>> > data
>>> > in an existing but not closed correctly TsFile. Then
>>> everything is
>>> > fine
>>> > for you :D
>>> >
>>> > Best,
>>> > -----------------------------------
>>> > Xiangdong Huang
>>> > School of Software, Tsinghua University
>>> >
>>> > 黄向东
>>> > 清华大学 软件学院
>>> >
>>> >
>>> > Julian Feinauer <j....@pragmaticminds.de> 于2019年3月4日周一
>>> 下午8:08写道：
>>> >
>>> > > Hey Xiangdong,
>>> > >
>>> > > thanks for the great explanation.
>>> > > And in fact, I agree with you that it would be best if we
>>> start to
>>> > play
>>> > > around with it and reply all our findings or wishes back to
>>> this
>>> > list (in
>>> > > fact that proved to be beneficial in plc4x as well).
>>> > >
>>> > > You confirm my thoughts about the two "levels" of APIs (DB
>>> and file)
>>> > and
>>> > > the file api is exactly what we looked for for our use case.
>>> > > As we do not care much about data loss (when an edge device
>>> fails
>>> > its...
>>> > > gone).
>>> > > The crucial point for us is that no corrupt files can be
>>> generated.
>>> > > This means I'm fine when the last data submitted is lost but
>>> I'm not
>>> > fine
>>> > > if we can get to a situation where the last datafile is
>>> completely
>>> > lost
>>> > > (well, perhaps this could be acceptable).
>>> > >
>>> > > @tim: Perhaps its best when you give some more information to
>>> > Xiangdong
>>> > > about our idea, and we can also point to our current code in
>>> github
>>> > >
>>> > > Julian
>>> > >
>>> > > Am 04.03.19, 13:03 schrieb "Xiangdong Huang" <
>>> sainthxd@gmail.com>:
>>> > >
>>> > > Hi,
>>> > >
>>> > > TsFile API is not deprecated. In fact, it is designed
>>> for this
>>> > > scenario and
>>> > > MapReduce/Spark computing.
>>> > >
>>> > > If you just use Reader and Writer API, there is
>>> something you
>>> > need to
>>> > > know:
>>> > >
>>> > > Let's suppose your block size is x Bytes,
>>> > (tsfile-format.properties:
>>> > > group_size_in_byte).
>>> > >
>>> > > 1. If you write data and a shutdown occurs, then all
>>> data that is
>>> > > flushed
>>> > > on disk is ok, and you can read the data ( class
>>> > > org.apache.iotdb.tsfile.TsFileSequenceRead is an
>>> example, but
>>> > you need
>>> > > to
>>> > > change it a little. I think I can write an example.)
>>> > >
>>> > > 2. Actually, TsFile has the ability to allow you
>>> continue to
>>> > write
>>> > > data at
>>> > > the end of the incomplete file. However, We do not
>>> provide this
>>> > API
>>> > > now...
>>> > > If needed, I can add the API.
>>> > >
>>> > > 3. In this scenario, you will lose at most x Bytes data.
>>> If you
>>> > do not
>>> > > accept that, something like WAL is needed. (It is not
>>> very
>>> > complex,
>>> > > but I
>>> > > am not sure that whether it should be an embedded
>>> function for
>>> > TsFile).
>>> > >
>>> > > Up to now, we can consider that TsFile API is suitable
>>> for your
>>> > > scenario
>>> > > (even though we need to add a little more API if you
>>> desire).
>>> > And you
>>> > > can
>>> > > get the ability to compress data, and query data from
>>> the TsFile
>>> > rather
>>> > > than scan the data from the head to the tail.
>>> > >
>>> > > However, TsFile has one constraint: You can not write
>>> > out-of-order data
>>> > > into a TsFile, otherwise the query API may return
>>> incomplete
>>> > result.
>>> > > But I think it is ok for real applications, because I do
>>> not
>>> > think
>>> > > that a
>>> > > device can generate out-of-order data....
>>> > >
>>> > > For example, If you write two devices' data into one
>>> TsFile, it
>>> > is ok
>>> > > if
>>> > > you write data like:
>>> > > - d1.t1, d1.t2, d2.t1, d2.t2, d2.t3, d1.t4, d1.t5 ....
>>> > > or:
>>> > > - d1.m1.t1, d1.m1.t2, d1.m2.t1, d1.m2.t2, d2.m1.t1 ...
>>> > >
>>> > > But you can not write data like:
>>> > > - d1.m1.t2, d1.m1.t1 ...
>>> > >
>>> > > I think it is a good chance to improve TsFile to make it
>>> more
>>> > suitable
>>> > > for
>>> > > real applications, so please do not hesitate to tell me
>>> more
>>> > about
>>> > > what you
>>> > > think TsFile should want to have?
>>> > >
>>> > > Best,
>>> > > -----------------------------------
>>> > > Xiangdong Huang
>>> > > School of Software, Tsinghua University
>>> > >
>>> > > 黄向东
>>> > > 清华大学 软件学院
>>> > >
>>> > >
>>> > > Julian Feinauer <j....@pragmaticminds.de>
>>> 于2019年3月4日周一
>>> > 下午7:17写道：
>>> > >
>>> > > > Hi Xiangdong,
>>> > > >
>>> > > > thanks for the info.
>>> > > > How is it in the case when you use the Reader / Writer
>>> API for
>>> > the
>>> > > tsfiles
>>> > > > directly (or should this be considered "deprecated")?
>>> > > > Can these files come to corrupted state?
>>> > > >
>>> > > > One Situation where we have to deal with these
>>> situations is
>>> > "at the
>>> > > edge"
>>> > > > when we have devices inside large machines.
>>> > > > Usually at the end of the shift these machines (and
>>> therefore
>>> > our
>>> > > device)
>>> > > > is powered off hard, so no shutdown or
>>> de-initialization is
>>> > possible.
>>> > > >
>>> > > > Best
>>> > > > Julian
>>> > > >
>>> > > > Am 04.03.19, 12:14 schrieb "Xiangdong Huang" <
>>> > sainthxd@gmail.com>:
>>> > > >
>>> > > > Hi,
>>> > > >
>>> > > > IoTDB can support either on a server with 7*24 or a
>>> > RaspberryPi.
>>> > > We
>>> > > > have
>>> > > > tested both the two scenario.
>>> > > >
>>> > > > When you shutdown an IoTDB instance in force
>>> (e.g., power
>>> > off)
>>> > > and
>>> > > > restart
>>> > > > it again, no data loses ( if you enable the WAL).
>>> > > >
>>> > > > However, currently we do not optimize the time
>>> cost of the
>>> > > restart
>>> > > > process.
>>> > > > It is an important feature that we need to do,
>>> because we
>>> > hope
>>> > > IoTDB
>>> > > > can
>>> > > > support data management either on the edge devices
>>> or the
>>> > data
>>> > > center.
>>> > > >
>>> > > > And, the default configuration is not so suitable
>>> for
>>> > running on
>>> > > the
>>> > > > edge
>>> > > > device. (e.g., block size is 128MB, which is too
>>> large for
>>> > a
>>> > > > RaspberryPi,
>>> > > > and will slow down the restart process because
>>> there are
>>> > too
>>> > > much WAL
>>> > > > data
>>> > > > on disk).
>>> > > >
>>> > > > Best,
>>> > > > -----------------------------------
>>> > > > Xiangdong Huang
>>> > > > School of Software, Tsinghua University
>>> > > >
>>> > > > 黄向东
>>> > > > 清华大学 软件学院
>>> > > >
>>> > > >
>>> > > > Tim Mitsch <t....@pragmaticindustries.de>
>>> 于2019年3月4日周一
>>> > > 下午6:53写道：
>>> > > >
>>> > > > > Hello development-team
>>> > > > >
>>> > > > > First of all thanks for developing this kind of
>>> > interesting
>>> > > project
>>> > > > and
>>> > > > > bringing it into apache incubator.
>>> > > > >
>>> > > > > I have a question regarding the place of
>>> operation and
>>> > > robustness:
>>> > > > >
>>> > > > > * Is iotDB concepted as application on a
>>> server
>>> > which is
>>> > > running
>>> > > > 24/7
>>> > > > > or
>>> > > > > * Is it also possible to run it on a device
>>> like
>>> > > RaspberryPi or
>>> > > > IPC,
>>> > > > > where operation can interrupt.
>>> > > > > I’m asking because i’m searching for solution
>>> for a
>>> > temporary
>>> > > > storage that
>>> > > > > is robust against spontaneous interrupt, e.g.
>>> switch off
>>> > > electricity
>>> > > > > without regular shutdown of OS – have u tested
>>> something
>>> > like
>>> > > this
>>> > > > yet?
>>> > > > >
>>> > > > > Best regards
>>> > > > > Tim
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> >
>>> >
>>> >
>>>
>>>
>>>