You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by 林欣欣 <li...@163.com> on 2021/11/27 10:38:46 UTC
The question of Edge-Cloud synchronization
Hi everyone,
I am a graduate student who is studying edge computing. I have two very doubtable questions to consult everyone.
First, I want to know why the IoTDB chose to use Tsfile’s data file incremental upload over point-by-point write during IoTDB’s cloud synchronization, I understand this for two reasons: 1. The Tsfile file transfer reduces the amount of data transferred because of file consolidation and data compression; 2. Files sent to the cloud are organized between internal data and files, reducing duplication of work in the cloud. I don’t know if this is right？
Second, based on the current data file uploads, would it be okay, or would it have to wait a while, to synchronize all the Edge database data (the latest unwrapped data) in the cloud?
Can I understand that the current IoTDB synchronization module in the edge cloud is the basic phase? It can only synchronize data over a period of time, not the latest data?
Hope to get your reply and help, thank you.
Lin,
Re:Re: Re: The question of Edge-Cloud synchronization
Posted by 林欣欣 <li...@163.com>.
OK，thank you, I will join it
在 2021-11-28 01:41:03，"Jialin Qiao" <qi...@apache.org> 写道：
>Hi,
>
>The order should be maintained by the sender. Actually, the TsFile-sync is
>also redesigned now, it didn't consider the compaction and deletion
>previously.
>Welcome to join the design of the new sync module. We will put the design
>doc in the confluence(@Sicheng):
>https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177051872
>
>Thanks,
>—————————————————
>Jialin Qiao
>School of Software, Tsinghua University
>
>乔嘉林
>清华大学 软件学院
>
>
>林欣欣 <li...@163.com> 于2021年11月27日周六 下午11:18写道：
>
>> thank you, and I want to know Is the TsFile file transferred to the cloud
>> in order?How to achieve order?Is it because the sender pipelining job loads
>> the file into the send queue or the receiver reprocesses the order in which
>> the files are received？
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> At 2021-11-27 20:49:58, "Jialin Qiao" <qi...@apache.org> wrote:
>> >Hi,
>> >
>> >>First, I want to know why the IoTDB chose to use Tsfile’s data file
>> >incremental upload over point-by-point write during IoTDB’s cloud
>> >synchronization, I understand this for two reasons: 1. The Tsfile file
>> >transfer reduces the amount of data transferred because of file
>> >consolidation and data compression; 2. Files sent to the cloud are
>> >organized between internal data and files, reducing duplication of work in
>> >the cloud. I don’t know if this is right？
>> >
>> >You are right :)
>> >
>> >> Second, based on the current data file uploads, would it be okay, or
>> >would it have to wait a while, to synchronize all the Edge database data
>> >(the latest unwrapped data) in the cloud?
>> >It can only synchronize data over a period of time, not the latest data.
>> >
>> >Yes, the TsFile-sync has some delay.
>> >
>> >To synchronize the latest data, we plan to add a new synchronization based
>> >on log, which has low latency but high overhead.
>> >
>> >Thanks,
>> >—————————————————
>> >Jialin Qiao
>> >School of Software, Tsinghua University
>> >
>> >乔嘉林
>> >清华大学 软件学院
>> >
>> >
>> >林欣欣 <li...@163.com> 于2021年11月27日周六 下午6:38写道：
>> >
>> >> Hi everyone,
>> >> I am a graduate student who is studying edge computing. I have two
>> >> very doubtable questions to consult everyone.
>> >> First, I want to know why the IoTDB chose to use Tsfile’s data
>> file
>> >> incremental upload over point-by-point write during IoTDB’s cloud
>> >> synchronization, I understand this for two reasons: 1. The Tsfile file
>> >> transfer reduces the amount of data transferred because of file
>> >> consolidation and data compression; 2. Files sent to the cloud are
>> >> organized between internal data and files, reducing duplication of work
>> in
>> >> the cloud. I don’t know if this is right？
>> >> Second, based on the current data file uploads, would it be okay,
>> or
>> >> would it have to wait a while, to synchronize all the Edge database data
>> >> (the latest unwrapped data) in the cloud?
>> >> Can I understand that the current IoTDB synchronization module in
>> >> the edge cloud is the basic phase? It can only synchronize data over a
>> >> period of time, not the latest data?
>> >>
>> >>
>> >>
>> >> Hope to get your reply and help, thank you.
>> >>
>> >>
>> >>
>> >>
>> >> Lin,
>>
Re: Re: The question of Edge-Cloud synchronization
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
The order should be maintained by the sender. Actually, the TsFile-sync is
also redesigned now, it didn't consider the compaction and deletion
previously.
Welcome to join the design of the new sync module. We will put the design
doc in the confluence(@Sicheng):
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177051872
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
林欣欣 <li...@163.com> 于2021年11月27日周六 下午11:18写道：
> thank you, and I want to know Is the TsFile file transferred to the cloud
> in order?How to achieve order?Is it because the sender pipelining job loads
> the file into the send queue or the receiver reprocesses the order in which
> the files are received？
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2021-11-27 20:49:58, "Jialin Qiao" <qi...@apache.org> wrote:
> >Hi,
> >
> >>First, I want to know why the IoTDB chose to use Tsfile’s data file
> >incremental upload over point-by-point write during IoTDB’s cloud
> >synchronization, I understand this for two reasons: 1. The Tsfile file
> >transfer reduces the amount of data transferred because of file
> >consolidation and data compression; 2. Files sent to the cloud are
> >organized between internal data and files, reducing duplication of work in
> >the cloud. I don’t know if this is right？
> >
> >You are right :)
> >
> >> Second, based on the current data file uploads, would it be okay, or
> >would it have to wait a while, to synchronize all the Edge database data
> >(the latest unwrapped data) in the cloud?
> >It can only synchronize data over a period of time, not the latest data.
> >
> >Yes, the TsFile-sync has some delay.
> >
> >To synchronize the latest data, we plan to add a new synchronization based
> >on log, which has low latency but high overhead.
> >
> >Thanks,
> >—————————————————
> >Jialin Qiao
> >School of Software, Tsinghua University
> >
> >乔嘉林
> >清华大学 软件学院
> >
> >
> >林欣欣 <li...@163.com> 于2021年11月27日周六 下午6:38写道：
> >
> >> Hi everyone,
> >> I am a graduate student who is studying edge computing. I have two
> >> very doubtable questions to consult everyone.
> >> First, I want to know why the IoTDB chose to use Tsfile’s data
> file
> >> incremental upload over point-by-point write during IoTDB’s cloud
> >> synchronization, I understand this for two reasons: 1. The Tsfile file
> >> transfer reduces the amount of data transferred because of file
> >> consolidation and data compression; 2. Files sent to the cloud are
> >> organized between internal data and files, reducing duplication of work
> in
> >> the cloud. I don’t know if this is right？
> >> Second, based on the current data file uploads, would it be okay,
> or
> >> would it have to wait a while, to synchronize all the Edge database data
> >> (the latest unwrapped data) in the cloud?
> >> Can I understand that the current IoTDB synchronization module in
> >> the edge cloud is the basic phase? It can only synchronize data over a
> >> period of time, not the latest data?
> >>
> >>
> >>
> >> Hope to get your reply and help, thank you.
> >>
> >>
> >>
> >>
> >> Lin,
>
Re:Re: The question of Edge-Cloud synchronization
Posted by 林欣欣 <li...@163.com>.
thank you, and I want to know Is the TsFile file transferred to the cloud in order?How to achieve order?Is it because the sender pipelining job loads the file into the send queue or the receiver reprocesses the order in which the files are received？
At 2021-11-27 20:49:58, "Jialin Qiao" <qi...@apache.org> wrote:
>Hi,
>
>>First, I want to know why the IoTDB chose to use Tsfile’s data file
>incremental upload over point-by-point write during IoTDB’s cloud
>synchronization, I understand this for two reasons: 1. The Tsfile file
>transfer reduces the amount of data transferred because of file
>consolidation and data compression; 2. Files sent to the cloud are
>organized between internal data and files, reducing duplication of work in
>the cloud. I don’t know if this is right？
>
>You are right :)
>
>> Second, based on the current data file uploads, would it be okay, or
>would it have to wait a while, to synchronize all the Edge database data
>(the latest unwrapped data) in the cloud?
>It can only synchronize data over a period of time, not the latest data.
>
>Yes, the TsFile-sync has some delay.
>
>To synchronize the latest data, we plan to add a new synchronization based
>on log, which has low latency but high overhead.
>
>Thanks,
>—————————————————
>Jialin Qiao
>School of Software, Tsinghua University
>
>乔嘉林
>清华大学 软件学院
>
>
>林欣欣 <li...@163.com> 于2021年11月27日周六 下午6:38写道：
>
>> Hi everyone,
>> I am a graduate student who is studying edge computing. I have two
>> very doubtable questions to consult everyone.
>> First, I want to know why the IoTDB chose to use Tsfile’s data file
>> incremental upload over point-by-point write during IoTDB’s cloud
>> synchronization, I understand this for two reasons: 1. The Tsfile file
>> transfer reduces the amount of data transferred because of file
>> consolidation and data compression; 2. Files sent to the cloud are
>> organized between internal data and files, reducing duplication of work in
>> the cloud. I don’t know if this is right？
>> Second, based on the current data file uploads, would it be okay, or
>> would it have to wait a while, to synchronize all the Edge database data
>> (the latest unwrapped data) in the cloud?
>> Can I understand that the current IoTDB synchronization module in
>> the edge cloud is the basic phase? It can only synchronize data over a
>> period of time, not the latest data?
>>
>>
>>
>> Hope to get your reply and help, thank you.
>>
>>
>>
>>
>> Lin,
Re: The question of Edge-Cloud synchronization
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
>First, I want to know why the IoTDB chose to use Tsfile’s data file
incremental upload over point-by-point write during IoTDB’s cloud
synchronization, I understand this for two reasons: 1. The Tsfile file
transfer reduces the amount of data transferred because of file
consolidation and data compression; 2. Files sent to the cloud are
organized between internal data and files, reducing duplication of work in
the cloud. I don’t know if this is right？
You are right :)
> Second, based on the current data file uploads, would it be okay, or
would it have to wait a while, to synchronize all the Edge database data
(the latest unwrapped data) in the cloud?
It can only synchronize data over a period of time, not the latest data.
Yes, the TsFile-sync has some delay.
To synchronize the latest data, we plan to add a new synchronization based
on log, which has low latency but high overhead.
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
林欣欣 <li...@163.com> 于2021年11月27日周六 下午6:38写道：
> Hi everyone,
> I am a graduate student who is studying edge computing. I have two
> very doubtable questions to consult everyone.
> First, I want to know why the IoTDB chose to use Tsfile’s data file
> incremental upload over point-by-point write during IoTDB’s cloud
> synchronization, I understand this for two reasons: 1. The Tsfile file
> transfer reduces the amount of data transferred because of file
> consolidation and data compression; 2. Files sent to the cloud are
> organized between internal data and files, reducing duplication of work in
> the cloud. I don’t know if this is right？
> Second, based on the current data file uploads, would it be okay, or
> would it have to wait a while, to synchronize all the Edge database data
> (the latest unwrapped data) in the cloud?
> Can I understand that the current IoTDB synchronization module in
> the edge cloud is the basic phase? It can only synchronize data over a
> period of time, not the latest data?
>
>
>
> Hope to get your reply and help, thank you.
>
>
>
>
> Lin,