You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Xiangdong Huang <sa...@gmail.com> on 2020/09/29 03:50:23 UTC
Re: [jira] [Created] (IOTDB-923) [Distributed] Sync tool to support
edge send data to cloud
Hi,
Any ideas? Send to some node and let the node partition the data file and
then forward to other nodes?
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
WangChao (Jira) <ji...@apache.org> 于2020年9月28日周一 上午10:15写道：
> WangChao created IOTDB-923:
> ------------------------------
>
> Summary: [Distributed] Sync tool to support edge send data to
> cloud
> Key: IOTDB-923
> URL: https://issues.apache.org/jira/browse/IOTDB-923
> Project: Apache IoTDB
> Issue Type: New Feature
> Components: Core/Cluster
> Reporter: WangChao
>
>
> Now we have sync tool to support edge send data to cloud in single mode.
>
> We also need support sync tool in cluster version.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>
Re: [jira] [Created] (IOTDB-923) [Distributed] Sync tool to
support edge send data to cloud
Posted by Houliang Qi <ne...@163.com>.
Hi,
Some supplements：
For sender：
The leader of the DataGroup is responsible for sending data to the remote;
The sender just need to know the address of the remote cluster nodes, the routing information should be transparent to the sender;
For follower:
Provide the interface of sync TsFile in session api;
Forward the TsFile to the node that should manage the TsFile(we can cache the routing information in the session of the client);
> I think we could redesign the sync tool to support sync data to the formats of other storage system
Big +1, if we can provide tools to transfer TsFile format to other file format(CSV、Parquet、Avro), this will make it easier for other OLAP to analyze TsFile and enrich our ecosystem.
Thanks,
---------------------------------------
Houliang Qi
On 09/30/2020 10:39，runhuster@foxmail.com<ru...@foxmail.com> wrote：
Hi,
here are my opinions:
1. the Leader of each DataGroup should sync data to remote, and ask the destination node of the data file before sending the data.
2. the remote IoTDB tells the destination of the data file to the edge IoTDB.
3. the remote Leader of the DataGroup receive the data file and sync to the follower.
And,
I think we could redesign the sync tool to support sync data to the formats of other storage system.
This will enrich our ecosystem.
Thanks.
Chao Wang
BONC Ltd
From: Xiangdong Huang
Date: 2020-09-29 11:50
To: dev
CC: notifications
Subject: Re: [jira] [Created] (IOTDB-923) [Distributed] Sync tool to support edge send data to cloud
Hi,
Any ideas? Send to some node and let the node partition the data file and
then forward to other nodes?
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
WangChao (Jira) <ji...@apache.org> 于2020年9月28日周一 上午10:15写道：
WangChao created IOTDB-923:
------------------------------
Summary: [Distributed] Sync tool to support edge send data to
cloud
Key: IOTDB-923
URL: https://issues.apache.org/jira/browse/IOTDB-923
Project: Apache IoTDB
Issue Type: New Feature
Components: Core/Cluster
Reporter: WangChao
Now we have sync tool to support edge send data to cloud in single mode.
We also need support sync tool in cluster version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Re: [jira] [Created] (IOTDB-923) [Distributed] Sync tool to support edge send data to cloud
Posted by "runhuster@foxmail.com" <ru...@foxmail.com>.
Hi,
here are my opinions:
1. the Leader of each DataGroup should sync data to remote, and ask the destination node of the data file before sending the data.
2. the remote IoTDB tells the destination of the data file to the edge IoTDB.
3. the remote Leader of the DataGroup receive the data file and sync to the follower.
And,
I think we could redesign the sync tool to support sync data to the formats of other storage system.
This will enrich our ecosystem.
Thanks.
Chao Wang
BONC Ltd
From: Xiangdong Huang
Date: 2020-09-29 11:50
To: dev
CC: notifications
Subject: Re: [jira] [Created] (IOTDB-923) [Distributed] Sync tool to support edge send data to cloud
Hi,
Any ideas? Send to some node and let the node partition the data file and
then forward to other nodes?
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
WangChao (Jira) <ji...@apache.org> 于2020年9月28日周一 上午10:15写道：
> WangChao created IOTDB-923:
> ------------------------------
>
> Summary: [Distributed] Sync tool to support edge send data to
> cloud
> Key: IOTDB-923
> URL: https://issues.apache.org/jira/browse/IOTDB-923
> Project: Apache IoTDB
> Issue Type: New Feature
> Components: Core/Cluster
> Reporter: WangChao
>
>
> Now we have sync tool to support edge send data to cloud in single mode.
>
> We also need support sync tool in cluster version.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>