You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Zesong Sun <sa...@foxmail.com> on 2019/09/08 12:24:09 UTC
回复：Enable to choose storage in local file system or HDFS
Hi,
I had intended to implement this requirement in the first way, but now I think the second and third are better for much less modification of current codes... Though the first way may support more than HDFS storage, it may still take a lot more time to modify codes in TsFile module based on current work.
BR,
------------------
Zesong Sun
School of Software, Tsinghua University
孙泽嵩
清华大学 软件学院
------------------ 原始邮件 ------------------
发件人: "Jialin Qiao"<qj...@mails.tsinghua.edu.cn>;
发送时间: 2019年9月8日(星期天) 下午2:08
收件人: "dev"<de...@iotdb.apache.org>;
主题: Enable to choose storage in local file system or HDFS
Hi,
This issue is to let user directly uses spark to read data in IoTDB for analyzing.
This function can be done in many ways in IoTDB:
(1) Storing all TsFiles (data files) and other files (system files, WALs) on HDFS, then use spark-tsfile to read TsFiles on HDFS.
(2) Storing only TsFiles on HDFS, and other files on local file system, then use spark-tsfile to read TsFiles on HDFS.
(3) Storing all files on local file system and let user use spark-iotdb-connector to read data from IoTDB, regardless where TsFiles store.
Personally, I prefer the second and the third. If we use the second way, do we need the FileFactory for all Files?
Best,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -----原始邮件-----
> 发件人: "Zesong Sun (Jira)" <ji...@apache.org>
> 发送时间: 2019-08-29 19:34:00 (星期四)
> 收件人: dev@iotdb.apache.org
> 抄送:
> 主题: [jira] [Created] (IOTDB-187) Enable to choose storage in local file system or HDFS
>
> Zesong Sun created IOTDB-187:
> --------------------------------
>
> Summary: Enable to choose storage in local file system or HDFS
> Key: IOTDB-187
> URL: https://issues.apache.org/jira/browse/IOTDB-187
> Project: Apache IoTDB
> Issue Type: Improvement
> Reporter: Zesong Sun
>
>
> Enable to choose storage in local file system or HDFS
> "is_hdfs_storage=false" by default
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.2#803003)
回复： Enable to choose storage in local file system or HDFS
Posted by Zesong Sun <sa...@foxmail.com>.
Hi,
Thanks for this helpful suggestion! I'll try it by implementing TsFileFactory and SystemFileFactory.
(They are factories to create files according to different File System, so I think the name doesn't need to be "...FileFSFactory"?)
BR,
------------------
Zesong Sun
School of Software, Tsinghua University
孙泽嵩
清华大学 软件学院
------------------ 原始邮件 ------------------
发件人: "Xiangdong Huang"<sa...@gmail.com>;
发送时间: 2019年9月9日(星期一) 凌晨0:03
收件人: "dev"<de...@iotdb.apache.org>;
主题: Re: Enable to choose storage in local file system or HDFS
Hi,
> do we need the FileFactory for all Files?
A solution is having two FileSystemUtil classes (or FileFactory),
TsFileFSFactory and the rest.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Zesong Sun <sa...@foxmail.com> 于2019年9月8日周日 上午5:24写道：
> Hi,
>
>
> I had intended to implement this requirement in the first way, but now I
> think the second and third are better for much less modification of current
> codes... Though the first way may support more than HDFS storage, it may
> still take a lot more time to modify codes in TsFile module based on
> current work.
>
>
>
>
> BR,
> ------------------
> Zesong Sun
> School of Software, Tsinghua University
>
> 孙泽嵩
> 清华大学 软件学院
>
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Jialin Qiao"<qj...@mails.tsinghua.edu.cn>;
> 发送时间: 2019年9月8日(星期天) 下午2:08
> 收件人: "dev"<de...@iotdb.apache.org>;
>
> 主题: Enable to choose storage in local file system or HDFS
>
>
>
> Hi,
>
> This issue is to let user directly uses spark to read data in IoTDB for
> analyzing.
>
> This function can be done in many ways in IoTDB:
>
> (1) Storing all TsFiles (data files) and other files (system files, WALs)
> on HDFS, then use spark-tsfile to read TsFiles on HDFS.
> (2) Storing only TsFiles on HDFS, and other files on local file system,
> then use spark-tsfile to read TsFiles on HDFS.
> (3) Storing all files on local file system and let user use
> spark-iotdb-connector to read data from IoTDB, regardless where TsFiles
> store.
>
> Personally, I prefer the second and the third. If we use the second way,
> do we need the FileFactory for all Files?
>
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -----原始邮件-----
> > 发件人: "Zesong Sun (Jira)" <ji...@apache.org>
> > 发送时间: 2019-08-29 19:34:00 (星期四)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: [jira] [Created] (IOTDB-187) Enable to choose storage in local file
> system or HDFS
> >
> > Zesong Sun created IOTDB-187:
> > --------------------------------
> >
> > Summary: Enable to choose storage in local file system or
> HDFS
> > Key: IOTDB-187
> > URL: https://issues.apache.org/jira/browse/IOTDB-187
> > Project: Apache IoTDB
> > Issue Type: Improvement
> > Reporter: Zesong Sun
> >
> >
> > Enable to choose storage in local file system or HDFS
> > "is_hdfs_storage=false" by default
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.2#803003)
Re: Enable to choose storage in local file system or HDFS
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
> do we need the FileFactory for all Files?
A solution is having two FileSystemUtil classes (or FileFactory),
TsFileFSFactory and the rest.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Zesong Sun <sa...@foxmail.com> 于2019年9月8日周日 上午5:24写道：
> Hi,
>
>
> I had intended to implement this requirement in the first way, but now I
> think the second and third are better for much less modification of current
> codes... Though the first way may support more than HDFS storage, it may
> still take a lot more time to modify codes in TsFile module based on
> current work.
>
>
>
>
> BR,
> ------------------
> Zesong Sun
> School of Software, Tsinghua University
>
> 孙泽嵩
> 清华大学 软件学院
>
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Jialin Qiao"<qj...@mails.tsinghua.edu.cn>;
> 发送时间: 2019年9月8日(星期天) 下午2:08
> 收件人: "dev"<de...@iotdb.apache.org>;
>
> 主题: Enable to choose storage in local file system or HDFS
>
>
>
> Hi,
>
> This issue is to let user directly uses spark to read data in IoTDB for
> analyzing.
>
> This function can be done in many ways in IoTDB:
>
> (1) Storing all TsFiles (data files) and other files (system files, WALs)
> on HDFS, then use spark-tsfile to read TsFiles on HDFS.
> (2) Storing only TsFiles on HDFS, and other files on local file system,
> then use spark-tsfile to read TsFiles on HDFS.
> (3) Storing all files on local file system and let user use
> spark-iotdb-connector to read data from IoTDB, regardless where TsFiles
> store.
>
> Personally, I prefer the second and the third. If we use the second way,
> do we need the FileFactory for all Files?
>
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -----原始邮件-----
> > 发件人: "Zesong Sun (Jira)" <ji...@apache.org>
> > 发送时间: 2019-08-29 19:34:00 (星期四)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: [jira] [Created] (IOTDB-187) Enable to choose storage in local file
> system or HDFS
> >
> > Zesong Sun created IOTDB-187:
> > --------------------------------
> >
> > Summary: Enable to choose storage in local file system or
> HDFS
> > Key: IOTDB-187
> > URL: https://issues.apache.org/jira/browse/IOTDB-187
> > Project: Apache IoTDB
> > Issue Type: Improvement
> > Reporter: Zesong Sun
> >
> >
> > Enable to choose storage in local file system or HDFS
> > "is_hdfs_storage=false" by default
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.2#803003)