You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by liuyong <li...@163.com> on 2024/01/05 13:43:08 UTC

About Python interface of tsfile module

Hi, has the tsfile submodule planned to provide the native interface for prevalent analytical frameworks or programming languages? Especially, the Python community. The Python community consists of many packages powerful on time series data analysis, such as the sktime for Machine Learning and PyTorch for Deep Learning, which highly relies on training on a huge amount of time series data.


As we develop the native AI engine for IoTDB (AINode). We find that the tsfile, which takes advantage of the fast reading and dense compression of tsfile organization, may further benefit the incorporation of the DB and AI System with authority management and free from numerous and costly data transfers compared with loading from the local file system. And there are still some open problems as loading the data for training deep models in iterations and inference of the data as the stream, which pose novel challenges to the operator supported by the native interface.



Perhaps a good starting point would be to implement a PyTorch-like dataloader, where the data source could support the tsfile.




Re: About Python interface of tsfile module

Posted by Yuan Tian <ja...@gmail.com>.
Hi yong,

I think we may start to do it after we release our first version of tsfile
project and tsfile-python should be in another separated repo.

On Fri, Jan 5, 2024 at 9:43 PM liuyong <li...@163.com> wrote:

> Hi, has the tsfile submodule planned to provide the native interface for
> prevalent analytical frameworks or programming languages? Especially, the
> Python community. The Python community consists of many packages powerful
> on time series data analysis, such as the sktime for Machine Learning and
> PyTorch for Deep Learning, which highly relies on training on a huge amount
> of time series data.
>
>
> As we develop the native AI engine for IoTDB (AINode). We find that the
> tsfile, which takes advantage of the fast reading and dense compression of
> tsfile organization, may further benefit the incorporation of the DB and AI
> System with authority management and free from numerous and costly data
> transfers compared with loading from the local file system. And there are
> still some open problems as loading the data for training deep models in
> iterations and inference of the data as the stream, which pose novel
> challenges to the operator supported by the native interface.
>
>
>
> Perhaps a good starting point would be to implement a PyTorch-like
> dataloader, where the data source could support the tsfile.
>
>
>
>