Re: [Proposal] Load Tsfile Functionality Optimisation

[ITAMI SHO <sh...@hotmail.com>]

Hi Xiangzhi,

Thank you for sharing your work on the load tsfile feature. I am currently working on the Load function as well, and I have a question for you. Have you considered enhancing the Load function directly within the core code, rather than relying on the script to load tsfiles one by one? Additionally , do you think there would be a significant performance difference between the original batch load method and your script of loading files individually?

Your insights on this matter would be very valuable, and I believe it could lead to further optimizations!

Best regards,
Itami Sho

> 2024年5月30日 11:11，孟祥志 <xi...@timecho.com> 写道：
> 
> Hello everyone,
> 
> I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share with you a feature that I have been working on for the past few months.
> 
> load tsfile has always been an important feature for IoTDB, but due to the original design, he has some flaws in it, when load a folder fails to import some tsfile, it may cause the whole task fails, which is obviously a not very user-friendly design, so I decided to modify his.
> 
> I'll modify the original load tsfile script to traverse the entire folder first, find all the files, and then use the LOAD function to import individual files; and I've added a few interesting changes along the way, modifying the successful operation parameter to make it easier for people to deal with successful files, and added a parameter for failed files to make it easier for people to find those failed files, and finally added a new number of threads to make the LOAD function work more quickly.
> 
> 
> I hope you are interested in this feature and would like to participate in the development and testing. You can also leave your comments and suggestions in this thread. Appreciate any suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> Xiangzhi Meng
> 
> Reference:
> https://github.com/apache/iotdb/pull/12541