You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/01/11 03:00:32 UTC
[GitHub] [iotdb] limeng32 opened a new pull request #4764: Restructure export/import csv tool;
limeng32 opened a new pull request #4764:
URL: https://github.com/apache/iotdb/pull/4764
Hello, everyone. I recently used iotdb in the production environment and found that the import tool is almost unusable in the real scene. I studied this piece of code and found that the stream was not used correctly, so I refactored it, and I also added some other functions. What I do is as follows:
1. Optimize the implementation of import, so that the records in the backup file can be read into the memory, written to the database, and moved out of the memory line by line. The new tool no longer has any requirements on the size of the backup file;
2. Added compression options. It is a common behavior to compress while exporting. At the same time, importing from compressed files should also be supported. At present, I have implemented it in two ways: gzip and snappy;
3. There is corresponding progress feedback when importing/exporting. At present, I use feedback every time I export 10 million points or import 1 million points, so that no matter what type of data, I can have a suitable feedback rhythm;
Because the original code is almost completely unused, the new tools are named export-csv-new and import-csv-new, which coexist with existing tools;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] limeng32 closed pull request #4764: [Discussions-4763]Restructure export/import csv tool;
Posted by GitBox <gi...@apache.org>.
limeng32 closed pull request #4764:
URL: https://github.com/apache/iotdb/pull/4764
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] limeng32 commented on pull request #4764: [Discussions-4763]Restructure export/import csv tool;
Posted by GitBox <gi...@apache.org>.
limeng32 commented on pull request #4764:
URL: https://github.com/apache/iotdb/pull/4764#issuecomment-1014989905
see https://github.com/apache/iotdb/pull/4774
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org