You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/01/11 08:52:32 UTC
[GitHub] [iotdb] limeng32 opened a new pull request #4774: Improve export/import csv tool;
limeng32 opened a new pull request #4774:
URL: https://github.com/apache/iotdb/pull/4774
Hello, everyone. I recently used iotdb in the production environment and found that the import tool is almost unusable in the real scene. I studied this piece of code and found that the stream was not used correctly, so I refactored it, and I also added some other functions. What I do is as follows:
Optimize the implementation of import, so that the records in the backup file can be read into the memory, written to the database, and moved out of the memory line by line. The new tool no longer has any requirements on the size of the backup file;
Added compression options. It is a common behavior to compress while exporting. At the same time, importing from compressed files should also be supported. At present, I have implemented it in two ways: gzip and snappy;
There is corresponding progress feedback when importing/exporting. At present, I use feedback every time I export 10 million points or import 1 million points, so that no matter what type of data, I can have a suitable feedback rhythm;
Because the original code is almost completely unused, the new tools are named export-csv-new and import-csv-new, which coexist with existing tools;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] limeng32 closed pull request #4774: [Discussions-4763]Improve export/import csv tool;
Posted by GitBox <gi...@apache.org>.
limeng32 closed pull request #4774:
URL: https://github.com/apache/iotdb/pull/4774
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4774: [Discussions-4763]Improve export/import csv tool;
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4774:
URL: https://github.com/apache/iotdb/pull/4774#issuecomment-1009794813
[](https://coveralls.io/builds/46063399)
Coverage increased (+0.003%) to 67.75% when pulling **99b86ef992c29576023aff21c13c037e5691afb8 on limeng32:master** into **ce154b8d2e1cb3d723168b8fc2b103d223c39d8d on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #4774: [Discussions-4763]Improve export/import csv tool;
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #4774:
URL: https://github.com/apache/iotdb/pull/4774#issuecomment-1009794813
[](https://coveralls.io/builds/45523656)
Coverage increased (+0.03%) to 67.655% when pulling **40d953b946b346c90456cc895784565194779de7 on limeng32:master** into **a2dba1b81e6962e4d31315e75cb4136546948463 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org