You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by jackylk <gi...@git.apache.org> on 2018/03/01 03:20:34 UTC
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1825#discussion_r171454147
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java ---
@@ -239,24 +260,30 @@ private void notifyDataMapBlockEnd() {
blockletId = 0;
}
- private String constructFactFileFullPath() {
- String factFilePath =
- this.model.getCarbonDataDirectoryPath() + File.separator + this.carbonDataFileName;
- return factFilePath;
- }
/**
* Finish writing current file. It will flush stream, copy and rename temp file to final file
* @param copyInCurrentThread set to false if want to do data copy in a new thread
*/
protected void commitCurrentFile(boolean copyInCurrentThread) {
notifyDataMapBlockEnd();
CarbonUtil.closeStreams(this.fileOutputStream, this.fileChannel);
- if (copyInCurrentThread) {
- CarbonUtil.copyCarbonDataFileToCarbonStorePath(
- carbonDataFileTempPath, model.getCarbonDataDirectoryPath(),
- fileSizeInBytes);
+ if (enableDirectlyWriteData2Hdfs) {
+ if (copyInCurrentThread) {
+ CarbonUtil.completeRemainingHdfsReplicas(carbonDataFileHdfsPath,
+ FileFactory.FileType.HDFS);
+ } else {
+ executorServiceSubmitList.add(executorService.submit(
+ new CopyThread(carbonDataFileHdfsPath, FileFactory.FileType.HDFS)));
--- End diff --
Copy again? or just rename?
---