You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2018/01/18 03:28:43 UTC
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/1825
[CARBONDATA-2032][DataLoad] directly write carbon data files to HDFS
Currently in data loading, carbondata write the final data files to local disk and then copy it to HDFS.
For saving disk IO, carbondata can skip this procedure and directly write these files to HDFS.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [x] Any interfaces changed?
`Only internal interfaces has been changed`
- [x] Any backward compatibility impacted?
`No`
- [x] Document update required?
`No`
- [x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
`No`
- How it is tested? Please attach test report.
`Tested in local node and a 3-nodes cluster`
- Is it a performance related change? Please attach the performance test report.
`Yes. The disk IO has decreased`
- Any additional information to help reviewers in testing this change.
`No`
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
`Not related`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata 0118_opt_write_data_files_directly_to_hdfs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1825.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1825
----
commit fdced241cd5508d2fc7da457ed6e2e57dcaee4f1
Author: xuchuanyin <xu...@...>
Date: 2018-01-18T03:24:34Z
directly write carbon data files to HDFS
directly write carbon data files to hdfs to reduce disk I/O
----
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1763/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3475/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3245/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3688/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2624/
---
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1825#discussion_r171470463
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java ---
@@ -386,29 +380,42 @@ protected void writeIndexFile() throws IOException, CarbonDataWriterException {
.getIndexHeader(localCardinality, thriftColumnSchemaList, model.getBucketId());
// get the block index info thrift
List<BlockIndex> blockIndexThrift = CarbonMetadataUtil.getBlockIndexInfo(blockIndexInfoList);
- // randomly choose a temp location for index file
- String[] tempLocations = model.getStoreLocation();
- String chosenTempLocation = tempLocations[new Random().nextInt(tempLocations.length)];
- LOGGER.info("Randomly choose index file location: " + chosenTempLocation);
+ String indexFileName;
+ if (enableDirectlyWriteData2Hdfs) {
+ String rawFileName = model.getCarbonDataDirectoryPath() + File.separator + CarbonTablePath
+ .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
+ model.getBucketId(), model.getTaskExtension(),
+ "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
+ indexFileName = FileFactory.getUpdatedFilePath(rawFileName, FileFactory.FileType.HDFS);
+ } else {
+ // randomly choose a temp location for index file
+ String[] tempLocations = model.getStoreLocation();
+ String chosenTempLocation = tempLocations[new Random().nextInt(tempLocations.length)];
+ LOGGER.info("Randomly choose index file location: " + chosenTempLocation);
+ indexFileName = chosenTempLocation + File.separator + CarbonTablePath
+ .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
+ model.getBucketId(), model.getTaskExtension(),
+ "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
+ }
- String fileName = chosenTempLocation + File.separator + CarbonTablePath
- .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
- model.getBucketId(), model.getTaskExtension(),
- "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
CarbonIndexFileWriter writer = new CarbonIndexFileWriter();
// open file
- writer.openThriftWriter(fileName);
+ writer.openThriftWriter(indexFileName);
// write the header first
writer.writeThrift(indexHeader);
// write the indexes
for (BlockIndex blockIndex : blockIndexThrift) {
writer.writeThrift(blockIndex);
}
writer.close();
- // copy from temp to actual store location
- CarbonUtil.copyCarbonDataFileToCarbonStorePath(fileName,
- model.getCarbonDataDirectoryPath(),
- fileSizeInBytes);
+ if (enableDirectlyWriteData2Hdfs) {
+ executorServiceSubmitList.add(executorService.submit(
+ new CopyThread(indexFileName, FileFactory.FileType.HDFS)));
--- End diff --
use ‘completeHdfsBackendThread’ ~
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2844/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
@ravipesala thanks, I got your point...
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2993/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3090/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3685/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2132/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3876/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2341/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3914/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3912/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3998/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3152/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3457/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2990/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3216/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3995/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4089/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3368/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3666/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3124/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
this PR depends on #1952
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2986/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3689/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
@jackylk I haven't tested it in my cluster and the problems @ravipesala mentioned needs to be solved. I am working on it...
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1918/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2950/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3340/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2235/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2823/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2848/
---
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1825#discussion_r171454147
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java ---
@@ -239,24 +260,30 @@ private void notifyDataMapBlockEnd() {
blockletId = 0;
}
- private String constructFactFileFullPath() {
- String factFilePath =
- this.model.getCarbonDataDirectoryPath() + File.separator + this.carbonDataFileName;
- return factFilePath;
- }
/**
* Finish writing current file. It will flush stream, copy and rename temp file to final file
* @param copyInCurrentThread set to false if want to do data copy in a new thread
*/
protected void commitCurrentFile(boolean copyInCurrentThread) {
notifyDataMapBlockEnd();
CarbonUtil.closeStreams(this.fileOutputStream, this.fileChannel);
- if (copyInCurrentThread) {
- CarbonUtil.copyCarbonDataFileToCarbonStorePath(
- carbonDataFileTempPath, model.getCarbonDataDirectoryPath(),
- fileSizeInBytes);
+ if (enableDirectlyWriteData2Hdfs) {
+ if (copyInCurrentThread) {
+ CarbonUtil.completeRemainingHdfsReplicas(carbonDataFileHdfsPath,
+ FileFactory.FileType.HDFS);
+ } else {
+ executorServiceSubmitList.add(executorService.submit(
+ new CopyThread(carbonDataFileHdfsPath, FileFactory.FileType.HDFS)));
--- End diff --
Copy again? or just rename?
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2843/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3611/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2377/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1766/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1717/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2669/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2751/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3582/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4093/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2445/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1825
@xuchuanyin Do you observe any loading performance improvement?
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2996/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2099/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1760/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3604/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3335/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1825
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2366/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3432/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
@jackylk @ravipesala Can you review this PR?
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3763/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2926/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1693/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2667/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2668/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2983/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2626/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1825#discussion_r171454525
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java ---
@@ -386,29 +380,42 @@ protected void writeIndexFile() throws IOException, CarbonDataWriterException {
.getIndexHeader(localCardinality, thriftColumnSchemaList, model.getBucketId());
// get the block index info thrift
List<BlockIndex> blockIndexThrift = CarbonMetadataUtil.getBlockIndexInfo(blockIndexInfoList);
- // randomly choose a temp location for index file
- String[] tempLocations = model.getStoreLocation();
- String chosenTempLocation = tempLocations[new Random().nextInt(tempLocations.length)];
- LOGGER.info("Randomly choose index file location: " + chosenTempLocation);
+ String indexFileName;
+ if (enableDirectlyWriteData2Hdfs) {
+ String rawFileName = model.getCarbonDataDirectoryPath() + File.separator + CarbonTablePath
+ .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
+ model.getBucketId(), model.getTaskExtension(),
+ "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
+ indexFileName = FileFactory.getUpdatedFilePath(rawFileName, FileFactory.FileType.HDFS);
+ } else {
+ // randomly choose a temp location for index file
+ String[] tempLocations = model.getStoreLocation();
+ String chosenTempLocation = tempLocations[new Random().nextInt(tempLocations.length)];
+ LOGGER.info("Randomly choose index file location: " + chosenTempLocation);
+ indexFileName = chosenTempLocation + File.separator + CarbonTablePath
+ .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
+ model.getBucketId(), model.getTaskExtension(),
+ "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
+ }
- String fileName = chosenTempLocation + File.separator + CarbonTablePath
- .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
- model.getBucketId(), model.getTaskExtension(),
- "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
CarbonIndexFileWriter writer = new CarbonIndexFileWriter();
// open file
- writer.openThriftWriter(fileName);
+ writer.openThriftWriter(indexFileName);
// write the header first
writer.writeThrift(indexHeader);
// write the indexes
for (BlockIndex blockIndex : blockIndexThrift) {
writer.writeThrift(blockIndex);
}
writer.close();
- // copy from temp to actual store location
- CarbonUtil.copyCarbonDataFileToCarbonStorePath(fileName,
- model.getCarbonDataDirectoryPath(),
- fileSizeInBytes);
+ if (enableDirectlyWriteData2Hdfs) {
+ executorServiceSubmitList.add(executorService.submit(
+ new CopyThread(indexFileName, FileFactory.FileType.HDFS)));
--- End diff --
The name of CopyThread is confusing, is it copy or rename?
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3616/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3652/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3913/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2754/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1982/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4143/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1756/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2899/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1859/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3869/
---
[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1825#discussion_r171456256
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java ---
@@ -386,29 +380,42 @@ protected void writeIndexFile() throws IOException, CarbonDataWriterException {
.getIndexHeader(localCardinality, thriftColumnSchemaList, model.getBucketId());
// get the block index info thrift
List<BlockIndex> blockIndexThrift = CarbonMetadataUtil.getBlockIndexInfo(blockIndexInfoList);
- // randomly choose a temp location for index file
- String[] tempLocations = model.getStoreLocation();
- String chosenTempLocation = tempLocations[new Random().nextInt(tempLocations.length)];
- LOGGER.info("Randomly choose index file location: " + chosenTempLocation);
+ String indexFileName;
+ if (enableDirectlyWriteData2Hdfs) {
+ String rawFileName = model.getCarbonDataDirectoryPath() + File.separator + CarbonTablePath
+ .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
+ model.getBucketId(), model.getTaskExtension(),
+ "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
+ indexFileName = FileFactory.getUpdatedFilePath(rawFileName, FileFactory.FileType.HDFS);
+ } else {
+ // randomly choose a temp location for index file
+ String[] tempLocations = model.getStoreLocation();
+ String chosenTempLocation = tempLocations[new Random().nextInt(tempLocations.length)];
+ LOGGER.info("Randomly choose index file location: " + chosenTempLocation);
+ indexFileName = chosenTempLocation + File.separator + CarbonTablePath
+ .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
+ model.getBucketId(), model.getTaskExtension(),
+ "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
+ }
- String fileName = chosenTempLocation + File.separator + CarbonTablePath
- .getCarbonIndexFileName(model.getCarbonDataFileAttributes().getTaskId(),
- model.getBucketId(), model.getTaskExtension(),
- "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
CarbonIndexFileWriter writer = new CarbonIndexFileWriter();
// open file
- writer.openThriftWriter(fileName);
+ writer.openThriftWriter(indexFileName);
// write the header first
writer.writeThrift(indexHeader);
// write the indexes
for (BlockIndex blockIndex : blockIndexThrift) {
writer.writeThrift(blockIndex);
}
writer.close();
- // copy from temp to actual store location
- CarbonUtil.copyCarbonDataFileToCarbonStorePath(fileName,
- model.getCarbonDataDirectoryPath(),
- fileSizeInBytes);
+ if (enableDirectlyWriteData2Hdfs) {
+ executorServiceSubmitList.add(executorService.submit(
+ new CopyThread(indexFileName, FileFactory.FileType.HDFS)));
--- End diff --
Sometimes it's copy，sometimes it's setReplication.
How about using the name ‘CompleteHdfsBackupsThread’?
Or we can still use the name ‘CopyThread’ and add some comments to explain since we may remove ‘write temp fact to local’ in the further as we discussed before. At that that we can give it a proper name.
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3579/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2344/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4088/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4069/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3804/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2630/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3650/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3887/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2642/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3872/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
retest this please
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3429/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
@xuchuanyin There is a reason why we do copy instead of directly writing to HDFS.
1. We make sure that one complete carbondata file goes to one HDFS block only, while copying it to HDFS from local disk we specify the block size. Other wise it impacts query performance a lot.
2. Remove the overhead of writing to HDFS directly (it internally writes to replication as well) , so start copying in a different thread to avoid blocking of main loading flow.
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2372/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1825
@ravipesala I reconsidered the questions you mentioned and fixed it as below:
1. I use the user specified `table_blocksize` as the block size of data files. Actually in the current implementation, the block size is big enough to hold the entire file.
2. I directly write the data files to HDFS by specifying only 1 replication in the main thread and complete the remaining replications in another thread -- just the same way as before.
After I implement this, I tested it in a 3-node cluster and the data loading performance was just the same as before while the end-2-end `total size of disk write decreased by about 11%`.
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2996/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1825
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1753/
---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1825
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2976/
---