You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/10/08 15:56:20 UTC
[jira] [Commented] (CARBONDATA-287) Save the sorted temp files to
multi local dirs to improve dataloading perfomance
[ https://issues.apache.org/jira/browse/CARBONDATA-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558202#comment-15558202 ]
ASF GitHub Bot commented on CARBONDATA-287:
-------------------------------------------
GitHub user QiangCai opened a pull request:
https://github.com/apache/incubator-carbondata/pull/217
[CARBONDATA-287]Using multi local directorys to improve dataloading perfomance
**1 Analysis**
Now for each dataloading, we use only one different local directory to save the sorted temp files. I think it is neccessary to use multi local directorys in multi disks for each dataloading to improve dataloading performance.
**2 Solution**
Modify SortDataRows class to separate the sorted temp files into different local directorys in different disks..
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/incubator-carbondata localDirsWorkLoadBalance
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/217.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #217
----
commit 4d0a276bb7283dc0683efe9f3684fdff4cadf729
Author: c00318382 <c0...@huaweiobz.com>
Date: 2016-10-08T12:54:32Z
localdirs
commit 41c027833c19c3f3331470f48759f469159eb59c
Author: QiangCai <qi...@qq.com>
Date: 2016-10-08T15:26:39Z
clean temp folder
----
> Save the sorted temp files to multi local dirs to improve dataloading perfomance
> ---------------------------------------------------------------------------------
>
> Key: CARBONDATA-287
> URL: https://issues.apache.org/jira/browse/CARBONDATA-287
> Project: CarbonData
> Issue Type: Improvement
> Components: data-load
> Affects Versions: 0.2.0-incubating
> Reporter: QiangCai
> Assignee: QiangCai
> Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Now for each dataloading, we use only a different local dir to save the sorted temp files. I think it is neccessary to use multi local dirs for each dataloading to improve dataloading performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)