You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2018/04/22 05:27:12 UTC
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/2207
[WIP] Support flat folder for managed carbon table
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata no-segment-folder
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2207.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2207
----
commit d62c8d00051d785ddbabe4370e8a865d482174b8
Author: ravipesala <ra...@...>
Date: 2018-04-20T02:04:00Z
Support flat folder
----
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4431/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5266/
---
[GitHub] carbondata pull request #2207: [CARBONDATA-2428] Support flat folder for man...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r195635301
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
@@ -1007,6 +1007,19 @@ public boolean canAllow(CarbonTable carbonTable, TableOperation operation) {
return indexColumn;
}
+ /**
+ * Whether this table supports flat folder structure, it means all data files directly written
+ * under table path
+ */
+ public boolean isSupportFlatFolder() {
+ boolean supportFlatFolder = Boolean.parseBoolean(CarbonCommonConstants.DEFAULT_FLAT_FOLDER);
+ Map<String, String> tblProps = getTableInfo().getFactTable().getTableProperties();
+ if (tblProps.containsKey(CarbonCommonConstants.FLAT_FOLDER)) {
--- End diff --
Can the user set FLAT_FOLDER property in CarbonProperties(applicable for all tables) instead of specifying for each table??
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r184838574
--- Diff: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java ---
@@ -80,18 +80,18 @@ public static String getSegmentWithBlockFromTID(String Tid) {
* Returns block path from tuple id
*
* @param tid
- * @param factPath
+ * @param tablePath
--- End diff --
Please add isSegmentFile and related describe
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5516/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5106/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4629/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5039/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4302/
---
[GitHub] carbondata pull request #2207: [CARBONDATA-2428] Support flat folder for man...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r186599210
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -140,15 +141,19 @@ public static String genSegmentFileName(String segmentId, String UUID) {
/**
* Write segment file to the metadata folder of the table
+ *
* @param tablePath table path
* @param segmentId segment id
- * @param UUID a UUID string used to construct the segment file name
+ * @param UUID a UUID string used to construct the segment file name
* @return segment file name
*/
public static String writeSegmentFile(String tablePath, String segmentId, String UUID)
throws IOException {
String segmentPath = CarbonTablePath.getSegmentPath(tablePath, segmentId);
CarbonFile segmentFolder = FileFactory.getCarbonFile(segmentPath);
+ boolean supportFlatFolder = Boolean.parseBoolean(CarbonProperties.getInstance()
--- End diff --
I think this property should be in CreateTable-TBLProperties instead of in CarbonProperty. Otherwise, user may change the configuration in separate data loading.
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5164/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2207
LGTM
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4662/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5301/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4687/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4668/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4686/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6329/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5089/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5280/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5246/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6324/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4452/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5574/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6268/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5103/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6190/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6262/
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r184839135
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala ---
@@ -127,12 +126,16 @@ object DeleteExecution {
var result = List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))]()
while (records.hasNext) {
val ((key), (rowCountDetailsVO, groupedRows)) = records.next
+ val segmentId = key.substring(0, key.indexOf(CarbonCommonConstants.FILE_SEPARATOR))
--- End diff --
Is it possible that return -1 when run key.indexOf?
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4618/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5290/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5613/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4363/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2207
please check SDV
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5235/
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r185525720
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala ---
@@ -114,6 +109,10 @@ object DeleteExecution {
CarbonUpdateUtil
.createBlockDetailsMap(blockMappingVO, segmentUpdateStatusMngr)
+ val metadataDetails = SegmentStatusManager.readTableStatusFile(
+ CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+
+
--- End diff --
ok
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6251/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5099/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5287/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4116/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5281/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6307/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5221/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6258/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5228/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5100/
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r185525800
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -162,21 +167,44 @@ public static String writeSegmentFile(String tablePath, String segmentId, String
for (CarbonFile file : indexFiles) {
folderDetails.getFiles().add(file.getName());
}
- String segmentRelativePath = segmentPath.substring(tablePath.length(), segmentPath.length());
+ String segmentRelativePath = "/";
+ if (!supportFlatFolder) {
+ segmentRelativePath = segmentPath.substring(tablePath.length(), segmentPath.length());
+ }
segmentFile.addPath(segmentRelativePath, folderDetails);
- String segmentFileFolder = CarbonTablePath.getSegmentFilesLocation(tablePath);
+ String segmentFileFolder = CarbonTablePath.getSegmentFilesLocation(tablePath);
CarbonFile carbonFile = FileFactory.getCarbonFile(segmentFileFolder);
if (!carbonFile.exists()) {
carbonFile.mkdirs(segmentFileFolder, FileFactory.getFileType(segmentFileFolder));
}
String segmentFileName = genSegmentFileName(segmentId, UUID) + CarbonTablePath.SEGMENT_EXT;
// write segment info to new file.
writeSegmentFile(segmentFile, segmentFileFolder + File.separator + segmentFileName);
+
+ // Move all files to table path from segment folder.
+ if (supportFlatFolder) {
+ moveFromTempFolder(segmentPath, tablePath);
+ }
+
return segmentFileName;
}
return null;
}
+ /**
+ * Move the loaded data from temp folder to respective partition folder.
+ * @param tablePath
--- End diff --
ok
---
[GitHub] carbondata pull request #2207: [CARBONDATA-2428] Support flat folder for man...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r195644961
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
@@ -1007,6 +1007,19 @@ public boolean canAllow(CarbonTable carbonTable, TableOperation operation) {
return indexColumn;
}
+ /**
+ * Whether this table supports flat folder structure, it means all data files directly written
+ * under table path
+ */
+ public boolean isSupportFlatFolder() {
+ boolean supportFlatFolder = Boolean.parseBoolean(CarbonCommonConstants.DEFAULT_FLAT_FOLDER);
+ Map<String, String> tblProps = getTableInfo().getFactTable().getTableProperties();
+ if (tblProps.containsKey(CarbonCommonConstants.FLAT_FOLDER)) {
--- End diff --
Because of old tables also the property will be applicable if property set at System level, that causes hybrid storage like some data in segment folders and some data under table path. thats why it is not set at system level
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r185525675
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala ---
@@ -127,12 +126,16 @@ object DeleteExecution {
var result = List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))]()
while (records.hasNext) {
val ((key), (rowCountDetailsVO, groupedRows)) = records.next
+ val segmentId = key.substring(0, key.indexOf(CarbonCommonConstants.FILE_SEPARATOR))
--- End diff --
Not possible as in above method it adds segmentid.
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6337/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5238/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5232/
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r184838170
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -140,15 +141,19 @@ public static String genSegmentFileName(String segmentId, String UUID) {
/**
* Write segment file to the metadata folder of the table
+ *
* @param tablePath table path
* @param segmentId segment id
- * @param UUID a UUID string used to construct the segment file name
--- End diff --
please keep it
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4404/
---
[GitHub] carbondata pull request #2207: [CARBONDATA-2428] Support flat folder for man...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2207
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2207
Please add describe for this PR
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4542/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5403/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2207
Currently `segment` is a key terminology in carbon. The compaction/update/streaming.. all rely on the management of segment. will happen to the `flat folder` carbon table for the above scenario?
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5297/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5161/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4236/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5167/
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r185525770
--- Diff: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java ---
@@ -80,18 +80,18 @@ public static String getSegmentWithBlockFromTID(String Tid) {
* Returns block path from tuple id
*
* @param tid
- * @param factPath
+ * @param tablePath
--- End diff --
ok
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r185525923
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/CompactionSupportGlobalSortBigFileTest.scala ---
@@ -72,8 +72,8 @@ class CompactionSupportGlobalSortBigFileTest extends QueryTest with BeforeAndAft
}
override def afterEach {
- sql("DROP TABLE IF EXISTS compaction_globalsort")
- sql("DROP TABLE IF EXISTS carbon_localsort")
+// sql("DROP TABLE IF EXISTS compaction_globalsort")
--- End diff --
ok
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r184838441
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -162,21 +167,44 @@ public static String writeSegmentFile(String tablePath, String segmentId, String
for (CarbonFile file : indexFiles) {
folderDetails.getFiles().add(file.getName());
}
- String segmentRelativePath = segmentPath.substring(tablePath.length(), segmentPath.length());
+ String segmentRelativePath = "/";
+ if (!supportFlatFolder) {
+ segmentRelativePath = segmentPath.substring(tablePath.length(), segmentPath.length());
+ }
segmentFile.addPath(segmentRelativePath, folderDetails);
- String segmentFileFolder = CarbonTablePath.getSegmentFilesLocation(tablePath);
+ String segmentFileFolder = CarbonTablePath.getSegmentFilesLocation(tablePath);
CarbonFile carbonFile = FileFactory.getCarbonFile(segmentFileFolder);
if (!carbonFile.exists()) {
carbonFile.mkdirs(segmentFileFolder, FileFactory.getFileType(segmentFileFolder));
}
String segmentFileName = genSegmentFileName(segmentId, UUID) + CarbonTablePath.SEGMENT_EXT;
// write segment info to new file.
writeSegmentFile(segmentFile, segmentFileFolder + File.separator + segmentFileName);
+
+ // Move all files to table path from segment folder.
+ if (supportFlatFolder) {
+ moveFromTempFolder(segmentPath, tablePath);
+ }
+
return segmentFileName;
}
return null;
}
+ /**
+ * Move the loaded data from temp folder to respective partition folder.
+ * @param tablePath
--- End diff --
Please add segmentPath and related describe
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4350/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5527/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5178/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5566/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4351/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5175/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6349/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4617/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5121/
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r184839053
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala ---
@@ -114,6 +109,10 @@ object DeleteExecution {
CarbonUpdateUtil
.createBlockDetailsMap(blockMappingVO, segmentUpdateStatusMngr)
+ val metadataDetails = SegmentStatusManager.readTableStatusFile(
+ CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+
+
--- End diff --
please remove this blank line
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6261/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2207
LGTM
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6340/
---
[GitHub] carbondata pull request #2207: [CARBONDATA-2428] Support flat folder for man...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r194722392
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -140,15 +141,19 @@ public static String genSegmentFileName(String segmentId, String UUID) {
/**
* Write segment file to the metadata folder of the table
+ *
* @param tablePath table path
* @param segmentId segment id
- * @param UUID a UUID string used to construct the segment file name
+ * @param UUID a UUID string used to construct the segment file name
* @return segment file name
*/
public static String writeSegmentFile(String tablePath, String segmentId, String UUID)
throws IOException {
String segmentPath = CarbonTablePath.getSegmentPath(tablePath, segmentId);
CarbonFile segmentFolder = FileFactory.getCarbonFile(segmentPath);
+ boolean supportFlatFolder = Boolean.parseBoolean(CarbonProperties.getInstance()
--- End diff --
Yes, Added table property `flat_folder` , and default value is false.
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
@kunal642 The empty Fact folder is created because first the segment is written inside this folder like the normal flow and in the end while creating segment file it moves the data to the flat folder. We cannot simply delete the folder as the write path for segment needs to be changed and it has lot of impact on current code.
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4352/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
@xuchuanyin yes, all features which work on segment folder can also work in flat folder
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2207
retest this please
---
[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2207#discussion_r183230258
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/CompactionSupportGlobalSortBigFileTest.scala ---
@@ -72,8 +72,8 @@ class CompactionSupportGlobalSortBigFileTest extends QueryTest with BeforeAndAft
}
override def afterEach {
- sql("DROP TABLE IF EXISTS compaction_globalsort")
- sql("DROP TABLE IF EXISTS carbon_localsort")
+// sql("DROP TABLE IF EXISTS compaction_globalsort")
--- End diff --
please remove if not required
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5231/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5278/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4706/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5515/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4413/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2207
@ravipesala I think we can avoid writing the Path directory if flat_folder is true.
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5188/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6200/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5592/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5469/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5145/
---
[GitHub] carbondata issue #2207: [WIP] Support flat folder for managed carbon table
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4616/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2207
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5174/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5096/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5029/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6265/
---
[GitHub] carbondata issue #2207: [CARBONDATA-2428] Support flat folder for managed ca...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2207
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6282/
---