You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by BJangir <gi...@git.apache.org> on 2018/04/24 15:10:23 UTC
[GitHub] carbondata pull request #2224: [CARBONDATA-2393]TaskNo is not working for SD...
GitHub user BJangir opened a pull request:
https://github.com/apache/carbondata/pull/2224
[CARBONDATA-2393]TaskNo is not working for SDK
Issue:- Task No is not getting reflected in the Carbon Data file and in index file .
Cause :- Task No is getting overwritten in CarbonTableOutputformate even CarbonModel have taskNo.
Solution :- if CarbomModel has taskNo then no need to overwrite .
- [ ] Any interfaces changed?
NO
- [ ] Any backward compatibility impacted?
NO
- [ ] Document update required?
NO
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
added TestCase
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BJangir/incubator-carbondata CARBONDATA-2393
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2224.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2224
----
commit 6b19070ddcf632583b041fa7bee4111aaa3dc32e
Author: BJangir <ba...@...>
Date: 2018-04-24T14:42:54Z
[CARBONDATA-2393] TaskNo is not working for SDK
----
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on the issue:
https://github.com/apache/carbondata/pull/2224
LGTM
---
[GitHub] carbondata pull request #2224: [CARBONDATA-2393]TaskNo is not working for SD...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2224#discussion_r183800623
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java ---
@@ -205,4 +205,70 @@ public void testSchemaPersistence() throws IOException {
FileUtils.deleteDirectory(new File(path));
}
+ @Test
+ public void testTaskNo() throws IOException {
+ // TODO: write all data type and read by CarbonRecordReader to verify the content
+ String path = "./testWriteFiles";
+ FileUtils.deleteDirectory(new File(path));
+
+ Field[] fields = new Field[2];
+ fields[0] = new Field("stringField", DataTypes.STRING);
+ fields[1] = new Field("intField", DataTypes.INT);
+
+
+ try {
+ CarbonWriterBuilder builder = CarbonWriter.builder()
+ .withSchema(new Schema(fields))
+ .isTransactionalTable(true).taskNo("5")
+ .outputPath(path);
+
+ CarbonWriter writer = builder.buildWriterForCSVInput();
+
+ for (int i = 0; i < 2; i++) {
+ String[] row = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i)
+ };
+ writer.write(row);
+ }
+ writer.close();
+ } catch (Exception e) {
+ e.printStackTrace();
+ Assert.fail(e.getMessage());
+ }
+
+ File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
+ Assert.assertTrue(segmentFolder.exists());
+
+ File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
+ @Override public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
+ }
+ });
+ Assert.assertNotNull(dataFiles);
+ Assert.assertTrue(dataFiles.length > 0);
+ String[] splitedcarbonFileName = getSplitedcarbonFileName(dataFiles[0]);
--- End diff --
call this in finally block FileUtils.deleteDirectory(new File(path));
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2224
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4516/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5375/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by BJangir <gi...@git.apache.org>.
Github user BJangir commented on the issue:
https://github.com/apache/carbondata/pull/2224
retest this please
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4373/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by BJangir <gi...@git.apache.org>.
Github user BJangir commented on the issue:
https://github.com/apache/carbondata/pull/2224
retest please
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5532/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2224
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4513/
---
[GitHub] carbondata pull request #2224: [CARBONDATA-2393]TaskNo is not working for SD...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2224#discussion_r183800279
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java ---
@@ -205,4 +205,70 @@ public void testSchemaPersistence() throws IOException {
FileUtils.deleteDirectory(new File(path));
}
+ @Test
+ public void testTaskNo() throws IOException {
+ // TODO: write all data type and read by CarbonRecordReader to verify the content
+ String path = "./testWriteFiles";
+ FileUtils.deleteDirectory(new File(path));
+
+ Field[] fields = new Field[2];
+ fields[0] = new Field("stringField", DataTypes.STRING);
+ fields[1] = new Field("intField", DataTypes.INT);
+
+
+ try {
+ CarbonWriterBuilder builder = CarbonWriter.builder()
+ .withSchema(new Schema(fields))
+ .isTransactionalTable(true).taskNo("5")
+ .outputPath(path);
+
+ CarbonWriter writer = builder.buildWriterForCSVInput();
+
+ for (int i = 0; i < 2; i++) {
+ String[] row = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i)
+ };
+ writer.write(row);
+ }
+ writer.close();
+ } catch (Exception e) {
+ e.printStackTrace();
+ Assert.fail(e.getMessage());
+ }
+
+ File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
+ Assert.assertTrue(segmentFolder.exists());
+
+ File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
+ @Override public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
+ }
+ });
+ Assert.assertNotNull(dataFiles);
+ Assert.assertTrue(dataFiles.length > 0);
+ String[] splitedcarbonFileName = getSplitedcarbonFileName(dataFiles[0]);
--- End diff --
Use CarbonTablePath.getTaskNo method
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4367/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5539/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2224
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4632/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5536/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5529/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2224
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4635/
---
[GitHub] carbondata pull request #2224: [CARBONDATA-2393]TaskNo is not working for SD...
Posted by BJangir <gi...@git.apache.org>.
Github user BJangir commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2224#discussion_r183814030
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java ---
@@ -205,4 +205,70 @@ public void testSchemaPersistence() throws IOException {
FileUtils.deleteDirectory(new File(path));
}
+ @Test
+ public void testTaskNo() throws IOException {
+ // TODO: write all data type and read by CarbonRecordReader to verify the content
+ String path = "./testWriteFiles";
+ FileUtils.deleteDirectory(new File(path));
+
+ Field[] fields = new Field[2];
+ fields[0] = new Field("stringField", DataTypes.STRING);
+ fields[1] = new Field("intField", DataTypes.INT);
+
+
+ try {
+ CarbonWriterBuilder builder = CarbonWriter.builder()
+ .withSchema(new Schema(fields))
+ .isTransactionalTable(true).taskNo("5")
+ .outputPath(path);
+
+ CarbonWriter writer = builder.buildWriterForCSVInput();
+
+ for (int i = 0; i < 2; i++) {
+ String[] row = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i)
+ };
+ writer.write(row);
+ }
+ writer.close();
+ } catch (Exception e) {
+ e.printStackTrace();
+ Assert.fail(e.getMessage());
+ }
+
+ File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
+ Assert.assertTrue(segmentFolder.exists());
+
+ File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
+ @Override public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
+ }
+ });
+ Assert.assertNotNull(dataFiles);
+ Assert.assertTrue(dataFiles.length > 0);
+ String[] splitedcarbonFileName = getSplitedcarbonFileName(dataFiles[0]);
--- End diff --
OK.Fixed
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by BJangir <gi...@git.apache.org>.
Github user BJangir commented on the issue:
https://github.com/apache/carbondata/pull/2224
retest this please
---
[GitHub] carbondata pull request #2224: [CARBONDATA-2393]TaskNo is not working for SD...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2224
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4205/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2224
retest this please
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4208/
---
[GitHub] carbondata issue #2224: [CARBONDATA-2393]TaskNo is not working for SDK
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2224
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4376/
---