You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2018/05/18 07:45:32 UTC
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2318
[CARBONDATA-2491] Fix the error when reader read twice with SDK carbonReader
[CARBONDATA-2491] Fix the error when reader read twice with SDK carbonReader
1.Fix the error out of bound when reader read twice with SDK carbonReader
2.Fix the java.lang.NegativeArraySizeException
3.Add timestamp and bad record test case
This PR dependency on https://github.com/apache/carbondata/pull/2246
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
No
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
No
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata CARBONDATA-2491-OutOfBoundAndBadRecord
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2318.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2318
----
commit 8ae64ed8b3991753d2745f5e0e4a83b2bf970dd2
Author: xubo245 <xu...@...>
Date: 2018-05-17T04:13:01Z
[CARBONDATA-2413] After running CarbonWriter, there is null directory about datamap
if the segment id is null, then don't register all datamap writer and don't create null directory
remove line
commit 19043b6579d98908b1f9052b8ae4de8c9881161e
Author: xubo245 <xu...@...>
Date: 2018-05-18T07:40:16Z
[CARBONDATA-2491] Fix the error when reader read twice with SDK carbonReader
----
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6056/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2318
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5007/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189421188
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/AvroCarbonWriterTest.java ---
@@ -29,16 +29,27 @@
import org.apache.avro.generic.GenericData;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.CharEncoding;
+import org.junit.After;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.Test;
import tech.allegro.schema.json2avro.converter.JsonAvroConverter;
import org.apache.avro.Schema;
-
public class AvroCarbonWriterTest {
private String path = "./AvroCarbonWriterSuiteWriteFiles";
+ @Before
+ public void cleanFile() {
+ assert (TestUtil.cleanMdtFile());
--- End diff --
Is there another PR to remove the creation of system folder when user uses SDK to write data?
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4909/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4897/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5985/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2318
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4921/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6069/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6093/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r190279377
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -504,7 +505,24 @@ public QueryModel createQueryModel(InputSplit inputSplit, TaskAttemptContext tas
String projectionString = getColumnProjection(configuration);
String[] projectColumns;
if (projectionString != null) {
- projectColumns = projectionString.split(",");
+ if (projectionString.equalsIgnoreCase("*")) {
--- End diff --
instead of passing `*`, I think better to add another function to project all columns. You can add `projectAllColumns()`
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4925/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r190103023
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -77,6 +85,24 @@ public void testWriteAndReadFiles() throws IOException, InterruptedException {
Assert.assertEquals(i, 100);
reader.close();
+
+ // Read again
+ CarbonReader reader2 = CarbonReader
+ .builder(path, "_temp")
+ .projection(new String[]{"name", "age"})
--- End diff --
ok, done
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5963/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5975/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189827847
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
@@ -267,6 +267,8 @@ public TableDataMap getDataMap(CarbonTable table, DataMapSchema dataMapSchema) {
}
}
}
+ } else {
+ dataMap.clear();
--- End diff --
I move it into carbonReader close method
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
@sounakr Ok, done
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189923456
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -77,6 +85,24 @@ public void testWriteAndReadFiles() throws IOException, InterruptedException {
Assert.assertEquals(i, 100);
reader.close();
+
+ // Read again
+ CarbonReader reader2 = CarbonReader
+ .builder(path, "_temp")
+ .projection(new String[]{"name", "age"})
--- End diff --
Add a test case of two sequential reads but without closing the 1st reader, 2nd reader starts.
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189423096
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
@@ -267,6 +267,8 @@ public TableDataMap getDataMap(CarbonTable table, DataMapSchema dataMapSchema) {
}
}
}
+ } else {
+ dataMap.clear();
--- End diff --
I think it loses the purpose of cache if you clear for every retrieval. clear should be called only flush the cache. why do you need to flush cache for every call?
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r190151696
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -177,4 +239,134 @@ public void testWriteAndReadFilesNonTransactional() throws IOException, Interrup
reader.close();
FileUtils.deleteDirectory(new File(path));
}
+
+ CarbonProperties carbonProperties;
+
+ @Override
+ public void setUp() {
+ carbonProperties = CarbonProperties.getInstance();
+ }
+
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(CarbonReaderTest.class.getName());
+
+ @Test
+ public void testTimeStampAndBadRecord() throws IOException, InterruptedException {
+ String timestampFormat = carbonProperties.getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT);
+ String badRecordAction = carbonProperties.getProperty(CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION,
+ CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION_DEFAULT);
+ String badRecordLoc = carbonProperties.getProperty(CarbonCommonConstants.CARBON_BADRECORDS_LOC,
+ CarbonCommonConstants.CARBON_BADRECORDS_LOC_DEFAULT_VAL);
+ String rootPath = new File(this.getClass().getResource("/").getPath()
+ + "../../").getCanonicalPath();
+ String storeLocation = rootPath + "/target/";
+ carbonProperties
+ .addProperty(CarbonCommonConstants.CARBON_BADRECORDS_LOC, storeLocation)
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd hh:mm:ss")
+ .addProperty(CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION, "REDIRECT");
+ String path = "./testWriteFiles";
+ FileUtils.deleteDirectory(new File(path));
+
+ Field[] fields = new Field[9];
+ fields[0] = new Field("stringField", DataTypes.STRING);
+ fields[1] = new Field("intField", DataTypes.INT);
+ fields[2] = new Field("shortField", DataTypes.SHORT);
+ fields[3] = new Field("longField", DataTypes.LONG);
+ fields[4] = new Field("doubleField", DataTypes.DOUBLE);
+ fields[5] = new Field("boolField", DataTypes.BOOLEAN);
+ fields[6] = new Field("dateField", DataTypes.DATE);
+ fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
+ fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
+
+ try {
+ CarbonWriterBuilder builder = CarbonWriter.builder()
+ .isTransactionalTable(true)
+ .persistSchemaFile(true)
+ .outputPath(path);
+
+ CarbonWriter writer = builder.buildWriterForCSVInput(new Schema(fields));
+
+ for (int i = 0; i < 100; i++) {
+ String[] row = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i),
+ String.valueOf(i),
+ String.valueOf(Long.MAX_VALUE - i),
+ String.valueOf((double) i / 2),
+ String.valueOf(true),
+ "2018-05-12",
+ "2018-05-12",
+ "12.345"
+ };
+ writer.write(row);
+ String[] row2 = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i),
+ String.valueOf(i),
+ String.valueOf(Long.MAX_VALUE - i),
+ String.valueOf((double) i / 2),
+ String.valueOf(true),
+ "2019-03-02",
+ "2019-02-12 03:03:34",
+ "12.345"
+ };
+ writer.write(row2);
+ }
+ writer.close();
+ } catch (Exception e) {
+ e.printStackTrace();
+ Assert.fail(e.getMessage());
+ }
+ LOGGER.audit("Bad record location:" + storeLocation);
+ File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
+ Assert.assertTrue(segmentFolder.exists());
+
+ File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
+ @Override
+ public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
+ }
+ });
+ Assert.assertNotNull(dataFiles);
+ Assert.assertTrue(dataFiles.length > 0);
+
+ CarbonReader reader = CarbonReader.builder(path, "_temp")
+ .projection(new String[]{
+ "stringField"
+ , "shortField"
+ , "intField"
+ , "longField"
+ , "doubleField"
+ , "boolField"
+ , "dateField"
+ , "timeField"
+ , "decimalField"}).build();
+
+ int i = 0;
+ while (reader.hasNext()) {
--- End diff --
ok, done
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2318
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5060/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4887/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4872/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r190102945
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -77,6 +85,24 @@ public void testWriteAndReadFiles() throws IOException, InterruptedException {
Assert.assertEquals(i, 100);
reader.close();
+
--- End diff --
Ok, I will add. What's more, search mode has used CarbonRecordReader, there are some test case to concurrent run in org.apache.carbondata.examples.SearchModeExample.
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189480763
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/AvroCarbonWriterTest.java ---
@@ -29,16 +29,27 @@
import org.apache.avro.generic.GenericData;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.CharEncoding;
+import org.junit.After;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.Test;
import tech.allegro.schema.json2avro.converter.JsonAvroConverter;
import org.apache.avro.Schema;
-
public class AvroCarbonWriterTest {
private String path = "./AvroCarbonWriterSuiteWriteFiles";
+ @Before
+ public void cleanFile() {
+ assert (TestUtil.cleanMdtFile());
--- End diff --
@jackylk This PR dependency on PR2246, the first commit it cherry-pick from PR2246.
After pr2246 merged, this pr need rebase.
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2318
LGTM
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5959/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4818/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6091/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6084/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by sraghunandan <gi...@git.apache.org>.
Github user sraghunandan commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/2318
LGTM
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6031/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
@jackylk CI pass, please check.
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6046/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2318
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5066/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4932/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6081/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5987/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189421068
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
@@ -267,6 +267,8 @@ public TableDataMap getDataMap(CarbonTable table, DataMapSchema dataMapSchema) {
}
}
}
+ } else {
+ dataMap.clear();
--- End diff --
why is it required? please add comment
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189480675
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/AvroCarbonWriterTest.java ---
@@ -29,16 +29,27 @@
import org.apache.avro.generic.GenericData;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.CharEncoding;
+import org.junit.After;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.Test;
import tech.allegro.schema.json2avro.converter.JsonAvroConverter;
import org.apache.avro.Schema;
-
public class AvroCarbonWriterTest {
private String path = "./AvroCarbonWriterSuiteWriteFiles";
+ @Before
+ public void cleanFile() {
+ assert (TestUtil.cleanMdtFile());
--- End diff --
PR 2246 fix another problem, can't solve this one. @ravipesala
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4930/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2318
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5071/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
@jackylk @ravipesala Hello, sounakr give LGTM and CI pass. Can you help to check and merge it if there are no problem, please.
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
@sounakr Doneļ¼and CI pass
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2318
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5082/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2318
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4986/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4902/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189422955
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/AvroCarbonWriterTest.java ---
@@ -29,16 +29,27 @@
import org.apache.avro.generic.GenericData;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.CharEncoding;
+import org.junit.After;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.Test;
import tech.allegro.schema.json2avro.converter.JsonAvroConverter;
import org.apache.avro.Schema;
-
public class AvroCarbonWriterTest {
private String path = "./AvroCarbonWriterSuiteWriteFiles";
+ @Before
+ public void cleanFile() {
+ assert (TestUtil.cleanMdtFile());
--- End diff --
I think pr 2246 can solve it
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4892/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r190444380
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -504,7 +505,24 @@ public QueryModel createQueryModel(InputSplit inputSplit, TaskAttemptContext tas
String projectionString = getColumnProjection(configuration);
String[] projectColumns;
if (projectionString != null) {
- projectColumns = projectionString.split(",");
+ if (projectionString.equalsIgnoreCase("*")) {
--- End diff --
ok, I will raise new PR for it.
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4802/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4827/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r189922516
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -77,6 +85,24 @@ public void testWriteAndReadFiles() throws IOException, InterruptedException {
Assert.assertEquals(i, 100);
reader.close();
+
--- End diff --
This test case points to sequential read. One reader gets closed and second one starts. What exactly happens when there is parallel read of two readers. Can we have a test case for that?
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6062/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4829/
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4805/
---
[GitHub] carbondata pull request #2318: [CARBONDATA-2491] Fix the error when reader r...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2318#discussion_r190140779
--- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -177,4 +239,134 @@ public void testWriteAndReadFilesNonTransactional() throws IOException, Interrup
reader.close();
FileUtils.deleteDirectory(new File(path));
}
+
+ CarbonProperties carbonProperties;
+
+ @Override
+ public void setUp() {
+ carbonProperties = CarbonProperties.getInstance();
+ }
+
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(CarbonReaderTest.class.getName());
+
+ @Test
+ public void testTimeStampAndBadRecord() throws IOException, InterruptedException {
+ String timestampFormat = carbonProperties.getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT);
+ String badRecordAction = carbonProperties.getProperty(CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION,
+ CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION_DEFAULT);
+ String badRecordLoc = carbonProperties.getProperty(CarbonCommonConstants.CARBON_BADRECORDS_LOC,
+ CarbonCommonConstants.CARBON_BADRECORDS_LOC_DEFAULT_VAL);
+ String rootPath = new File(this.getClass().getResource("/").getPath()
+ + "../../").getCanonicalPath();
+ String storeLocation = rootPath + "/target/";
+ carbonProperties
+ .addProperty(CarbonCommonConstants.CARBON_BADRECORDS_LOC, storeLocation)
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd hh:mm:ss")
+ .addProperty(CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION, "REDIRECT");
+ String path = "./testWriteFiles";
+ FileUtils.deleteDirectory(new File(path));
+
+ Field[] fields = new Field[9];
+ fields[0] = new Field("stringField", DataTypes.STRING);
+ fields[1] = new Field("intField", DataTypes.INT);
+ fields[2] = new Field("shortField", DataTypes.SHORT);
+ fields[3] = new Field("longField", DataTypes.LONG);
+ fields[4] = new Field("doubleField", DataTypes.DOUBLE);
+ fields[5] = new Field("boolField", DataTypes.BOOLEAN);
+ fields[6] = new Field("dateField", DataTypes.DATE);
+ fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
+ fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
+
+ try {
+ CarbonWriterBuilder builder = CarbonWriter.builder()
+ .isTransactionalTable(true)
+ .persistSchemaFile(true)
+ .outputPath(path);
+
+ CarbonWriter writer = builder.buildWriterForCSVInput(new Schema(fields));
+
+ for (int i = 0; i < 100; i++) {
+ String[] row = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i),
+ String.valueOf(i),
+ String.valueOf(Long.MAX_VALUE - i),
+ String.valueOf((double) i / 2),
+ String.valueOf(true),
+ "2018-05-12",
+ "2018-05-12",
+ "12.345"
+ };
+ writer.write(row);
+ String[] row2 = new String[]{
+ "robot" + (i % 10),
+ String.valueOf(i),
+ String.valueOf(i),
+ String.valueOf(Long.MAX_VALUE - i),
+ String.valueOf((double) i / 2),
+ String.valueOf(true),
+ "2019-03-02",
+ "2019-02-12 03:03:34",
+ "12.345"
+ };
+ writer.write(row2);
+ }
+ writer.close();
+ } catch (Exception e) {
+ e.printStackTrace();
+ Assert.fail(e.getMessage());
+ }
+ LOGGER.audit("Bad record location:" + storeLocation);
+ File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
+ Assert.assertTrue(segmentFolder.exists());
+
+ File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
+ @Override
+ public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
+ }
+ });
+ Assert.assertNotNull(dataFiles);
+ Assert.assertTrue(dataFiles.length > 0);
+
+ CarbonReader reader = CarbonReader.builder(path, "_temp")
+ .projection(new String[]{
+ "stringField"
+ , "shortField"
+ , "intField"
+ , "longField"
+ , "doubleField"
+ , "boolField"
+ , "dateField"
+ , "timeField"
+ , "decimalField"}).build();
+
+ int i = 0;
+ while (reader.hasNext()) {
--- End diff --
Please try out another test case. Open the reader and then close the reader and then try to do readNextRow.
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/2318
@xubo245 Better to allow "*" as input in reader projection. This will help the user to specify all columns. Just like SQL select *.
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2318
retest this please
---
[GitHub] carbondata issue #2318: [CARBONDATA-2491] Fix the error when reader read twi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2318
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6051/
---