You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by sgururajshetty <gi...@git.apache.org> on 2018/08/02 14:22:00 UTC
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
GitHub user sgururajshetty opened a pull request:
https://github.com/apache/carbondata/pull/2603
[Documentation] Editorial review comment fixed
Minor issues fixed (spelling, syntax, and missing info)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sgururajshetty/carbondata editorial_review1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2603.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2603
----
commit 529f80dda6db3ce34e0baf766b03a9a13190b286
Author: sgururajshetty <sg...@...>
Date: 2018-07-25T12:44:07Z
Documentation for support for COLUMN_META_CACHE in create table and alter table properties
commit d816aaa7a89155b3579906f960ed6a0ba4d4a59f
Author: sgururajshetty <sg...@...>
Date: 2018-07-25T12:48:43Z
Documentation to support for CACHE_LEVEL in create table and alter table properties
commit 8ac243f8e9cff8359b6064352deb823eda7b9835
Author: sgururajshetty <sg...@...>
Date: 2018-07-25T13:24:52Z
Review comment fixed
commit 98501d35cfd110bcb9e75eb02628f3bce0c0f4ab
Author: sgururajshetty <sg...@...>
Date: 2018-07-25T13:26:58Z
review comment fixed
commit 62caf822cbcde1e519501c1d5db3c5cfc05fbd63
Author: Indhumathi27 <in...@...>
Date: 2018-07-21T10:46:21Z
[CARBONDATA-2606]Fix Complex array Pushdown and block auto merge compaction
1.Check for if Complex Column contains ArrayType at n levels and add parent to projection if contains array.
2.Block Auto merge compaction for table containing complex datatype columns.
3.Fix Decimal Datatype scale and precision with two level struct type
4.Fix Dictionary Include for ComplexDataType
- If other complex columns other than first complex column is given in dictionary include, then its insertion fails.
5.Fix BadRecord and dateformat for Complex primitive type-DATE
This closes #2535
commit d287a102b5c96e54261ac00c77038a1a56161fe9
Author: kumarvishal09 <ku...@...>
Date: 2018-07-24T14:40:54Z
[CARBONDATA-2779]Fixed filter query issue in case of V1/v2 format store
Problem:
Filter query is failing for V1/V2 carbondata store
Root Cause:
in V1 store measure min max was not added in blockminmaxindex in executor when filter is applied min max pruning is failing with array index out of cound exception
Solution:
Need to add min max for measure column same as already handled in driver block pruning
This closes #2550
commit b08745f68624ff066e0b23a41ce12d4a99618ac5
Author: Manhua <ke...@...>
Date: 2018-07-25T08:51:49Z
[CARBONDATA-2783][BloomDataMap][Doc] Update document for bloom filter datamap
add example for enable/disable datamap
This closes #2554
commit 964d26866468df6be130e9d65d339439cb4cf3b0
Author: praveenmeenakshi56 <pr...@...>
Date: 2018-07-25T15:31:37Z
[CARBONDATA-2750] Added Documentation for Local Dictionary Support
Added Documentation for Local Dictionary Support
This closes #2520
commit 1fa9f64d70123d0bc988427a34c0750283f5daae
Author: BJangir <ba...@...>
Date: 2018-07-23T16:44:12Z
[CARBONDATA-2772] Size based dictionary fallback is failing even threshold is not reached.
Issue:- Size Based Fallback happened even threshold is not reached.
RootCause:- Current size calculation is wrong. it is calculated for each data. instead of generated dictionary data .
Solution :- Current size should be calculated only for generated dictionary data.
This closes #2542
commit eae5817e56a20aecb7694c8d387dbb05b96e1045
Author: kunal642 <ku...@...>
Date: 2018-07-24T10:42:54Z
[CARBONDATA-2778]Fixed bug when select after delete and cleanup is showing empty records
Problem: In case if delete operation when it is found that the data being deleted is leading to a state where one complete block data is getting
deleted. In that case the status if that block is marked for delete and during the next delete operation run the block is deleted along with its
carbonIndex file. The problem arises due to deletion of carbonIndex file because for multiple blocks there can be one carbonIndex file as one
carbonIndex file represents one task.
Solution: Do not delete the carbondata and carbonIndex file. After compaction it will automatically take care of deleting the stale data and stale segments.
This closes #2548
commit 6d6874a11482a8aa79f2280f6572e84b5e3cbc93
Author: dhatchayani <dh...@...>
Date: 2018-07-25T09:11:58Z
[CARBONDATA-2753][Compatibility] Row count of page is calculated wrong for old store(V2 store)
Row count of page is calculated wrong for V2 store.
commit b6f5af6af96140876ec10ff09c3313d9b35ceb36
Author: Sssan520 <li...@...>
Date: 2018-07-25T11:36:00Z
[CARBONDATA-2782]delete dead code in class 'CarbonCleanFilesCommand'
The variables(dms、indexDms) in function processMetadata are nerver used.
This closes #2557
commit cf6bb46d89a9630209ef7f7e797ca8e156369206
Author: ajantha-bhat <aj...@...>
Date: 2018-07-24T07:03:47Z
[CARBONDATA-2775] Adaptive encoding fails for Unsafe OnHeap. if, target datatype is SHORT_INT
problem:
[CARBONDATA-2775] Adaptive encoding fails for Unsafe OnHeap if, target data type is SHORT_INT
solution: If ENABLE_OFFHEAP_SORT = false, in carbon property. UnsafeFixLengthColumnPage.java will use different compress logic. Not the raw compression. In that case, for SHORT_INT data type , conversion need to handle.
This closes #2546
commit 1fd3889cfec902e30cdcb9af6ed7b7f0df0f20e4
Author: mohammadshahidkhan <mo...@...>
Date: 2018-07-25T06:44:38Z
[HOTFIX] Fixed random test failure
Fixed random test failure
This closes #2553
commit 0f1aabb87b81670554188b698fc9fb4c598d8942
Author: Jatin <ja...@...>
Date: 2018-07-25T19:12:50Z
[HOTFIX] CreateDataMapPost Event was skipped in case of preaggregate datamap
CreateDataMapPost Event was skipped in case of preaggregate datamap
This closes #2562
commit 9b05edbace9ccca5c7def6d5e4cc157ed0eb0d66
Author: ajantha-bhat <aj...@...>
Date: 2018-07-25T19:05:36Z
[CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed:Forever blocking wait with more than 21 batch of data
problem: [CARBONDATA-2784]
[SDK writer] Forever blocking wait with more than 21 batch of data, when consumer is dead due to data loading exception (bad record / out of memory)
root cause:
When the consumer is dead due to data loading exception, writer will be forcefully closed. but queue.clear() cleared only snapshot of entries (10 batches) and close is set to true after that. In between clear() and close = true, If more than 10 batches of data is again put into queue. For 11th batch, queue.put() goes for forever block as consumer is dead.
Solution:
set close = true, before clearing the queue. This will avoid adding more batches to queue from write().
problem [CARBONDATA-2786] NPE when SDK writer tries to write a file
solution and cause:
#2387 , in CarbonProperties.java
After systemLocation = getStorePath(); Null validation missing for systemLocation.
because this can be null in SDK case. As Store location is not applicable for SDK.
All a null validation.
This closes #2561
commit 1c620f83e3f5b1adb28b73f38c40021b2f06e1d0
Author: Indhumathi27 <in...@...>
Date: 2018-07-27T06:52:25Z
[CARBONDATA-2791]Fix Encoding for Double if exceeds LONG.Max_value
If Factor(decimalcount) * absMaxValue exceeds LONG.MAX_VALUE, then go for direct compression.
This closes #2569
commit 39773ccf01345111bc950d8411646b886111f897
Author: manishgupta88 <to...@...>
Date: 2018-07-27T08:41:11Z
[HOTFIX] Removed file existence check to improve dataMap loading performance
Problem
DataMap loading performance degraded after adding file existence check.
Analysis
When carbonIndex file is read and carbondata file path to its metadata Info map is prepared, file physical existence is getting checked every time which in case of HDFS file system is a namenode call. This degrades the dataMap loading performance. This is done to avoid failures for
Handle IUD scenario where after delete operation carbondata file is deleted but the entry still exists in index file.
Fix
Modified code to check for physical file existence only in case when any IUD operation has happened on the table
This closes #2560
commit a55722c1f6dad54d58e7cd69b9f48bfb3693c2a0
Author: Indhumathi27 <in...@...>
Date: 2018-07-27T18:09:50Z
[CARBONDATA-2794]Distinct count fails on ArrayOfStruct
This PR fixes Code Generator Error thrown when Select filter contains more than one count of distinct of ArrayofStruct with group by Clause
This closes #2573
commit 6c4a2056f25ddc61d0fa0b205cb65a95c4298549
Author: mohammadshahidkhan <mo...@...>
Date: 2018-07-17T11:29:35Z
[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written
during datalaod, iud or compaction when disk is full.
Problem:
When a failure happens due to disk full during load, IUD or Compaction,
then while updating the tablestatus file, the tablestaus.tmp file during
atomic file operation remains empty, and in the finally block the empty
tablestaus.tmp file is getting renamed to the actual file.
This leads to empty tablestatus file. Once such problem happens the
tablestatus file can not be retrieved and the already loaded data can be used.
Solution:
If the failures happens during write then the the schema rename in the finally
block must be avoided.
This closes #2517
commit 0a5431c1b645af8820486a55dd33530583ea9ae1
Author: Jimmy Casey <ca...@...>
Date: 2018-07-29T21:35:27Z
Fixed Spelling
Fixed Spelling
This closes #2584
commit 6fb852ad86cc7ee7dd97af211a6382ec8f4eb750
Author: chenliang613 <ch...@...>
Date: 2018-07-27T02:51:45Z
[CARBONDATA-2789] Support Hadoop 2.8.3 eco-system integration
Add hadoop 2.8.3 profile and passed the compile
This closes #2566
commit 937f68cf91bf6708de5c51fe787ea7672b3ee126
Author: ravipesala <ra...@...>
Date: 2018-07-30T12:45:32Z
[CARBONDATA-2801]Added documentation for flat folder
[CARBONDATA-2801]Added documentation for flat folder
This closes #2582
commit e1abff71c7feba90be0ab338ec3b5aabf9cf9380
Author: ajantha-bhat <aj...@...>
Date: 2018-07-25T13:51:02Z
[CARBONDATA-2606][Complex DataType Enhancements]Fix Null result if projection column have null primitive column and struct
Problem:
In case if the actual value of the primitive data type is null, by PR#2489, we are moving all the null values to the end of the collected row without considering the data type.
Solution:
Place null in the end of output iff the null value is of complex primitive column.
This closes #2559
commit 917b230b57cff9c350524c81d939b06af6a35cde
Author: akashrn5 <ak...@...>
Date: 2018-07-27T07:01:48Z
[CARBONDATA-2585]disable local dictionary by default
make local dictionary false by default
This closes #2570
commit 5d2d6cf053accdd76bf26943db0a0f09dfc492e1
Author: xuchuanyin <xu...@...>
Date: 2018-07-26T15:22:58Z
[HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in incorrect query result with bloom datamap
This PR solve two problems which will affect the correctness of the query on bloom.
Revert PR2539
After review the code, we found that modification in PR2539 is not needed, so we revert that PR.
Bugs in overflow for blocklet count
Carbondata stores blocklet count for each block in byte data type, when a block contains more than 128 blocklets, it will overflow the byte
limits. Here we change the data type to short.
For cache_leve=block, after pruning by main BlockDataMap, the blockletNo in Blocklet is -1, which indicate that the following percedure will scan
the whole block -- all the blocklets in the block. So, when doing intersection with the pruned result from BloomDataMap, we
need to take care of these blocklets. In this implementation, we added the result from BloomDataMap based on the blocklet's existence in BlockDataMap.
This closes #2565
commit ec73f91817062495b085a231279f5133e44b1af8
Author: Indhumathi27 <in...@...>
Date: 2018-07-30T08:48:44Z
[CARBONDATA-2798] Fix Dictionary_Include for ComplexDataType
Problem1:
Select Filter is throwing BufferUnderFlow Exception as cardinality is filled for Non-Dictionary columns.
Solution:
Check if a complex column has Encoding => Dictionary and fill cardinality for that column only.
Problem2:
Transactional Table is throwing NullPointerException if csv fileheader is not proper.
Solution:
Throw CarbonDataLoadingException if csv fileheader is not proper.
This closes #2578
commit 6fc498055801ab6cf5181b24ee4db47c0510f172
Author: akashrn5 <ak...@...>
Date: 2018-07-30T13:52:29Z
[CARBONDATA-2805] Fix the ordering mismatch of segment numbers during cutom compaction
Problem:
when we have segments from 0 to 6 and i give 1, 2, 3 for custom compaction, then it should create 1.1 as compacted segment, but sometimes
it will create 3.1 as compacted segment which is wrong. This is beacuse custom Segment IDs were passing in hashset and finally inserted in
hashmap, while identifying segments to be merged. hashmap and hashset does not guarantee the insertion order which may lead to missmatch of segment numbers.
Solution:
Use LinkedHashSet and LinkedHashMap which always sure about the insertion order.
This closes #2585
commit 2f828ce2f438de67dabf565be25b9bbad76b49fc
Author: rahul <ra...@...>
Date: 2018-07-04T14:01:51Z
[CARBONDATA-2625] While BlockletDataMap loading, avoid multiple times listing of files
CarbonReader is very slow for many files as blockletDataMap lists files
of folder while loading each segment. This optimization lists once across segment loads.
This closes #2441
commit 436c8642185c7d262582c55c6c0a077922125be5
Author: xuchuanyin <xu...@...>
Date: 2018-07-27T03:54:21Z
[CARBONDATA-2790][BloomDataMap]Optimize default parameter for bloomfilter datamap
To provide better query performance for bloomfilter datamap by default,
we optimize bloom_size from 32000 to 640000 and optimize bloom_fpp from
0.01 to 0.00001.
This closes #2567
----
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7739/
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7760/
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6487/
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by chetandb <gi...@git.apache.org>.
Github user chetandb commented on the issue:
https://github.com/apache/carbondata/pull/2603
LGTM
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2603
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6129/
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2603
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6152/
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2603
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2603
LGTM
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2603
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6146/
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by sgururajshetty <gi...@git.apache.org>.
Github user sgururajshetty commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2603#discussion_r207516087
--- Diff: docs/configuration-parameters.md ---
@@ -140,7 +140,7 @@ This section provides the details of all the configurations required for CarbonD
| carbon.enableMinMax | true | Min max is feature added to enhance query performance. To disable this feature, set it false. |
| carbon.dynamicallocation.schedulertimeout | 5 | Specifies the maximum time (unit in seconds) the scheduler can wait for executor to be active. Minimum value is 5 sec and maximum value is 15 sec. |
| carbon.scheduler.minregisteredresourcesratio | 0.8 | Specifies the minimum resource (executor) ratio needed for starting the block distribution. The default value is 0.8, which indicates 80% of the requested resource is allocated for starting block distribution. The minimum value is 0.1 min and the maximum value is 1.0. |
-| carbon.search.enabled | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
+| carbon.search.enabled (Alpha Feature) | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
* **Global Dictionary Configurations**
--- End diff --
This issue is handled in a different PR #2576
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by chetandb <gi...@git.apache.org>.
Github user chetandb commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2603#discussion_r207462001
--- Diff: docs/configuration-parameters.md ---
@@ -140,7 +140,7 @@ This section provides the details of all the configurations required for CarbonD
| carbon.enableMinMax | true | Min max is feature added to enhance query performance. To disable this feature, set it false. |
| carbon.dynamicallocation.schedulertimeout | 5 | Specifies the maximum time (unit in seconds) the scheduler can wait for executor to be active. Minimum value is 5 sec and maximum value is 15 sec. |
| carbon.scheduler.minregisteredresourcesratio | 0.8 | Specifies the minimum resource (executor) ratio needed for starting the block distribution. The default value is 0.8, which indicates 80% of the requested resource is allocated for starting block distribution. The minimum value is 0.1 min and the maximum value is 1.0. |
-| carbon.search.enabled | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
+| carbon.search.enabled (Alpha Feature) | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
* **Global Dictionary Configurations**
--- End diff --
In Local Dictionary section the following updates needs to be done.
1) Remove the line: “44ad8fb40… Updated documentation on Local Dictionary Supoort |” in Page no: 7 at the Local Dictionary Configuration section in the Opensource PDF.
2) Change the description for “Local dictionary threshold” from: “The maximum cardinality for local dictionary generation (maximum - 100000)”
to
“The maximum cardinality for local dictionary generation (maximum value is 100000 and minimum value is 1000. If the “local_dictionary_threshold” value is set below 1000 or above 100000, then it would take the default value 10000)”
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7763/
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by chetandb <gi...@git.apache.org>.
Github user chetandb commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2603#discussion_r207457391
--- Diff: docs/datamap/bloomfilter-datamap-guide.md ---
@@ -44,7 +44,7 @@ A Bloom filter is a space-efficient probabilistic data structure that is used to
Carbondata introduce BloomFilter as an index datamap to enhance the performance of querying with precise value.
--- End diff --
Change introduce to introduced
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by sgururajshetty <gi...@git.apache.org>.
Github user sgururajshetty commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2603#discussion_r207516006
--- Diff: docs/configuration-parameters.md ---
@@ -140,7 +140,7 @@ This section provides the details of all the configurations required for CarbonD
| carbon.enableMinMax | true | Min max is feature added to enhance query performance. To disable this feature, set it false. |
| carbon.dynamicallocation.schedulertimeout | 5 | Specifies the maximum time (unit in seconds) the scheduler can wait for executor to be active. Minimum value is 5 sec and maximum value is 15 sec. |
| carbon.scheduler.minregisteredresourcesratio | 0.8 | Specifies the minimum resource (executor) ratio needed for starting the block distribution. The default value is 0.8, which indicates 80% of the requested resource is allocated for starting block distribution. The minimum value is 0.1 min and the maximum value is 1.0. |
-| carbon.search.enabled | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
+| carbon.search.enabled (Alpha Feature) | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
* **Global Dictionary Configurations**
--- End diff --
The minimum value need not be mentioned now
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by chetandb <gi...@git.apache.org>.
Github user chetandb commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2603#discussion_r207459790
--- Diff: docs/sdk-guide.md ---
@@ -351,7 +351,7 @@ public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOpti
* @throws IOException
* @throws InvalidLoadOptionException
*/
-public CarbonWriter buildWriterForAvroInput() throws IOException, InvalidLoadOptionException;
+public CarbonWriter buildWriterForAvroInput(org.apache.avro.Schema schema) throws IOException, InvalidLoadOptionException;
```
--- End diff --
TestSdkJson example code needs to be corrected. testJsonSdkWriter should be static and IOException should be handled
import java.io.IOException;
import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
import org.apache.carbondata.core.util.CarbonProperties;
import org.apache.carbondata.sdk.file.CarbonWriter;
import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
import org.apache.carbondata.sdk.file.Field;
import org.apache.carbondata.sdk.file.Schema;
public class TestSdkJson {
public static void main(String[] args) throws InvalidLoadOptionException {
testJsonSdkWriter();
}
public void testJsonSdkWriter() throws InvalidLoadOptionException {
String path = "./target/testJsonSdkWriter";
Field[] fields = new Field[2];
fields[0] = new Field("name", DataTypes.STRING);
fields[1] = new Field("age", DataTypes.INT);
Schema CarbonSchema = new Schema(fields);
CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path);
// initialize json writer with carbon schema
CarbonWriter writer = builder.buildWriterForJsonInput(CarbonSchema);
// one row of json Data as String
String JsonRow = "{\"name\":\"abcd\", \"age\":10}";
int rows = 5;
for (int i = 0; i < rows; i++) {
writer.write(JsonRow);
}
writer.close();
}
}
8.2
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7769/
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2603
retest this please
---
[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...
Posted by chetandb <gi...@git.apache.org>.
Github user chetandb commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2603#discussion_r207460915
--- Diff: docs/configuration-parameters.md ---
@@ -140,7 +140,7 @@ This section provides the details of all the configurations required for CarbonD
| carbon.enableMinMax | true | Min max is feature added to enhance query performance. To disable this feature, set it false. |
| carbon.dynamicallocation.schedulertimeout | 5 | Specifies the maximum time (unit in seconds) the scheduler can wait for executor to be active. Minimum value is 5 sec and maximum value is 15 sec. |
| carbon.scheduler.minregisteredresourcesratio | 0.8 | Specifies the minimum resource (executor) ratio needed for starting the block distribution. The default value is 0.8, which indicates 80% of the requested resource is allocated for starting block distribution. The minimum value is 0.1 min and the maximum value is 1.0. |
-| carbon.search.enabled | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
+| carbon.search.enabled (Alpha Feature) | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. |
* **Global Dictionary Configurations**
--- End diff --
In S3 section.
1. there should not be any Space in parameter . Should be carbon.storelocation.
2. "Concurrent queries are not supported" should be changed to "Only concurrent put (data management operations like load,insert,update)are supported."
3. The "Another way of setting the authentication parameters is as follows" should be removed.
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6465/
---
[GitHub] carbondata issue #2603: [Documentation] Editorial review comment fixed
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2603
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6493/
---