You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@carbondata.apache.org by Liang Chen <ch...@apache.org> on 2018/06/02 03:37:18 UTC
[ANNOUNCE] Apache CarbonData 1.4.0 release
Hi
Apache CarbonData community is pleased to announce the release of the
Version 1.4.0 in The Apache Software Foundation (ASF).
CarbonData is a high-performance big data store solution that supports fast
filter lookups and ad-hoc OLAP analysis. Due to varied business driven
analysis, and the demand for flexibility of data analytics, big data domain
is shadowed with data duplication and increased data management cost.
CarbonData provides a new converged data storage to address data
de-duplication, and supports various application scenarios. CarbonData has
been deployed in 20+ enterprise production environments, largest single
cluster (100+ nodes) managing data of tens of trillions. The I/O scanning
and computing performance is improved by leveraging features such as
multi-level index, dictionary encoding, pre-aggregation, dynamic
partitioning, and quasi-real-time data query; there by achieving
second-level response to analytics query on tens of trillions of data.
We encourage everyone to download the release
https://dist.apache.org/repos/dist/release/carbondata/1.4.0/, and feedback
through mailing list.(dev@carbondata.apache.org or
user@carbondata.apache.org)
This release note provides information on the new features, improvements,
and bug fixes of this release.
What’s New in Version 1.4.0?
In this version of CarbonData, following are the new features added for
performance improvements, compatibility, and usability of CarbonData.
Supports SDK
Provided Carbon SDK to write and read CarbonData files through Java API,
supporting Avro schema and JSON data.
Supports External Table with Location
Now you can create external table by specifying the location of Carbon data
files.
Supports Streaming with Pre-Aggregate Table
Now you can create pre-aggregate table on streaming tables. This enhances
OLAP type of query performance on streaming tables.
Supports Partition with Pre-Aggregate
Now when you drop the partition column in the main table, the same column
can be dropped in the aggregate table keeping both in sync.
Enhanced Data Load performance
Now the data load performance has been enhanced
Supports Lucene Index for Text Search (Alpha feature)
This feature allows you to perform text search on Carbon data.
Supports S3 Read on CarbonData FilesSupports Search Mode (Alpha feature)
Supports search mode to improve concurrent queries performance.
Supports Bloom Filter Index (Alpha feature)
This feature fastens blocklet pruning.
Please find the detailed JIRA list:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12341005&styleName=Html&projectId=12320220&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED%7C72f8d21d9927bf947fc8c0dfb7f69263d4048efb%7Clout
Sub-task
- [CARBONDATA-1522
<https://issues.apache.org/jira/browse/CARBONDATA-1522>] - 6. Loading
aggregation tables for streaming data tables.
- [CARBONDATA-1575
<https://issues.apache.org/jira/browse/CARBONDATA-1575>] - Support large
scale data on DataMap
- [CARBONDATA-1601
<https://issues.apache.org/jira/browse/CARBONDATA-1601>] - Add carbon
store module
- [CARBONDATA-1998
<https://issues.apache.org/jira/browse/CARBONDATA-1998>] - Support
FileReader Java API for file level carbondata
- [CARBONDATA-2165
<https://issues.apache.org/jira/browse/CARBONDATA-2165>] - Remove spark
dependency in carbon-hadoop module
- [CARBONDATA-2189
<https://issues.apache.org/jira/browse/CARBONDATA-2189>] - Support add
and drop interface
- [CARBONDATA-2206
<https://issues.apache.org/jira/browse/CARBONDATA-2206>] - Integrate
lucene as datamap
- [CARBONDATA-2247
<https://issues.apache.org/jira/browse/CARBONDATA-2247>] - Support
writing index in CarbonWriter
- [CARBONDATA-2294
<https://issues.apache.org/jira/browse/CARBONDATA-2294>] - Support
preaggregate table creation on partition tables
- [CARBONDATA-2301
<https://issues.apache.org/jira/browse/CARBONDATA-2301>] - Support query
interface in CarbonStore
- [CARBONDATA-2359
<https://issues.apache.org/jira/browse/CARBONDATA-2359>] - Support
applicable load options and table properties for Non Transactional table
- [CARBONDATA-2360
<https://issues.apache.org/jira/browse/CARBONDATA-2360>] - Insert into
and Insert Into overwrite support for Non Transactional table
- [CARBONDATA-2361
<https://issues.apache.org/jira/browse/CARBONDATA-2361>] - Refactor Read
Committed Scope implementation.
- [CARBONDATA-2369
<https://issues.apache.org/jira/browse/CARBONDATA-2369>] - Add a
document for Non Transactional table with SDK writer guide
- [CARBONDATA-2388
<https://issues.apache.org/jira/browse/CARBONDATA-2388>] - Avro Nested
Datatype Support
- [CARBONDATA-2423
<https://issues.apache.org/jira/browse/CARBONDATA-2423>] - CarbonReader
Support To Read Non Transactional Table
- [CARBONDATA-2430
<https://issues.apache.org/jira/browse/CARBONDATA-2430>] - Reshuffling
of Columns given by user in SDK
- [CARBONDATA-2433
<https://issues.apache.org/jira/browse/CARBONDATA-2433>] - Executor OOM
because of GC when blocklet pruning is done using Lucene datamap
- [CARBONDATA-2443
<https://issues.apache.org/jira/browse/CARBONDATA-2443>] - Multi Level
Complex Type Support for AVRO SDK
- [CARBONDATA-2457
<https://issues.apache.org/jira/browse/CARBONDATA-2457>] - Add converter
to get Carbon SDK Schema from Avro schema directly.
- [CARBONDATA-2474
<https://issues.apache.org/jira/browse/CARBONDATA-2474>] - Support
Modular Plan
- [CARBONDATA-2475
<https://issues.apache.org/jira/browse/CARBONDATA-2475>] - Support
Materialized View query rewrite
- [CARBONDATA-2484
<https://issues.apache.org/jira/browse/CARBONDATA-2484>] - Refactor the
datamap code and clear the datamap from executor on table drop
Bug
- [CARBONDATA-1114
<https://issues.apache.org/jira/browse/CARBONDATA-1114>] - Failed to run
tests in windows env
- [CARBONDATA-1990
<https://issues.apache.org/jira/browse/CARBONDATA-1990>] - Null values
shown when the basic word count example is tried on carbon streaming table
- [CARBONDATA-2002
<https://issues.apache.org/jira/browse/CARBONDATA-2002>] - Streaming
segment status is not getting updated to finished or success
- [CARBONDATA-2056
<https://issues.apache.org/jira/browse/CARBONDATA-2056>] - Hadoop
Configuration with access key and secret key should be passed while
creating InputStream of distributed carbon file.
- [CARBONDATA-2080
<https://issues.apache.org/jira/browse/CARBONDATA-2080>] - Hadoop Conf
not propagated from driver to executor in S3
- [CARBONDATA-2085
<https://issues.apache.org/jira/browse/CARBONDATA-2085>] - It's
different between load twice and create datamap with load again after load
data and create datamap
- [CARBONDATA-2130
<https://issues.apache.org/jira/browse/CARBONDATA-2130>] - Find some
Spelling error in CarbonData
- [CARBONDATA-2147
<https://issues.apache.org/jira/browse/CARBONDATA-2147>] - Exception
displays while loading data with streaming
- [CARBONDATA-2152
<https://issues.apache.org/jira/browse/CARBONDATA-2152>] - Min function
working incorrectly for string type with dictionary include in presto.
- [CARBONDATA-2155
<https://issues.apache.org/jira/browse/CARBONDATA-2155>] - IS NULL not
working correctly on string datatype with dictionary_include in presto
integration
- [CARBONDATA-2161
<https://issues.apache.org/jira/browse/CARBONDATA-2161>] - Compacted
Segment of Streaming Table should update "mergeTo" column
- [CARBONDATA-2194
<https://issues.apache.org/jira/browse/CARBONDATA-2194>] - Exception
message is improper when use incorrect bad record action type
- [CARBONDATA-2198
<https://issues.apache.org/jira/browse/CARBONDATA-2198>] - Streaming
data to a table with bad_records_action as IGNORE throws ClassCastException
- [CARBONDATA-2199
<https://issues.apache.org/jira/browse/CARBONDATA-2199>] - Exception
occurs when change the datatype of measure having sort_column
- [CARBONDATA-2207
<https://issues.apache.org/jira/browse/CARBONDATA-2207>] - TestCase
Fails using Hive Metastore
- [CARBONDATA-2208
<https://issues.apache.org/jira/browse/CARBONDATA-2208>] - Pre aggregate
datamap creation is failing when count(*) present in query
- [CARBONDATA-2209
<https://issues.apache.org/jira/browse/CARBONDATA-2209>] - Rename table
with partitions not working issue and batch_sort and no_sort with partition
table issue
- [CARBONDATA-2211
<https://issues.apache.org/jira/browse/CARBONDATA-2211>] - Alter Table
Streaming DDL should blocking DDL like other DDL ( All DDL are blocking DDL)
- [CARBONDATA-2213
<https://issues.apache.org/jira/browse/CARBONDATA-2213>] - Wrong version
in datamap example module cause compilation failure
- [CARBONDATA-2216
<https://issues.apache.org/jira/browse/CARBONDATA-2216>] - Error in
compilation and execution in sdvtest
- [CARBONDATA-2217
<https://issues.apache.org/jira/browse/CARBONDATA-2217>] - nullpointer
issue drop partition where column does not exists and clean files issue
after second level of compaction
- [CARBONDATA-2219
<https://issues.apache.org/jira/browse/CARBONDATA-2219>] - Add
validation for external partition location to use same schema
- [CARBONDATA-2221
<https://issues.apache.org/jira/browse/CARBONDATA-2221>] - Drop table
should throw exception when metastore operation failed
- [CARBONDATA-2222
<https://issues.apache.org/jira/browse/CARBONDATA-2222>] - Update the
FAQ doc for some mistakes
- [CARBONDATA-2229
<https://issues.apache.org/jira/browse/CARBONDATA-2229>] - Unable to
save dataframe as carbontable with specified external database path
- [CARBONDATA-2232
<https://issues.apache.org/jira/browse/CARBONDATA-2232>] - Wrong logic
in spilling unsafe pages to disk
- [CARBONDATA-2235
<https://issues.apache.org/jira/browse/CARBONDATA-2235>] - add system
configuration to filter datamaps from show tables command
- [CARBONDATA-2236
<https://issues.apache.org/jira/browse/CARBONDATA-2236>] - Add SDV Test
Cases for Standard Partition
- [CARBONDATA-2237
<https://issues.apache.org/jira/browse/CARBONDATA-2237>] - Scala Parser
failures are accumulated into memory form thread local
- [CARBONDATA-2241
<https://issues.apache.org/jira/browse/CARBONDATA-2241>] - Wrong Query
written in Preaggregation Document
- [CARBONDATA-2244
<https://issues.apache.org/jira/browse/CARBONDATA-2244>] - When there
are some invisibility INSERT_IN_PROGRESS/INSERT_OVERWRITE_IN_PROGRESS
segments on main table, it can not create preaggregate table on it.
- [CARBONDATA-2248
<https://issues.apache.org/jira/browse/CARBONDATA-2248>] - Removing
parsers thread local objects after parsing of carbon query
- [CARBONDATA-2249
<https://issues.apache.org/jira/browse/CARBONDATA-2249>] - Not able to
query data through presto with local carbondata-store
- [CARBONDATA-2261
<https://issues.apache.org/jira/browse/CARBONDATA-2261>] - Support Set
segment command for Streaming Table
- [CARBONDATA-2264
<https://issues.apache.org/jira/browse/CARBONDATA-2264>] - There is
error when we create table using CarbonSource
- [CARBONDATA-2265
<https://issues.apache.org/jira/browse/CARBONDATA-2265>] - [DFX]-Load]:
Load job fails if 1 folder contains 1000 files
- [CARBONDATA-2266
<https://issues.apache.org/jira/browse/CARBONDATA-2266>] - All Examples
are throwing NoSuchElement Exception in current master branch
- [CARBONDATA-2274
<https://issues.apache.org/jira/browse/CARBONDATA-2274>] - Partition
table having more than 4 column giving zero record
- [CARBONDATA-2275
<https://issues.apache.org/jira/browse/CARBONDATA-2275>] - Query Failed
for 0 byte deletedelta file
- [CARBONDATA-2277
<https://issues.apache.org/jira/browse/CARBONDATA-2277>] - Filter on
default values are not working
- [CARBONDATA-2287
<https://issues.apache.org/jira/browse/CARBONDATA-2287>] - Add event to
alter partition table
- [CARBONDATA-2289
<https://issues.apache.org/jira/browse/CARBONDATA-2289>] - If carbon
merge index is enabled then after IUD operation if some blocks of a segment
is deleted, then during query and IUD operation the driver is throwing
FileNotFoundException while preparing BlockMetaInfo.
- [CARBONDATA-2302
<https://issues.apache.org/jira/browse/CARBONDATA-2302>] - Fix some bugs
when separate visible and invisible segments info into two files
- [CARBONDATA-2303
<https://issues.apache.org/jira/browse/CARBONDATA-2303>] - If dataload
is failed for parition table then cleanup is not working.
- [CARBONDATA-2307
<https://issues.apache.org/jira/browse/CARBONDATA-2307>] - OOM when
using DataFrame.coalesce
- [CARBONDATA-2308
<https://issues.apache.org/jira/browse/CARBONDATA-2308>] - Compaction
should be allow when loading is in progress
- [CARBONDATA-2314
<https://issues.apache.org/jira/browse/CARBONDATA-2314>] - Data mismatch
in Pre-Aggregate table after Streaming load due to threadset issue
- [CARBONDATA-2319
<https://issues.apache.org/jira/browse/CARBONDATA-2319>] -
carbon_scan_time and carbon_IO_time are incorrect in task statistics
- [CARBONDATA-2320
<https://issues.apache.org/jira/browse/CARBONDATA-2320>] - Fix error in
lucene coarse grain datamap suite
- [CARBONDATA-2321
<https://issues.apache.org/jira/browse/CARBONDATA-2321>] - Selecton
after a Concurrent Load Failing for Partition columns
- [CARBONDATA-2327
<https://issues.apache.org/jira/browse/CARBONDATA-2327>] - invalid
schema name _system shows when executed show schemas in presto
- [CARBONDATA-2329
<https://issues.apache.org/jira/browse/CARBONDATA-2329>] - Non
Serializable extra info in session is overwritten by values from thread
- [CARBONDATA-2333
<https://issues.apache.org/jira/browse/CARBONDATA-2333>] - Block insert
overwrite on parent table if any of the child tables are not partitioned on
the specified partition columns
- [CARBONDATA-2335
<https://issues.apache.org/jira/browse/CARBONDATA-2335>] - Autohandoff
is failing when preaggregate is created on streaming table
- [CARBONDATA-2337
<https://issues.apache.org/jira/browse/CARBONDATA-2337>] - Fix
duplicately acquiring 'streaming.lock' error when integrating with
spark-streaming
- [CARBONDATA-2343
<https://issues.apache.org/jira/browse/CARBONDATA-2343>] - Improper
filter resolver cause more filter scan on data that could be skipped
- [CARBONDATA-2346
<https://issues.apache.org/jira/browse/CARBONDATA-2346>] - Dropping
partition failing with null error for Partition table with Pre-Aggregate
tables
- [CARBONDATA-2347
<https://issues.apache.org/jira/browse/CARBONDATA-2347>] - Fix
Functional issues in LuceneDatamap in load and query and make stable
- [CARBONDATA-2350
<https://issues.apache.org/jira/browse/CARBONDATA-2350>] - Fix bugs in
minmax datamap example
- [CARBONDATA-2364
<https://issues.apache.org/jira/browse/CARBONDATA-2364>] - Remove
useless and time consuming code block
- [CARBONDATA-2366
<https://issues.apache.org/jira/browse/CARBONDATA-2366>] - Concurrent
Datamap creation is failing when using hive metastore
- [CARBONDATA-2374
<https://issues.apache.org/jira/browse/CARBONDATA-2374>] - Fix bugs in
minmax datamap example
- [CARBONDATA-2386
<https://issues.apache.org/jira/browse/CARBONDATA-2386>] - Query on
Pre-Aggregate table is slower
- [CARBONDATA-2391
<https://issues.apache.org/jira/browse/CARBONDATA-2391>] - Thread leak
in compaction operation if prefetch is enabled and compaction process is
killed
- [CARBONDATA-2394
<https://issues.apache.org/jira/browse/CARBONDATA-2394>] - Setting
segments in thread local space but not getting reflected in the driver
- [CARBONDATA-2401
<https://issues.apache.org/jira/browse/CARBONDATA-2401>] - Date and
Timestamp options are not working in SDK
- [CARBONDATA-2406
<https://issues.apache.org/jira/browse/CARBONDATA-2406>] - Dictionary
Server and Dictionary Client MD5 Validation failed with
hive.server2.enable.doAs = true
- [CARBONDATA-2408
<https://issues.apache.org/jira/browse/CARBONDATA-2408>] - Before
register to master, the master maybe not finished the start service.
- [CARBONDATA-2410
<https://issues.apache.org/jira/browse/CARBONDATA-2410>] - Error message
correction when column value length exceeds 320000 charactor
- [CARBONDATA-2413
<https://issues.apache.org/jira/browse/CARBONDATA-2413>] - After running
CarbonWriter, there is null directory about datamap
- [CARBONDATA-2417
<https://issues.apache.org/jira/browse/CARBONDATA-2417>] - SDK writer
goes to infinite wait when consumer thread goes dead
- [CARBONDATA-2419
<https://issues.apache.org/jira/browse/CARBONDATA-2419>] - sortColumns
Order we are getting wrong as we set for external table is fixed
- [CARBONDATA-2426
<https://issues.apache.org/jira/browse/CARBONDATA-2426>] - IOException
after compaction on Pre-Aggregate table on Partition table
- [CARBONDATA-2427
<https://issues.apache.org/jira/browse/CARBONDATA-2427>] - Fix
SearchMode Serialization Issue during Load
- [CARBONDATA-2431
<https://issues.apache.org/jira/browse/CARBONDATA-2431>] - Incremental
data added after table creation is not reflecting while doing select query.
- [CARBONDATA-2432
<https://issues.apache.org/jira/browse/CARBONDATA-2432>] - BloomFilter
DataMap should be contained in carbon assembly jar
- [CARBONDATA-2435
<https://issues.apache.org/jira/browse/CARBONDATA-2435>] - SDK
dependency Spark jar
- [CARBONDATA-2436
<https://issues.apache.org/jira/browse/CARBONDATA-2436>] - Block pruning
problem post the carbon schema restructure.
- [CARBONDATA-2437
<https://issues.apache.org/jira/browse/CARBONDATA-2437>] - Complex Type
data loading is failing is for null values
- [CARBONDATA-2438
<https://issues.apache.org/jira/browse/CARBONDATA-2438>] - Remove
spark/hadoop related classes in carbon assembly
- [CARBONDATA-2439
<https://issues.apache.org/jira/browse/CARBONDATA-2439>] - Update guava
version for bloom datamap
- [CARBONDATA-2440
<https://issues.apache.org/jira/browse/CARBONDATA-2440>] - In SDK user
can not specified the Unsafe memory , so it should take complete from Heap
, and it should not be sorted using unsafe.
- [CARBONDATA-2441
<https://issues.apache.org/jira/browse/CARBONDATA-2441>] - Implement
distribute interface for bloom datamap
- [CARBONDATA-2442
<https://issues.apache.org/jira/browse/CARBONDATA-2442>] - Reading two
sdk writer output with differnt schema should prompt exception
- [CARBONDATA-2463
<https://issues.apache.org/jira/browse/CARBONDATA-2463>] - if two insert
operations are running concurrently 1 task fails and causes wrong no of
records in select
- [CARBONDATA-2464
<https://issues.apache.org/jira/browse/CARBONDATA-2464>] - Fixed OOM in
case of complex type
- [CARBONDATA-2465
<https://issues.apache.org/jira/browse/CARBONDATA-2465>] - Improve the
carbondata file reliability in data load when direct hdfs write is enabled
- [CARBONDATA-2468
<https://issues.apache.org/jira/browse/CARBONDATA-2468>] - sortcolumns
considers all dimension also if few columns specified for sort_columns prop
- [CARBONDATA-2469
<https://issues.apache.org/jira/browse/CARBONDATA-2469>] - External
Table must show its location instead of default store path in describe
formatted
- [CARBONDATA-2472
<https://issues.apache.org/jira/browse/CARBONDATA-2472>] - Refactor
NonTransactional table code for Index file IO performance
- [CARBONDATA-2476
<https://issues.apache.org/jira/browse/CARBONDATA-2476>] - Fix bug in
bloom datamap cache
- [CARBONDATA-2477
<https://issues.apache.org/jira/browse/CARBONDATA-2477>] - No dictionary
Complex type with double/date/decimal data type table creation is failing
- [CARBONDATA-2479
<https://issues.apache.org/jira/browse/CARBONDATA-2479>] - Multiple
issue in sdk writer and external table flow
- [CARBONDATA-2480
<https://issues.apache.org/jira/browse/CARBONDATA-2480>] - Search mode
RuntimeException: Error while resolving filter expression
- [CARBONDATA-2486
<https://issues.apache.org/jira/browse/CARBONDATA-2486>] - set search
mode information is not updated in the documentation
- [CARBONDATA-2487
<https://issues.apache.org/jira/browse/CARBONDATA-2487>] - Block filters
for lucene with more than one text_match udf
- [CARBONDATA-2489
<https://issues.apache.org/jira/browse/CARBONDATA-2489>] - Fix coverity
reported warnings
- [CARBONDATA-2492
<https://issues.apache.org/jira/browse/CARBONDATA-2492>] - Thread leak
issue in case of any data load failure
- [CARBONDATA-2493
<https://issues.apache.org/jira/browse/CARBONDATA-2493>] -
DataType.equals() failes for complex types
- [CARBONDATA-2498
<https://issues.apache.org/jira/browse/CARBONDATA-2498>] - Change
CarbonWriterBuilder interface to take schema while creating writer
- [CARBONDATA-2503
<https://issues.apache.org/jira/browse/CARBONDATA-2503>] - Data write
fails if empty value is provided for sort columns in sdk
- [CARBONDATA-2520
<https://issues.apache.org/jira/browse/CARBONDATA-2520>] - datamap
writers are not getting closed on task failure
- [CARBONDATA-2538
<https://issues.apache.org/jira/browse/CARBONDATA-2538>] - No exception
is thrown if writer path has only lock files
- [CARBONDATA-2545
<https://issues.apache.org/jira/browse/CARBONDATA-2545>] - Fix some
spell error in CarbonData
- [CARBONDATA-2552
<https://issues.apache.org/jira/browse/CARBONDATA-2552>] - Fix Data
Mismatch for Complex Data type Array of Timestamp with Dictionary Include
- [CARBONDATA-2555
<https://issues.apache.org/jira/browse/CARBONDATA-2555>] - SDK Reader
should have isTransactionalTable = false by default, to be inline with SDK
writer
New Feature
- [CARBONDATA-1516
<https://issues.apache.org/jira/browse/CARBONDATA-1516>] - Support
pre-aggregate tables and timeseries in carbondata
- [CARBONDATA-2055
<https://issues.apache.org/jira/browse/CARBONDATA-2055>] - Support
integrating Streaming table with Spark Streaming
- [CARBONDATA-2242
<https://issues.apache.org/jira/browse/CARBONDATA-2242>] - Support
materialized view
- [CARBONDATA-2253
<https://issues.apache.org/jira/browse/CARBONDATA-2253>] - Support write
JSON/Avro data to carbon files
- [CARBONDATA-2262
<https://issues.apache.org/jira/browse/CARBONDATA-2262>] - Create table
should support using carbondata and stored as carbondata
- [CARBONDATA-2267
<https://issues.apache.org/jira/browse/CARBONDATA-2267>] - Implement
Reading Of Carbon Partition From Presto
- [CARBONDATA-2276
<https://issues.apache.org/jira/browse/CARBONDATA-2276>] - Support SDK
API to read schema in data file and schema file
- [CARBONDATA-2278
<https://issues.apache.org/jira/browse/CARBONDATA-2278>] - Save the
datamaps to system folder of warehouse
- [CARBONDATA-2291
<https://issues.apache.org/jira/browse/CARBONDATA-2291>] - Add datamap
status and refresh command to sync data manually to datamaps
- [CARBONDATA-2296
<https://issues.apache.org/jira/browse/CARBONDATA-2296>] - Test famework
should take the location of local module target folder if not integrtion
module
- [CARBONDATA-2297
<https://issues.apache.org/jira/browse/CARBONDATA-2297>] - Support
SEARCH_MODE for basic filter query
- [CARBONDATA-2312
<https://issues.apache.org/jira/browse/CARBONDATA-2312>] - Support In
Memory catalog
- [CARBONDATA-2323
<https://issues.apache.org/jira/browse/CARBONDATA-2323>] - Distributed
search mode using gRPC
- [CARBONDATA-2371
<https://issues.apache.org/jira/browse/CARBONDATA-2371>] - Add Profiler
output in EXPLAIN command
- [CARBONDATA-2373
<https://issues.apache.org/jira/browse/CARBONDATA-2373>] - Add bloom
filter datamap to support precise query
- [CARBONDATA-2378
<https://issues.apache.org/jira/browse/CARBONDATA-2378>] - Support
enable/disable search mode in ThriftServer
- [CARBONDATA-2380
<https://issues.apache.org/jira/browse/CARBONDATA-2380>] - Support
visible/invisible datamap for performance tuning
- [CARBONDATA-2415
<https://issues.apache.org/jira/browse/CARBONDATA-2415>] - All DataMap
should support REFRESH command
- [CARBONDATA-2416
<https://issues.apache.org/jira/browse/CARBONDATA-2416>] - Index DataMap
should support immediate load and deferred load when creating the DataMap
Improvement
- [CARBONDATA-1663
<https://issues.apache.org/jira/browse/CARBONDATA-1663>] - Decouple
spark in carbon modules
- [CARBONDATA-2018
<https://issues.apache.org/jira/browse/CARBONDATA-2018>] - Optimization
in reading/writing for sort temp row during data loading
- [CARBONDATA-2032
<https://issues.apache.org/jira/browse/CARBONDATA-2032>] - Skip writing
final data files to local disk to save disk IO in data loading
- [CARBONDATA-2099
<https://issues.apache.org/jira/browse/CARBONDATA-2099>] - Refactor on
query scan process to improve readability
- [CARBONDATA-2139
<https://issues.apache.org/jira/browse/CARBONDATA-2139>] - Optimize CTAS
documentation and test case
- [CARBONDATA-2140
<https://issues.apache.org/jira/browse/CARBONDATA-2140>] - Presto
Integration - Code Refactoring
- [CARBONDATA-2148
<https://issues.apache.org/jira/browse/CARBONDATA-2148>] - Use Row
parser to replace current default parser:CSVStreamParserImp
- [CARBONDATA-2159
<https://issues.apache.org/jira/browse/CARBONDATA-2159>] - Remove
carbon-spark dependency for sdk module
- [CARBONDATA-2168
<https://issues.apache.org/jira/browse/CARBONDATA-2168>] - Support
global sort on partition tables
- [CARBONDATA-2184
<https://issues.apache.org/jira/browse/CARBONDATA-2184>] - Improve
memory reuse for heap memory in `HeapMemoryAllocator`
- [CARBONDATA-2187
<https://issues.apache.org/jira/browse/CARBONDATA-2187>] - Restructure
the partition folders as per the standard hive folders
- [CARBONDATA-2196
<https://issues.apache.org/jira/browse/CARBONDATA-2196>] - during stream
sometime carbontable is null in executor side
- [CARBONDATA-2204
<https://issues.apache.org/jira/browse/CARBONDATA-2204>] - Access
tablestatus file too many times during query
- [CARBONDATA-2223
<https://issues.apache.org/jira/browse/CARBONDATA-2223>] - Adding
Listener Support for Partition
- [CARBONDATA-2226
<https://issues.apache.org/jira/browse/CARBONDATA-2226>] - Refactor UT's
to remove duplicate test scenarios to improve CI time for PreAggregate
create and drop feature
- [CARBONDATA-2227
<https://issues.apache.org/jira/browse/CARBONDATA-2227>] - Add Partition
Values and Location information in describe formatted for Standard
partition feature
- [CARBONDATA-2230
<https://issues.apache.org/jira/browse/CARBONDATA-2230>] - Add a path
into table path to store lock files and delete useless segment lock files
before loading
- [CARBONDATA-2231
<https://issues.apache.org/jira/browse/CARBONDATA-2231>] - Refactor FT's
to remove duplicate test scenarios to improve CI time for Streaming feature
- [CARBONDATA-2234
<https://issues.apache.org/jira/browse/CARBONDATA-2234>] - Support UTF-8
with BOM encoding in CSVInputFormat
- [CARBONDATA-2250
<https://issues.apache.org/jira/browse/CARBONDATA-2250>] - Reduce
massive object generation in global sort
- [CARBONDATA-2251
<https://issues.apache.org/jira/browse/CARBONDATA-2251>] - Refactored
sdv failures running on different environment
- [CARBONDATA-2254
<https://issues.apache.org/jira/browse/CARBONDATA-2254>] - Optimize
CarbonData documentation
- [CARBONDATA-2255
<https://issues.apache.org/jira/browse/CARBONDATA-2255>] - Should rename
the streaming examples to make it easy to understand
- [CARBONDATA-2256
<https://issues.apache.org/jira/browse/CARBONDATA-2256>] - Adding sdv
Testcases for SET_Parameter_Dynamically_Feature
- [CARBONDATA-2258
<https://issues.apache.org/jira/browse/CARBONDATA-2258>] - Separate
visible and invisible segments info into two files to reduce the size of
tablestatus file.
- [CARBONDATA-2260
<https://issues.apache.org/jira/browse/CARBONDATA-2260>] -
CarbonThriftServer should support S3 carbon table
- [CARBONDATA-2271
<https://issues.apache.org/jira/browse/CARBONDATA-2271>] - Collect SQL
execution information to driver side
- [CARBONDATA-2285
<https://issues.apache.org/jira/browse/CARBONDATA-2285>] - spark
integration code refactor
- [CARBONDATA-2295
<https://issues.apache.org/jira/browse/CARBONDATA-2295>] - Add
UNSAFE_WORKING_MEMORY_IN_MB as a configuration parameter in presto
integration
- [CARBONDATA-2298
<https://issues.apache.org/jira/browse/CARBONDATA-2298>] - Delete
segment lock files before update metadata
- [CARBONDATA-2299
<https://issues.apache.org/jira/browse/CARBONDATA-2299>] - Support
showing all segment information(include visible and invisible segments)
- [CARBONDATA-2304
<https://issues.apache.org/jira/browse/CARBONDATA-2304>] - Enhance
compaction performance by enabling prefetch
- [CARBONDATA-2310
<https://issues.apache.org/jira/browse/CARBONDATA-2310>] - Refactored
code to improve Distributable interface
- [CARBONDATA-2315
<https://issues.apache.org/jira/browse/CARBONDATA-2315>] - DataLoad is
showing success and failure message in log,when no data is loaded into
table during LOAD
- [CARBONDATA-2316
<https://issues.apache.org/jira/browse/CARBONDATA-2316>] - Even though
one of the Compaction task failed at executor. All the executor task is
showing success in UI and Job fails from driver.
- [CARBONDATA-2317
<https://issues.apache.org/jira/browse/CARBONDATA-2317>] - concurrent
datamap with same name and schema creation throws exception
- [CARBONDATA-2324
<https://issues.apache.org/jira/browse/CARBONDATA-2324>] - Support
config ExecutorService in search mode
- [CARBONDATA-2325
<https://issues.apache.org/jira/browse/CARBONDATA-2325>] - Page level
uncompress and Query performance improvement for Unsafe No Dictionary
- [CARBONDATA-2338
<https://issues.apache.org/jira/browse/CARBONDATA-2338>] - Add example
to upload data to S3 by using SDK
- [CARBONDATA-2341
<https://issues.apache.org/jira/browse/CARBONDATA-2341>] - Add CleanUp
for Pre-Aggregate table
- [CARBONDATA-2353
<https://issues.apache.org/jira/browse/CARBONDATA-2353>] - Add cache for
DataMap schema provider to avoid IO for each read
- [CARBONDATA-2357
<https://issues.apache.org/jira/browse/CARBONDATA-2357>] - Add column
name and index mapping in lucene datamap writer
- [CARBONDATA-2358
<https://issues.apache.org/jira/browse/CARBONDATA-2358>] - Dataframe
overwrite does not work properly if the table is already created and has
deleted segments
- [CARBONDATA-2365
<https://issues.apache.org/jira/browse/CARBONDATA-2365>] - Add
QueryExecutor in SearchMode for row-based CarbonRecordReader
- [CARBONDATA-2375
<https://issues.apache.org/jira/browse/CARBONDATA-2375>] - Add CG prune
before FG prune
- [CARBONDATA-2376
<https://issues.apache.org/jira/browse/CARBONDATA-2376>] - Improve
Lucene datamap performance by eliminating blockid while writing and reading
index.
- [CARBONDATA-2379
<https://issues.apache.org/jira/browse/CARBONDATA-2379>] - Support
Search mode run in the cluster and fix some error
- [CARBONDATA-2381
<https://issues.apache.org/jira/browse/CARBONDATA-2381>] - Improve
compaction performance by filling batch result in columnar format and
performing IO at blocklet level
- [CARBONDATA-2384
<https://issues.apache.org/jira/browse/CARBONDATA-2384>] - SDK support
write/read data into/from S3
- [CARBONDATA-2390
<https://issues.apache.org/jira/browse/CARBONDATA-2390>] - Refresh
Lucene data map for the exists table with data
- [CARBONDATA-2392
<https://issues.apache.org/jira/browse/CARBONDATA-2392>] - Add close
method for CarbonReader
- [CARBONDATA-2396
<https://issues.apache.org/jira/browse/CARBONDATA-2396>] - Add CTAS
support for using DataSource Syntax
- [CARBONDATA-2404
<https://issues.apache.org/jira/browse/CARBONDATA-2404>] - Add
documentation for using carbondata and stored as carbondata
- [CARBONDATA-2407
<https://issues.apache.org/jira/browse/CARBONDATA-2407>] - Removed All
Unused Executor BTree code
- [CARBONDATA-2414
<https://issues.apache.org/jira/browse/CARBONDATA-2414>] - Optimize
documents for sort_column_bounds
- [CARBONDATA-2422
<https://issues.apache.org/jira/browse/CARBONDATA-2422>] - Search mode
Master port should be dynamic
- [CARBONDATA-2448
<https://issues.apache.org/jira/browse/CARBONDATA-2448>] - Adding
compacted segments to load and alter events
- [CARBONDATA-2454
<https://issues.apache.org/jira/browse/CARBONDATA-2454>] - Add false
positive probability property for bloom filter datamap
- [CARBONDATA-2455
<https://issues.apache.org/jira/browse/CARBONDATA-2455>] - Fix _System
Folder creation and lucene AND,OR,NOT Filter fix
- [CARBONDATA-2458
<https://issues.apache.org/jira/browse/CARBONDATA-2458>] - Remove
unnecessary TableProvider interface
- [CARBONDATA-2459
<https://issues.apache.org/jira/browse/CARBONDATA-2459>] - Support cache
for bloom datamap
- [CARBONDATA-2467
<https://issues.apache.org/jira/browse/CARBONDATA-2467>] - Null is
printed in the SDK writer logs for operations logged
- [CARBONDATA-2470
<https://issues.apache.org/jira/browse/CARBONDATA-2470>] - Refactor
AlterTableCompactionPostStatusUpdateEvent usage in compaction flow
- [CARBONDATA-2473
<https://issues.apache.org/jira/browse/CARBONDATA-2473>] - Support
Materialized View as enhanced Preaggregate DataMap
- [CARBONDATA-2494
<https://issues.apache.org/jira/browse/CARBONDATA-2494>] - Improve
Lucene datamap size and performnace.
- [CARBONDATA-2495
<https://issues.apache.org/jira/browse/CARBONDATA-2495>] - Add document
for bloomfilter datamap
- [CARBONDATA-2496
<https://issues.apache.org/jira/browse/CARBONDATA-2496>] - Chnage the
bloom implementation to hadoop for better performance and compression
- [CARBONDATA-2524
<https://issues.apache.org/jira/browse/CARBONDATA-2524>] - Support
create carbonReader with default projection
Test
- [CARBONDATA-2073
<https://issues.apache.org/jira/browse/CARBONDATA-2073>] - Add test
cases for pre aggregate table
- [CARBONDATA-2257
<https://issues.apache.org/jira/browse/CARBONDATA-2257>] - Add SDV test
cases for Partition with Global Sort
- [CARBONDATA-2352
<https://issues.apache.org/jira/browse/CARBONDATA-2352>] - Add SDV Test
Cases for Partition with Pre-Aggregate table
- [CARBONDATA-2356
<https://issues.apache.org/jira/browse/CARBONDATA-2356>] - Adding UT for
Lucene DataMap
Task
-
- [CARBONDATA-1827
<https://issues.apache.org/jira/browse/CARBONDATA-1827>] - Add
Support to provide S3 Functionality in Carbondata
- [CARBONDATA-1959
<https://issues.apache.org/jira/browse/CARBONDATA-1959>] - Support
compaction on S3 table
- [CARBONDATA-1960
<https://issues.apache.org/jira/browse/CARBONDATA-1960>] - Add
example for creating a local table and load CSV data which is
stored in S3.
- [CARBONDATA-1961
<https://issues.apache.org/jira/browse/CARBONDATA-1961>] - Support
data update/delete on S3 table
- [CARBONDATA-2135
<https://issues.apache.org/jira/browse/CARBONDATA-2135>] -
Documentation for Table Comment and Column Comment
- [CARBONDATA-2138
<https://issues.apache.org/jira/browse/CARBONDATA-2138>] -
Documentation for HEADER option
- [CARBONDATA-2214
<https://issues.apache.org/jira/browse/CARBONDATA-2214>] - Remove
config 'spark.sql.hive.thriftServer.singleSession' from
installation-guide.md
- [CARBONDATA-2215
<https://issues.apache.org/jira/browse/CARBONDATA-2215>] - Add the
description of Carbon Stream Parser into streaming-guide.md
- [CARBONDATA-2259
<https://issues.apache.org/jira/browse/CARBONDATA-2259>] - Add auto
CI for examples
- [CARBONDATA-2300
<https://issues.apache.org/jira/browse/CARBONDATA-2300>] - Add
ENABLE_UNSAFE_IN_QUERY_EXECUTION as a configuration parameter in presto
integration
- [CARBONDATA-2370
<https://issues.apache.org/jira/browse/CARBONDATA-2370>] - Document
for Presto cluster setup for carbondata
- [CARBONDATA-2424
<https://issues.apache.org/jira/browse/CARBONDATA-2424>] - Add
documentation for properties of Pre-Aggregate tables
- [CARBONDATA-2434
<https://issues.apache.org/jira/browse/CARBONDATA-2434>] - Add
ExternalTableExample and LuceneDataMapExample
- [CARBONDATA-2507
<https://issues.apache.org/jira/browse/CARBONDATA-2507>] - Some
properties not validate in CarbonData, like enable.offheap.sort
Updated release notes . Re: [ANNOUNCE] Apache CarbonData 1.4.0
release
Posted by Liang Chen <ch...@gmail.com>.
Hi
Please find the updated 1.4.0 release notes:
https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.4.0+Release
Regards
Liang
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/