You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "SHREELEKHYA GAMPA (Jira)" <ji...@apache.org> on 2021/06/24 05:44:00 UTC
[jira] [Created] (CARBONDATA-4231) On update operation with 3.1v,
cloned spark session is used and set properties are lost.
SHREELEKHYA GAMPA created CARBONDATA-4231:
---------------------------------------------
Summary: On update operation with 3.1v, cloned spark session is used and set properties are lost.
Key: CARBONDATA-4231
URL: https://issues.apache.org/jira/browse/CARBONDATA-4231
Project: CarbonData
Issue Type: Bug
Reporter: SHREELEKHYA GAMPA
*Update operation with bad records property fails with 3.1v.*
*[Steps to reproduce]:*
0: jdbc:hive2://linux-221:22550/> set carbon.options.bad.records.action=force;
+------------------------------------+--------+
| key | value |
+------------------------------------+--------+
| carbon.options.bad.records.action | force |
+------------------------------------+--------+
1 row selected (0.04 seconds)
0: jdbc:hive2://linux-221:22550/> create table t_carbn1(item_type_cd int, sell_price bigint, profit decimal(10,4), item_name string, update_time timestamp) stored a
+---------+
| Result |
+---------+
+---------+
No rows selected (2.117 seconds)
0: jdbc:hive2://linux-221:22550/> insert into t_carbn1 select 2, 100000,23.3,'Apple','2012-11-11 11:11:11';
INFO : Execution ID: 858
+-------------+
| Segment ID |
+-------------+
| 0 |
+-------------+
1 row selected (4.278 seconds)
0: jdbc:hive2://linux-221:22550/> update t_carbn1 set (item_type_cd) = (item_type_cd/1);
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.RuntimeException: Update operation failed. DataLoad failure
*[Root cause]:*
On update command, persist is called and with latest 3.1 spark changes, spark returns a cloned SparkSession from cacheManager with all specified configurations disabled. As now its using different sparkSession for 3.1 which is not initialized in CarbonEnv. So CarbonEnv.init is called where new CarbonSessionInfo is created with no sessionParams. So, the properties set were not accessible.
Spark creates cloned spark session based on following properties:
1. spark.sql.optimizer.canChangeCachedPlanOutputPartitioning
2. spark.sql.sources.bucketing.autoBucketedScan.enabled
3. spark.sql.adaptive.enabled
--
This message was sent by Atlassian Jira
(v8.3.4#803005)