[jira] [Created] (SPARK-40885) Spark will filter out data field sorting when dynamic partitions and data fields are sorted at the same time

["zzzzming95 (Jira)" <ji...@apache.org>]

zzzzming95 created SPARK-40885:
----------------------------------

             Summary: Spark will filter out data field sorting when dynamic partitions and data fields are sorted at the same time
                 Key: SPARK-40885
                 URL: https://issues.apache.org/jira/browse/SPARK-40885
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.2.2, 3.3.0, 3.1.2
            Reporter: zzzzming95
             Fix For: 3.4.0


When using dynamic partitions to write data and sort partitions and data fields, Spark will filter the sorting of data fields.

 

reproduce sql:
{code:java}
CREATE TABLE `sort_table`(
  `id` int,
  `name` string
  )
PARTITIONED BY (
  `dt` string)
stored as textfile
LOCATION 'sort_table';CREATE TABLE `test_table`(
  `id` int,
  `name` string)
PARTITIONED BY (
  `dt` string)
stored as textfile
LOCATION
  'test_table';//gen test data
insert into test_table partition(dt=20221011) select 10,"15" union all select 1,"10" union  all select 5,"50" union  all select 20,"2" union  all select 30,"14"  ;
set spark.hadoop.hive.exec.dynamici.partition=true
set spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict
insert overwrite table sort_table partition(dt) select id,name,dt from test_table order by name,dt;
 {code}
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org