You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "duanyongvictory (via GitHub)" <gi...@apache.org> on 2023/04/07 14:15:12 UTC
[GitHub] [hudi] duanyongvictory opened a new issue, #8405: [SUPPORT] sql query error after ddl operation
duanyongvictory opened a new issue, #8405:
URL: https://github.com/apache/hudi/issues/8405
spark3.2.1
set hoodie.schema.on.read.enable=true;
STEP1 create table：
create table test_02 (
column_name_1 string,
column_name_2 string,
column_name_3 string,
column_name_4 string,
column_name_5 string,
column_name_6 string,
column_name_7 string,
column_name_8 string,
column_name_9 string,
column_name_10 string,
column_name_11 string,
column_name_12 string,
column_name_13 string,
column_name_14 string,
column_name_15 string,
column_name_16 string,
column_name_17 string,
column_name_18 string,
column_name_19 string,
column_name_20 string,
column_name_21 string,
column_name_22 string,
column_name_23 string,
column_name_24 string,
column_name_25 string,
column_name_26 string,
column_name_27 string,
column_name_28 string,
column_name_29 string,
column_name_30 string,
column_name_31 string,
column_name_32 string,
column_name_33 string,
column_name_34 string,
column_name_35 string,
column_name_36 string,
column_name_37 string,
column_name_38 string,
column_name_39 string,
column_name_40 string,
column_name_41 string,
column_name_42 string,
column_name_43 string,
column_name_44 string,
column_name_45 string,
column_name_46 string,
column_name_47 string,
column_name_48 string,
column_name_49 string,
column_name_50 string,
column_name_51 string,
column_name_52 string,
column_name_53 string,
column_name_54 string,
column_name_55 string,
column_name_56 string,
column_name_57 string,
column_name_58 string,
column_name_59 string,
column_name_60 string,
column_name_61 string,
column_name_62 string,
column_name_63 string,
column_name_64 string,
column_name_65 string,
column_name_66 string,
column_name_67 string,
column_name_68 string,
column_name_69 string,
column_name_70 string,
column_name_71 string,
column_name_72 string,
column_name_73 string,
column_name_74 string,
column_name_75 string,
column_name_76 string,
column_name_77 string,
column_name_78 string,
column_name_79 string,
column_name_80 string,
column_name_81 string,
column_name_82 string,
column_name_83 string,
column_name_84 string,
column_name_85 string,
column_name_86 string,
column_name_87 string,
column_name_88 string,
column_name_89 string,
column_name_90 string,
column_name_91 string,
column_name_92 string,
column_name_93 string,
column_name_94 string,
column_name_95 string,
column_name_96 string,
column_name_97 string,
column_name_98 string,
column_name_99 string,
column_name_100 string,
column_name_101 string,
column_name_102 string,
column_name_103 string,
column_name_104 string,
column_name_105 string,
column_name_106 string,
column_name_107 string,
column_name_108 string,
column_name_109 string,
column_name_110 string,
column_name_111 string,
column_name_112 string,
column_name_113 string,
column_name_114 string,
column_name_115 string,
column_name_116 string,
column_name_117 string,
column_name_118 string,
column_name_119 string,
column_name_120 string,
column_name_121 string,
column_name_122 string,
column_name_123 string,
column_name_124 string,
column_name_125 string,
column_name_126 string,
column_name_127 string,
column_name_128 string,
column_name_129 string,
column_name_130 string,
column_name_131 string,
column_name_132 string,
column_name_133 string,
column_name_134 string,
column_name_135 string,
column_name_136 string,
column_name_137 string,
column_name_138 string,
column_name_139 string,
column_name_140 string,
column_name_141 string,
column_name_142 string,
column_name_143 string,
column_name_144 string,
column_name_145 string,
column_name_146 string,
column_name_147 string,
column_name_148 string,
column_name_149 string,
column_name_150 string,
column_name_151 string,
column_name_152 string,
column_name_153 string,
column_name_154 string,
column_name_155 string,
column_name_156 string,
column_name_157 string,
ts bigint)
using hudi
tblproperties (
type = 'mor',
primaryKey = 'column_name_1',
preCombineField = 'ts'
);
STEP2 insert data:
insert into test_02
select
'1',
'column_name_2',
'column_name_3',
'column_name_4',
'column_name_5',
'column_name_6',
'column_name_7',
'column_name_8',
'column_name_9',
'column_name_10',
'column_name_11',
'column_name_12',
'column_name_13',
'column_name_14',
'column_name_15',
'column_name_16',
'column_name_17',
'column_name_18',
'column_name_19',
'column_name_20',
'column_name_21',
'column_name_22',
'column_name_23',
'column_name_24',
'column_name_25',
'column_name_26',
'column_name_27',
'column_name_28',
'column_name_29',
'column_name_30',
'column_name_31',
'column_name_32',
'column_name_33',
'column_name_34',
'column_name_35',
'column_name_36',
'column_name_37',
'column_name_38',
'column_name_39',
'column_name_40',
'column_name_41',
'column_name_42',
'column_name_43',
'column_name_44',
'column_name_45',
'column_name_46',
'column_name_47',
'column_name_48',
'column_name_49',
'column_name_50',
'column_name_51',
'column_name_52',
'column_name_53',
'column_name_54',
'column_name_55',
'column_name_56',
'column_name_57',
'column_name_58',
'column_name_59',
'column_name_60',
'column_name_61',
'column_name_62',
'column_name_63',
'column_name_64',
'column_name_65',
'column_name_66',
'column_name_67',
'column_name_68',
'column_name_69',
'column_name_70',
'column_name_71',
'column_name_72',
'column_name_73',
'column_name_74',
'column_name_75',
'column_name_76',
'column_name_77',
'column_name_78',
'column_name_79',
'column_name_80',
'column_name_81',
'column_name_82',
'column_name_83',
'column_name_84',
'column_name_85',
'column_name_86',
'column_name_87',
'column_name_88',
'column_name_89',
'column_name_90',
'column_name_91',
'column_name_92',
'column_name_93',
'column_name_94',
'column_name_95',
'column_name_96',
'column_name_97',
'column_name_98',
'column_name_99',
'column_name_100',
'column_name_101',
'column_name_102',
'column_name_103',
'column_name_104',
'column_name_105',
'column_name_106',
'column_name_107',
'column_name_108',
'column_name_109',
'column_name_110',
'column_name_111',
'column_name_112',
'column_name_113',
'column_name_114',
'column_name_115',
'column_name_116',
'column_name_117',
'column_name_118',
'column_name_119',
'column_name_120',
'column_name_121',
'column_name_122',
'column_name_123',
'column_name_124',
'column_name_125',
'column_name_126',
'column_name_127',
'column_name_128',
'column_name_129',
'column_name_130',
'column_name_131',
'column_name_132',
'column_name_133',
'column_name_134',
'column_name_135',
'column_name_136',
'column_name_137',
'column_name_138',
'column_name_139',
'column_name_140',
'column_name_141',
'column_name_142',
'column_name_143',
'column_name_144',
'column_name_145',
'column_name_146',
'column_name_147',
'column_name_148',
'column_name_149',
'column_name_150',
'column_name_151',
'column_name_152',
'column_name_153',
'column_name_154',
'column_name_155',
'column_name_156',
'column_name_157',
1;
STEP3 query data is ok：
select * from test_02;
STEP4 change schema：
ALTER TABLE test_02 ADD COLUMNS (column_name_158 string);
STEP5 query data is error:
select * from test_02;
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 209.0 failed 1 times, most recent failure: Lost task 0.0 in stage 209.0 (TID 2569) (executor driver): java.lang.ArrayIndexOutOfBoundsException: -1
at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.isNullAt(OnHeapColumnVector.java:130)
at org.apache.spark.sql.vectorized.ColumnarBatchRow.isNullAt(ColumnarBatch.java:190)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.execution.RowDataSourceScanExec.$anonfun$doExecute$2(DataSourceScanExec.scala:123)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:350)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
when the number of fields is fewer, say 10 or 20 fields, it is ok;
could any one check this out?
thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8405: [SUPPORT] sql query error after ddl operation
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8405:
URL: https://github.com/apache/hudi/issues/8405#issuecomment-1538003319
@xiarixiaoyao Closing the issue as this is known thing with spark sql and it worked with the setting. Please reopen in case of any concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #8405: [SUPPORT] sql query error after ddl operation
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8405: [SUPPORT] sql query error after ddl operation
URL: https://github.com/apache/hudi/issues/8405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8405: [SUPPORT] sql query error after ddl operation
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8405:
URL: https://github.com/apache/hudi/issues/8405#issuecomment-1501692851
@duanyongvictory I was able to reproduce your issue in my local setup and after setting the configuration what @xiarixiaoyao suggested, I was able to query test_02.
Did this worked for you also? If yes can we close the issue or you need any other help on the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #8405: [SUPPORT] sql query error after ddl operation
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on issue #8405:
URL: https://github.com/apache/hudi/issues/8405#issuecomment-1500762333
pls set spark.sql.codegen.maxFields> number of your columns
eg: spark.sessionState.conf.setConfString("spark.sql.codegen.maxFields", "1000")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org