[GitHub] [spark] cxzl25 commented on pull request #35253: [SPARK-37965][SQL] Remove check field name when reading/writing existing data in Orc

[GitBox <gi...@apache.org>]

cxzl25 commented on PR #35253:
URL: https://github.com/apache/spark/pull/35253#issuecomment-1321866542

   I think we still check for the empty character case, or add the `supportFieldName` method back.
   
   Here are some tests.
   
   ## native
   #### native write
   ```sql
   set spark.sql.orc.impl=native;
   create table t_1 stored as orc  as select '' ;
   ```
   suceess.
   #### native read
   ````sql
   set spark.sql.orc.impl=native;
   select t_1;
   ````
   
   ```
   java.lang.IllegalArgumentException: Empty quoted field name at '``^'
           at org.apache.orc.impl.ParserUtils.parseName(ParserUtils.java:114)
           at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:352)
           at org.apache.orc.impl.OrcTail.<init>(OrcTail.java:72)
           at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:845)
           at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:566)
           at org.apache.orc.OrcFile.createReader(OrcFile.java:385)
           at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$2(OrcFileFormat.scala:147)
   
   ```
   
   #### hive read
   ```sql
   set spark.sql.orc.impl=hive;
   select t_1;
   ```
   
   ```
   java.lang.IllegalArgumentException: Empty quoted field name at '``^'
           at org.apache.orc.impl.ParserUtils.parseName(ParserUtils.java:114)
           at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:352)
           at org.apache.orc.impl.OrcTail.<init>(OrcTail.java:72)
           at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:845)
           at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:566)
           at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:63)
           at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:55)
           at org.apache.spark.sql.hive.orc.OrcFileOperator$.$anonfun$getFileReader$3(OrcFileOperator.scala:76)
   
   ```
   
   ## hive
   #### hive write
   ```sql
   set spark.sql.orc.impl=hive;
   create table t_1 stored as orc  as select '' ;
   ```
   
   ```
   java.lang.IllegalArgumentException: Error: name expected at the position 7 of 'struct<:string>' but ':' is found.
           at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:378)
           at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:502)
           at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:329)
           at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfoFromTypeString(TypeInfoUtils.java:831)
           at org.apache.spark.sql.hive.orc.OrcSerializer.<init>(OrcFileFormat.scala:242)
           at org.apache.spark.sql.hive.orc.OrcOutputWriter.<init>(OrcFileFormat.scala:279)
           at org.apache.spark.sql.hive.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:110)
           at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:161)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org