You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/06 10:28:44 UTC

[GitHub] [spark] StefanXiepj opened a new pull request #29010: [SPARK-32192][SQL] Print column name when throws ClassCastException

StefanXiepj opened a new pull request #29010:
URL: https://github.com/apache/spark/pull/29010

### What changes were proposed in this pull request?

When somebody changed the type of partition's field, spark will throw ClassCastException. For example, we have a table like this:
```
drop table if exists cast_exception_test;

create table cast_exception_test(c1 int, c2 string) partitioned by (dt string) stored as orc;

insert into table cast_exception_test partition(dt='2020-04-08') values('1', 'jeff_1');
```

If you change the field's type, spark will throw ClassCastException, but hive will not:
```
-- change the field's type
alter table cast_exception_test change column c1 c1 string;
-- hive correct, but spark throws ClassCastException
select * from cast_exception_test where dt='2020-04-08';
```

### Why are the changes needed?

When the table has many fields, we don's known which field has been changed. If we print out log about this exception, it will very helpful for us to troubleshoot.

### Does this PR introduce _any_ user-facing change?

When the ClassCastException is caused by changed field's type, you can search which field has problem in exexutor logs:
```
20/04/09 17:22:05 ERROR hive.HadoopTableReader: Exception thrown in field <c1>
```

### How was this patch tested?

```
drop table if exists cast_exception_test;
create table cast_exception_test(c1 int, c2 string) partitioned by (dt string) stored as orc;
insert into table cast_exception_test partition(dt='2020-04-08') values('1', 'jeff_1');
alter table cast_exception_test change column c1 c1 string;
select * from cast_exception_test where dt='2020-04-08';
```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org