You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (Jira)" <ji...@apache.org> on 2025/03/06 01:58:00 UTC
[jira] [Commented] (SPARK-49489) 'TTransportException: MaxMessageSize reached' occurs if get a table with a large number of partitions
[ https://issues.apache.org/jira/browse/SPARK-49489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932803#comment-17932803 ]
Yuming Wang commented on SPARK-49489:
-------------------------------------
Since Spark 4.0, Please try to {{set spark.hive.thrift.client.max.message.size=1gb}}.
> 'TTransportException: MaxMessageSize reached' occurs if get a table with a large number of partitions
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-49489
> URL: https://issues.apache.org/jira/browse/SPARK-49489
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Yuming Wang
> Assignee: Cheng Pan
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: Internal fix.png
>
>
> {noformat}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
> at org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
> at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
> at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
> at org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
> at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
> at org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
> at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
> at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org