You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "guojingfeng (Jira)" <ji...@apache.org> on 2020/07/03 09:01:00 UTC
[jira] [Commented] (IMPALA-9896) OutOfMemoryError: Requested array
size exceeds VM limit when LocalCatalog is enabled
[ https://issues.apache.org/jira/browse/IMPALA-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150850#comment-17150850 ]
guojingfeng commented on IMPALA-9896:
-------------------------------------
Hi, table information of catalogd web metrics is：
{code:java}
has-incremental-stats: false
memory-estimate-bytes: 5633372814
num-blocks: 5966619
num-files: 5966619
num-partitions: 856968
total-file-size-bytes: 157264108309954{code}
In catalogd logs the whole table size is
{code:java}
// code placeholder
Collected update: 1:TABLE:nameOfBigTable, version=145837, original size=1463488269, compressed size=310079247
{code}
> OutOfMemoryError: Requested array size exceeds VM limit when LocalCatalog is enabled
> ------------------------------------------------------------------------------------
>
> Key: IMPALA-9896
> URL: https://issues.apache.org/jira/browse/IMPALA-9896
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.2.0
> Reporter: abeltian
> Priority: Major
> Labels: crashed, performance
>
> OutOfMemoryError: Requested array size exceeds VM limit when LocalCatalog is enabled.
> The basic information of the large table is as follows:
> 101 columns, 785243 partitions, 5729866 files.
> {code:java}
> I0626 20:59:04.029678 3392438 jni-util.cc:256] java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> I0626 20:59:04.030231 3392438 status.cc:124] OutOfMemoryError: Requested array size exceeds VM limit
> @ 0xb35f19
> @ 0x113112e
> @ 0xb23b87
> @ 0xb0e339
> @ 0xc15a52
> @ 0xc09e4c
> @ 0xb01de9
> @ 0xf159e8
> @ 0xf0cd7e
> @ 0xf0dc11
> @ 0x11a1e3f
> @ 0x11a29e9
> @ 0x1790be9
> @ 0x7f55188a2e24
> @ 0x7f55185cf35c
> E0626 20:59:04.030258 3392438 catalog-server.cc:176] OutOfMemoryError: Requested array size exceeds VM limit
> {code}
> The source code corresponding to the error is as follows：
> {code:java}
> void GetPartialCatalogObject(TGetPartialCatalogObjectResponse& resp,
> const TGetPartialCatalogObjectRequest& req) override {
> // TODO(todd): capture detailed metrics on the types of inbound requests, lock
> // wait times, etc.
> // TODO(todd): add some kind of limit on the number of concurrent requests here
> // to avoid thread exhaustion -- eg perhaps it would be best to use a trylock
> // on the catalog locks, or defer these calls to a separate (bounded) queue,
> // so a heavy query workload against a table undergoing a slow refresh doesn't
> // end up taking down the catalog by creating thousands of threads.
> VLOG_RPC << "GetPartialCatalogObject(): request=" << ThriftDebugString(req);
> Status status = catalog_server_->catalog()->GetPartialCatalogObject(req, &resp);
> if (!status.ok()) LOG(ERROR) << status.GetDetail(); //catalog-server.cc:176
> TStatus thrift_status;
> status.ToThrift(&thrift_status);
> resp.__set_status(thrift_status);
> VLOG_RPC << "GetPartialCatalogObject(): response=" << ThriftDebugString(resp);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org