[jira] [Commented] (IMPALA-2376) Scan of array value with 100m elements with reasonable mem limit hits DCHECK.

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085019#comment-17085019 ] 

ASF subversion and git services commented on IMPALA-2376:
---------------------------------------------------------

Commit dc410a2cf47bcf06a0f4563d05a9d0a339af5fb2 in impala's branch refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=dc410a2 ]

IMPALA-9596: deflake test_tpch_mem_limit_single_node

This changes the test to use a debug action instead of
trying to hit the memory limit in the right spot, which
has tended to be flaky. This still exercises the error
handling code in the scanner, which was the original
point of the test (see IMPALA-2376).

This revealed an actual bug in the ORC scanner, where
it was not returning the error directly from
AssembleCollection(). Before I fixed that, the scanner
got stuck in an infinite loop when running the test.

Change-Id: I4678963c264b7c15fbac6f71721162b38676aa21
Reviewed-on: http://gerrit.cloudera.org:8080/15700
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Gabor Kaszab <ga...@cloudera.com>


> Scan of array value with 100m elements with reasonable mem limit hits DCHECK.
> -----------------------------------------------------------------------------
>
>                 Key: IMPALA-2376
>                 URL: https://issues.apache.org/jira/browse/IMPALA-2376
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Impala 2.3.0
>            Reporter: Alexander Behm
>            Assignee: Skye Wanderman-Milne
>            Priority: Blocker
>              Labels: crash, nested_types, resource-management
>
> The query below when run without a mem limit needs roughly 2.4g of memory in the scan.
> My expectation is that I get a mem limit exceeded error when running the same query with a mem limit below that 2.4g. However, we hit a DCHECK in the scanner.
> Repro:
> 1. Grab Parquet file from here:
> vd0212.halxg.cloudera.com:/data/1/huge_array_parquet/100m_array.parq
> 2. Copy file to HDFS and use CREATE TABLE LIKE FILE
> 3. The query below runs fine without a mem limit:
> {code}
> select cnt from huge_array_table t, (select count(item) cnt from t.f) v;
> {code}
> 4. Set the mem limit to 1g and run the query again. You will hit this DCHECK:
> {code}
> hdfs-parquet-scanner.cc:1299] Check failed: !parse_status_.ok()
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org