[jira] [Created] (IMPALA-8782) Impala errors can make profiles huge

["Gabor Kaszab (JIRA)" <ji...@apache.org>]

Gabor Kaszab created IMPALA-8782:
------------------------------------

             Summary: Impala errors can make profiles huge
                 Key: IMPALA-8782
                 URL: https://issues.apache.org/jira/browse/IMPALA-8782
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 2.9.0
            Reporter: Gabor Kaszab


We observed that in some cases Impala could flood the query profile with error messages causing other systems that read the profiles crash (e.g. Cloudera Manager) due to the size of the profile. Let's consider limiting the number of errors we write into the profile.

In the particular case I saw the profile was full of these errors, one of this for each row:
{code:java}
Error converting column: 4 to TIMESTAMP\nError converting column: 10 to INT\nError converting column: 11 to INT\nError converting column: 13 to INT\nError converting column: 15 to INT\nError converting column: 16 to INT\nError converting column: 19 to INT\nError converting column: 20 to INT\nError converting column: 21 to INT\nError converting column: 22 to INT\nError converting column: 23 to INT\nError converting column: 24 to INT\nError converting column: 26 to INT\nError converting column: 29 to INT\nError converting column: 31 to INT\nError converting column: 35 to INT\nError converting column: 36 to INT\nError converting column: 37 to INT\nError converting column: 39 to INT\nError converting column: 40 to INT\nError converting column: 41 to INT\nError converting column: 42 to INT\nError converting column: 43 to INT\nError converting column: 44 to INT\nError converting column: 45 to INT\nError converting column: 46 to INT\nError converting column: 48 to INT\nError converting column: 50 to INT\nError converting column: 51 to INT\nError converting column: 53 to INT\nError converting column: 54 to INT\nError converting column: 59 to INT\nError converting column: 60 to INT\nError converting column: 63 to INT\nError converting column: 64 to INT\nError converting column: 65 to INT\nError converting column: 69 to INT\nError converting column: 70 to INT\nError converting column: 71 to INT\nError converting column: 72 to INT\nError converting column: 73 to INT\nError converting column: 75 to INT\nError converting column: 76 to INT\nError converting column: 77 to INT\nError converting column: 78 to INT\nError converting column: 79 to INT\nError converting column: 80 to INT\nError converting column: 81 to INT\nError converting column: 82 to INT\nError converting column: 83 to INT\nError converting column: 84 to INT\nError converting column: 85 to INT\nError converting column: 87 to INT\nError converting column: 88 to INT\nError converting column: 90 to INT\nError converting column: 91 to INT\nError converting column: 95 to INT\nError converting column: 96 to INT\nError converting column: 97 to INT\nError converting column: 99 to INT\nError converting column: 102 to INT\nError converting column: 103 to INT\nError converting column: 104 to INT\nError converting column: 106 to INT\nError converting column: 107 to INT\nError converting column: 109 to INT\nError converting column: 117 to INT\nError converting column: 119 to INT\nError parsing row: file: hdfs://nameserviceci/data/CTL/encrypt/db/ingest/raw/capm/file=interfaces/dt=20170626/interfaces.1498453200039.snappy, before offset: 43276630\n
{code}
As a result, the uncompressed profile grew something like 700MB.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)