You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/07/17 04:18:00 UTC
[jira] [Commented] (IMPALA-13161) impalad crash -- impala::DelimitedTextParser::ParseFieldLocations
[ https://issues.apache.org/jira/browse/IMPALA-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866608#comment-17866608 ]
ASF subversion and git services commented on IMPALA-13161:
----------------------------------------------------------
Commit 5c4e771241a7f847d2349ae248bc268243e071ed in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5c4e77124 ]
IMPALA-13161: Fix column index overflow in DelimitedTextParser
DelimitedTextParser tracks the current column index inside the current
row that is under parsing. The row could have arbitrary numbers of
fields. The index, 'column_idx_', is defined as int type which could
overflow when there are more than 2^31 fields in the row. This index is
only used to check whether the current column should be materialized. It
doesn't make sense to track the index if it's larger than the number of
columns of the table.
This patch fixes the overflow issue by only bumping 'column_idx_' when
it's smaller than the number of columns of the table.
Tests
- Add e2e test
Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Reviewed-on: http://gerrit.cloudera.org:8080/21559
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> impalad crash -- impala::DelimitedTextParser<true>::ParseFieldLocations
> -----------------------------------------------------------------------
>
> Key: IMPALA-13161
> URL: https://issues.apache.org/jira/browse/IMPALA-13161
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0.0, Impala 4.4.0
> Reporter: nyq
> Assignee: Quanlong Huang
> Priority: Critical
>
> Impala version: 4.0.0
> Problem:
> impalad crash, by operating a text table, which has a 3GB data file that only contains '\x00' char
> Steps:
> python -c 'f=open("impala_0_3gb.data.csv", "wb");tmp="\x00"*1024*1024*3; [f.write(tmp) for i in range(1024)] ;f.close()'
> create table impala_0_3gb (id int)
> hdfs dfs -put impala_0_3gb.data.csv /user/hive/warehouse/impala_0_3gb/
> refresh impala_0_3gb
> select count(1) from impala_0_3gb
> Errors:
> Wrote minidump to 1dcf110f-5a2e-49a2-be4eb7a5-4709ed19.dmp
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x000000000181861c, pid=956182, tid=0x00007fc6b340e700
> #
> # JRE version: OpenJDK Runtime Environment (8.0) (build 1.8.0)
> # Java VM: OpenJDK 64-Bit Server VM
> # Problematic frame:
> # C [impalad+0x141861c] impala::DelimitedTextParser<true>::ParseFieldLocations(int, long, char**, char**, impala::FieldLocation*, int*, int*, char**)+0x7cc
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid956182.log
> #
> #
> C [impalad+0x141861c] impala::DelimitedTextParser<true>::ParseFieldLocations(int, long, char**, char**, impala::FieldLocation*, int*, int*, char**)+0x7cc
> C [impalad+0x136fe11] impala::HdfsTextScanner::ProcessRange(impala::RowBatch*, int*)+0x1a1
> C [impalad+0x137100e] impala::HdfsTextScanner::FinishScanRange(impala::RowBatch*)+0x3be
> C [impalad+0x13721ac] impala::HdfsTextScanner::GetNextInternal(impala::RowBatch*)+0x12c
> C [impalad+0x131cdfc] impala::HdfsScanner::ProcessSplit()+0x19c
> C [impalad+0x1443e17] impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::io::ScanRange*, long*)+0x7e7
> C [impalad+0x1447001] impala::HdfsScanNode::ScannerThread(bool, long)+0x541
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org