You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Daniel Becker (JIRA)" <ji...@apache.org> on 2019/08/06 12:24:00 UTC
[jira] [Commented] (IMPALA-8833) Check failed: bit_width <=
sizeof(T) * 8 (40 vs. 32) in BatchedBitReader::UnpackBatch()
[ https://issues.apache.org/jira/browse/IMPALA-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900972#comment-16900972 ]
Daniel Becker commented on IMPALA-8833:
---------------------------------------
The problem seems to be here:
[https://github.com/apache/impala/blob/bbe064ec194aff4ecf1e794bd4071df4ea4be166/be/src/util/dict-encoding.h#L256]
The dict decoder checks the bit width. So far the maximum bit width for bit packing was 32 and bit widths like 40 were caught here. Now that the maximum is 64, it is not caught but when the decoder tries to write it to a uint32_t it DCHECKs.
> Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32) in BatchedBitReader::UnpackBatch()
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-8833
> URL: https://issues.apache.org/jira/browse/IMPALA-8833
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: Tim Armstrong
> Assignee: Daniel Becker
> Priority: Blocker
> Labels: broken-build, crash, flaky
>
> {noformat}
> F0801 21:24:10.571285 15993 bit-stream-utils.inline.h:126] d04ba69d5da8ffd1:a9045b8200000001] Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32)
> *** Check failure stack trace: ***
> @ 0x52f63ac google::LogMessage::Fail()
> @ 0x52f7c51 google::LogMessage::SendToLog()
> @ 0x52f5d86 google::LogMessage::Flush()
> @ 0x52f934d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2b265b5 impala::BatchedBitReader::UnpackBatch<>()
> @ 0x2ae8623 impala::RleBatchDecoder<>::FillLiteralBuffer()
> @ 0x2b2cadb impala::RleBatchDecoder<>::DecodeLiteralValues<>()
> @ 0x2b27bfb impala::DictDecoder<>::DecodeNextValue()
> @ 0x2b16fed impala::ScalarColumnReader<>::ReadSlotsNoConversion()
> @ 0x2ac7252 impala::ScalarColumnReader<>::ReadSlots()
> @ 0x2a76cef impala::ScalarColumnReader<>::MaterializeValueBatchRepeatedDefLevel()
> @ 0x2a58faa impala::ScalarColumnReader<>::ReadValueBatch<>()
> @ 0x2a20e8e impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch()
> @ 0x29b189c impala::HdfsParquetScanner::AssembleRows()
> @ 0x29ac6de impala::HdfsParquetScanner::GetNextInternal()
> @ 0x29aa656 impala::HdfsParquetScanner::ProcessSplit()
> @ 0x249172d impala::HdfsScanNode::ProcessSplit()
> @ 0x2490902 impala::HdfsScanNode::ScannerThread()
> @ 0x248fc8b _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x2492253
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6915
> Log lines around the failure:
> {noformat}
> [gw5] PASSED query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw1] PASSED query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q7[protocol: beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol: beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw1] PASSED query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol: beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q10a[protocol: beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw10] PASSED query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw6] FAILED query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_decimal_ops[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}]
> [gw2] FAILED query_test/test_scanners.py::TestParquet::test_type_widening[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': 'HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:FAIL@0.5', 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw4] FAILED query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw12] FAILED query_test/test_scanners.py::TestParquet::test_huge_num_rows[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw8] ERROR query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw13] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol: beeswax | exec_option: {'debug_action': None, 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> [gw5] ERROR query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw11] FAILED query_test/test_scanners.py::TestParquet::test_corrupt_footer_len_decr[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw10] ERROR query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: text/none]
> [gw9] FAILED query_test/test_insert_parquet.py::TestInsertParquetQueries::test_insert_parquet[compression_codec: snappy | protocol: beeswax | exec_option: {'sync_ddl': 1, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none | file_size: 0-unique_database0]
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_width_bucket[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}]
> {noformat}
> I'm guessing this is the scanner fuzzing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org