You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2017/09/25 13:05:00 UTC
[jira] [Updated] (ARROW-1601) [C++] READ_NEXT_BITSET reads one byte
past the last byte on last iteration
[ https://issues.apache.org/jira/browse/ARROW-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney updated ARROW-1601:
--------------------------------
Summary: [C++] READ_NEXT_BITSET reads one byte past the last byte on last iteration (was: READ_NEXT_BITSET reads one byte past the last byte on last iteration)
> [C++] READ_NEXT_BITSET reads one byte past the last byte on last iteration
> --------------------------------------------------------------------------
>
> Key: ARROW-1601
> URL: https://issues.apache.org/jira/browse/ARROW-1601
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 0.7.0
> Environment: Mac OS X 10.11.6
> Reporter: Rene Sugar
> Assignee: Wes McKinney
> Labels: crash, pull-request-available
> Fix For: 0.7.1
>
> Attachments: 0001-ARROW-1601-increase-size-of-null_bitmap-allocation-i.patch
>
>
> (int) byte_offset_valid_bits = 131072
> (int) bit_offset_valid_bits = 0
> (uint8_t) bitset_valid_bits = '?'
> (int) i = 1048575
> (lldb) print *num_values
> (int64_t) $0 = 1048576
> (lldb) print valid_bits[131072]
> error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory
> (lldb) print valid_bits[131071]
> (uint8_t) $6 = '?'
> 1048576 / 8 = 131072 (number of values in the array)
> Last readable offset is 131071.
> In the last iteration, READ_NEXT_BITSET reads one byte past the last byte of the valid_bits array.
> When this allocation happens at the end of a block, a crash occurs.
> These macros are used in loops in several places in Arrow and Parquet.
> parquet-cpp/src/parquet/arrow/writer.cc:
> Status GenerateLevels(const Array& array, const std::shared_ptr<Field>& field,
> int64_t* values_offset, ::arrow::Type::type* values_type,
> int64_t* num_values, int64_t* num_levels,
> std::shared_ptr<Buffer>* def_levels,
> std::shared_ptr<Buffer>* rep_levels,
> std::shared_ptr<Array>* values_array) {
> [....]
> const uint8_t* valid_bits = array.null_bitmap_data();
> INIT_BITSET(valid_bits, static_cast<int>(array.offset()));
> for (int i = 0; i < array.length(); i++) {
> if (bitset_valid_bits & (1 << bit_offset_valid_bits)) {
> def_levels_ptr[i] = 1;
> } else {
> def_levels_ptr[i] = 0;
> }
> READ_NEXT_BITSET(valid_bits); <-- crashes here on last iteration
> }
> arrow/util/bitutil.h:
> #define INIT_BITSET(valid_bits_vector, valid_bits_index) \
> int byte_offset_##valid_bits_vector = (valid_bits_index) / 8; \
> int bit_offset_##valid_bits_vector = (valid_bits_index) % 8; \
> uint8_t bitset_##valid_bits_vector = valid_bits_vector[byte_offset_##valid_bits_vector];
> #define READ_NEXT_BITSET(valid_bits_vector) \
> bit_offset_##valid_bits_vector++; \
> if (bit_offset_##valid_bits_vector == 8) { \
> bit_offset_##valid_bits_vector = 0; \
> byte_offset_##valid_bits_vector++; \
> bitset_##valid_bits_vector = valid_bits_vector[byte_offset_##valid_bits_vector]; \
> }
> A quick fix is to allocate one more byte for null_bitmap_ in ArrayBuilder::Init and ArrayBuilder::Resize in arrow/cpp/src/arrow/builder.cc.
> The capacity of null_bitmap() is changed by this fix.
> The following tests FAILED:
> 2 - array-test (Failed)
> 14 - feather-test (Failed)
> A more extensive fix would require changing how INIT_BITSET and READ_NEXT_BITSET operate where they are used in Parquet and Arrow.
> To reproduce this problem:
> 1) Download the CSV file.
> Source: https://catalog.data.gov/dataset?res_format=CSV
> State Drug Utilization Data 2016
> https://data.medicaid.gov/api/views/3v6v-qk5s/rows.csv?accessType=DOWNLOAD
> 2) Run FileConvert (see https://github.com/renesugar/FileConvert)
> ./bin/FileConvert -i ./State_Drug_Utilization_Data_2016.csv -o ./State_Drug_Utilization_Data_2016.parquet
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)