[jira] [Updated] (ARROW-6425) ValidateArray fail for slice of list array

["Yan Zhou (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yan Zhou updated ARROW-6425:
----------------------------
    Description: 
The ValidateArray function fails for slice of list array (and MapArray I believe though haven't tested it yet). The validation will only pass for an unsliced array or the last slice.

 

For example,

 

 
{code:java}
auto value_builder = std::make_shared<::arrow::DoubleBuilder>();
    ::arrow::ListBuilder builder(
        ::arrow::default_memory_pool(), value_builder);
    auto status = builder.Append(true);
    status = value_builder->Append(1);
    status = builder.Append(true);
    status = value_builder->Append(2);
    std::shared_ptr<::arrow::Array> list;
    status = builder.Finish(&list);
    std::cout << ::arrow::ValidateArray(*list).ok() << std::endl;
    std::cout << ::arrow::ValidateArray(*list->Slice(0, 1)).ok() << std::endl;
    std::cout << ::arrow::ValidateArray(*list->Slice(1, 1)).ok() << std::endl;
 {code}
 

This will print 1, 0, 1.

 

The problem lies in array.cc around line 1167

 
{code:java}
    const int32_t last_offset = array.value_offset(array.length());
    if (array.values()->length() != last_offset) {
      return Status::Invalid("Final offset invariant not equal to values length: ",
                             last_offset, "!=", array.values()->length());
    }
{code}
Here values() does not account for the array offset and thus its length is the whole array while value_offset and array.length() will account for that offset.

  was:
The ValidateArray function fails for slice of list array (and MapArray I believe though haven't tested it yet). The validation will only pass for an unsliced array or the last slice.

 

For example,

 

 
{code:java}
auto value_builder = std::make_shared<::arrow::DoubleBuilder>(); ::arrow::ListBuilder builder( ::arrow::default_memory_pool(), value_builder); auto status = builder.Append(true); status = value_builder->Append(1); status = builder.Append(true); status = value_builder->Append(2); std::shared_ptr<::arrow::Array> list; status = builder.Finish(&list); std::cout << ::arrow::ValidateArray(*list).ok() << std::endl; std::cout << ::arrow::ValidateArray(*list->Slice(0, 1)).ok() << std::endl; std::cout << ::arrow::ValidateArray(*list->Slice(1, 1)).ok() << std::endl;
{code}
 

 

This will print 1, 0, 1.

 

The problem lies in array.cc around line 1167

 
{code:java}
    const int32_t last_offset = array.value_offset(array.length());
    if (array.values()->length() != last_offset) {
      return Status::Invalid("Final offset invariant not equal to values length: ",
                             last_offset, "!=", array.values()->length());
    }
{code}
Here values() does not account for the array offset and thus its length is the whole array while value_offset and array.length() will account for that offset.


> ValidateArray fail for slice of list array 
> -------------------------------------------
>
>                 Key: ARROW-6425
>                 URL: https://issues.apache.org/jira/browse/ARROW-6425
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++
>    Affects Versions: 0.14.1
>            Reporter: Yan Zhou
>            Priority: Major
>
> The ValidateArray function fails for slice of list array (and MapArray I believe though haven't tested it yet). The validation will only pass for an unsliced array or the last slice.
>  
> For example,
>  
>  
> {code:java}
> auto value_builder = std::make_shared<::arrow::DoubleBuilder>();
>     ::arrow::ListBuilder builder(
>         ::arrow::default_memory_pool(), value_builder);
>     auto status = builder.Append(true);
>     status = value_builder->Append(1);
>     status = builder.Append(true);
>     status = value_builder->Append(2);
>     std::shared_ptr<::arrow::Array> list;
>     status = builder.Finish(&list);
>     std::cout << ::arrow::ValidateArray(*list).ok() << std::endl;
>     std::cout << ::arrow::ValidateArray(*list->Slice(0, 1)).ok() << std::endl;
>     std::cout << ::arrow::ValidateArray(*list->Slice(1, 1)).ok() << std::endl;
>  {code}
>  
> This will print 1, 0, 1.
>  
> The problem lies in array.cc around line 1167
>  
> {code:java}
>     const int32_t last_offset = array.value_offset(array.length());
>     if (array.values()->length() != last_offset) {
>       return Status::Invalid("Final offset invariant not equal to values length: ",
>                              last_offset, "!=", array.values()->length());
>     }
> {code}
> Here values() does not account for the array offset and thus its length is the whole array while value_offset and array.length() will account for that offset.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)