[GitHub] [arrow] westonpace commented on issue #20385: Reading a StructArray column with an ExtensionType causes segfault

[GitBox <gi...@apache.org>]

westonpace commented on issue #20385:
URL: https://github.com/apache/arrow/issues/20385#issuecomment-1380541325

   So I've found the problem but haven't yet worked out the solution.  The segmentation fault occurs in `reader.cc` in `GetReader`.  The handling for extension type is:
   
   ```
     if (type_id == ::arrow::Type::EXTENSION) {
       auto storage_field = arrow_field->WithType(
           checked_cast<const ExtensionType&>(*arrow_field->type()).storage_type());
       RETURN_NOT_OK(GetReader(field, storage_field, ctx, out));
       *out = std::make_unique<ExtensionReader>(arrow_field, std::move(*out));
       return Status::OK();
     }
   ```
   
   However, if the nested field is not loaded, then the recursive `GetReader` call sets `out` to `nullptr` and this code creates an `ExtensionReader` with a null storage reader.  This later crashes.
   
   The fix is, unfortunately, not as simple as returning null.  The problem is that the Parquet reader is trying to maintain the nested structure.  As you see in your example that works, `column.one` yields a partial struct:
   
   ```
   assert table2.to_pylist() == [
       {"column": {"one": 10}},
       {"column": {"one": 20}},
       {"column": {"one": 30}},
   ]
   ```
   
   However, it is not clear that a partial "extension type" is a valid thing.  For example, imagine your extension type was a 2DPoint with "x" and "y".  What should be returned if the user loads `points.x`?  We can't maintain structure in that case.
   
   I'm a little new to parquet and nested references so I don't know if there is a syntax we can use to ask for the nested columns without structure.  In this case you would get:
   
   ```
   assert table2.to_pylist() == [
     { "one": 10 },
     { "one": 20 },
     { "one": 30 }
   ]
   ```
   
   I will put together a PR that at least returns an invalid status in this case instead of a segmentation fault.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org