[GitHub] [arrow] tachyonwill commented on a change in pull request #11984: PARQUET-2109: Check if Parquet page has too few values

[GitBox <gi...@apache.org>]

tachyonwill commented on a change in pull request #11984:
URL: https://github.com/apache/arrow/pull/11984#discussion_r775573699



##########
File path: cpp/src/parquet/column_reader.cc
##########
@@ -940,7 +940,7 @@ int64_t TypedColumnReaderImpl<DType>::ReadBatchWithDictionary(
     int64_t* indices_read, const T** dict, int32_t* dict_len) {
   bool has_dict_output = dict != nullptr && dict_len != nullptr;
   // Similar logic as ReadValues to get pages.
-  if (!HasNext()) {
+  if (batch_size == 0 || !HasNext()) {

Review comment:
       We should probably support this. I think the more important question is should we throw an exception for bad pages or just skip over the phantom values. Parquet-mr seems to do the latter but does other validation on the reported value counts. It would also make the batch size 0 case easier.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org