You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "chenyuanxing (via GitHub)" <gi...@apache.org> on 2023/02/23 05:33:49 UTC

[GitHub] [arrow] chenyuanxing opened a new issue, #34300: [C++] All fields must be same length when calling Flush

chenyuanxing opened a new issue, #34300:
URL: https://github.com/apache/arrow/issues/34300

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   when we use builder to generate  recordbatch . call the flush(). it returns the error: **All fields must be same length when calling Flush.** 
   and we look at the source code. it checks all children`s length in a struct and ensure they are the same. 
   but Our scenario is. baseStruct in topStruct. and in each baseStruct . it is a table. (each table has diff length.)  in java it is okay to generate recordBatch and Serialize or deSerialize it. 
   but in c++ . we meet this problem. 
   
   We know that the lengths of the fields under a struct should be the same, but if its sub-fields are all structs, We should think that the length of struct is 1 or 0, right? Then inside the subStruct are the respective lengths.
   
   ```
   void MakeStruct(arrow::ArrayBuilder* builder, int size) {
     if (builder == nullptr) {
       std::cout << "nullptr" << std::endl;
     }
     std::cout << builder->num_children() << ":" << builder->type()->id() << std::endl;
     for (int i = 0; i < builder->num_children(); ++i) {
       auto child_builder = builder->child(i);
       if (child_builder->type()->id() == arrow::Type::STRUCT) {
         MakeStruct(child_builder, size);
       } else if (child_builder->type()->id() == arrow::Type::INT64) {
         auto int64_builder =
             arrow::internal::checked_cast<arrow::Int64Builder*>(child_builder);
         if (int64_builder == nullptr) {
           std::cout << "int64_builder nullptr" << std::endl;
         }
         for (int j = 0; j < size; ++j) {
           auto status = int64_builder->Append(5);
         }
       }
     }
     auto struct_builder = arrow::internal::checked_cast<arrow::StructBuilder*>(builder);
     for (int i = 0; i < size; ++i) {
       auto res = struct_builder->Append(true);
     }
   }
   void RunSchema() {
     auto field_user = arrow::field("user", arrow::struct_({
       arrow::field("user_id", arrow::int64()),
       arrow::field("user_age", arrow::int64())
       }));
     //auto field_online = arrow::field("online", arrow::struct_({}));
     auto field_s = arrow::field("shared", arrow::struct_({field_user/*, field_online*/}));
     auto field_item = arrow::field("item", arrow::struct_({
       arrow::field("item_id", arrow::int64()),
       arrow::field("pv", arrow::int64()),
       arrow::field("uv", arrow::int64()) }));
     auto field_i = arrow::field("iterable", arrow::struct_({field_item/*, field_online*/}));
     auto schema = arrow::schema({field_s, field_i});
     arrow::MemoryPool *pool = arrow::default_memory_pool();
     std::unique_ptr<arrow::RecordBatchBuilder> builder;
     auto builder_res = arrow::RecordBatchBuilder::Make(schema, pool);
     if (builder_res.ok()) {
       builder.reset(builder_res.ValueUnsafe().release());
       std::cout << "builder ok" << std::endl;
     } else {
       std::cout << "builder fail" << std::endl;
       return;
     }
   
     auto field0_builder = builder->GetField(0);
     auto data_type0 = field0_builder->type();
     if (data_type0->id() == arrow::Type::STRUCT) {
       MakeStruct(field0_builder, 1);
     }
   
     auto field1_builder = builder->GetField(1);
     auto data_type1 = field1_builder->type();
     if (data_type1->id() == arrow::Type::STRUCT) {
       MakeStruct(field1_builder, 10);
     }
     auto result =  builder->Flush();
     if (result.ok()) {
       auto record = result.ValueUnsafe();
       std::cout << record->ToString() << std::endl;
     } else {
       std::cout << "builder flush fail " << result.status().ToString() << std::endl;
     }
   
     std::cout << schema->ToString() << std::endl;
     std::cout << schema->num_fields() << std::endl;
   }
   ```
   
   ### Component(s)
   
   C++, Java


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] js8544 commented on issue #34300: [C++] Struct and its substruct length problem

Posted by "js8544 (via GitHub)" <gi...@apache.org>.

js8544 commented on issue #34300:
URL: https://github.com/apache/arrow/issues/34300#issuecomment-1441619232

   > when we use builder to generate recordbatch . call the flush(). it returns the error: **All fields must be same length when calling Flush.**
   
   That's what you were original trying to solve :), isn't it? If they are of the same length, you can simply put them in a RecordBatch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org