You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "setop (via GitHub)" <gi...@apache.org> on 2024/05/27 07:36:33 UTC

Re: [I] `concat_batches` errors with "schema mismatch" error when only metadata differs [arrow-rs]

setop commented on issue #4799:
URL: https://github.com/apache/arrow-rs/issues/4799#issuecomment-2132836977

   Same issue with version 51.0.0
   
   From the one big CSV cut into two, I created the first partquet file with parquet-cpp-arrow
   
   ```
   Metadata for file: E2021.parquet
   
   version: 1
   num of rows: 4283692
   created by: parquet-cpp-arrow version 5.0.0
   metadata:
     ARROW:schema: /////4ABAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAYAAAAUAQAA0AAAAJwAAABoAAAAOAAAAAQAAAAU////AAABAxAAAAAcAAAABAAAAAAAAAAFAAAAcHJpY2UABgAIAAYABgAAAAAAAgBE////AAABAhAAAAAUAAAABAAAAAAAAAADAAAAZGF5ADD///8AAAABQAAAAHD///8AAAECEAAAABgAAAAEAAAAAAAAAAUAAABtb250aAAAAGD///8AAAABQAAAAKD///8AAAECEAAAABgAAAAEAAAAAAAAAAQAAAB5ZWFyAAAAAJD///8AAAABQAAAAND///8AAAECEAAAABgAAAAEAAAAAAAAAAQAAABmdWVsAAAAAMD///8AAAABQAAAABAAFAAIAAYABwAMAAAAEAAQAAAAAAABAhAAAAAgAAAABAAAAAAAAAAFAAAAcGR2aWQAAAAIAAwACAAHAAgAAAAAAAABQAAAAAAAAAA=
   message schema {
     OPTIONAL INT64 pdvid;
     OPTIONAL INT64 fuel;
     OPTIONAL INT64 year;
     OPTIONAL INT64 month;
     OPTIONAL INT64 day;
     OPTIONAL DOUBLE price;
   }
   ```
   
   Then, using the same schema (`message schema { ... }` in a file), I created the second half with parquet-rs:
   
   ```
   Metadata for file: E2022.parquet
   
   version: 1
   num of rows: 5044596
   created by: parquet-rs version 51.0.0
   metadata:
     ARROW:schema: /////5ABAAAQAAAAAAAKAAwACgAJAAQACgAAABAAAAAAAQQACAAIAAAABAAIAAAABAAAAAYAAAAoAQAA5AAAALAAAAB8AAAATAAAABQAAAAQABYAEAAOAA8ABAAAAAgAEAAAABgAAAAcAAAAAAABAxgAAAAAAAYACAAGAAYAAAAAAAIAAAAAAAUAAABwcmljZQAAAET///8QAAAAGAAAAAAAAQIUAAAANP///0AAAAAAAAABAAAAAAMAAABkYXkAcP///xAAAAAYAAAAAAABAhQAAABg////QAAAAAAAAAEAAAAABQAAAG1vbnRoAAAAoP///xAAAAAYAAAAAAABAhQAAACQ////QAAAAAAAAAEAAAAABAAAAHllYXIAAAAA0P///xAAAAAYAAAAAAABAhQAAADA////QAAAAAAAAAEAAAAABAAAAGZ1ZWwAAAAAEAAUABAADgAPAAQAAAAIABAAAAAYAAAAIAAAAAAAAQIcAAAACAAMAAQACwAIAAAAQAAAAAAAAAEAAAAABQAAAHBkdmlkAAAA
   message arrow_schema {
     OPTIONAL INT64 pdvid;
     OPTIONAL INT64 fuel;
     OPTIONAL INT64 year;
     OPTIONAL INT64 month;
     OPTIONAL INT64 day;
     OPTIONAL DOUBLE price;
   }
   ```
   
   When I try to concat them, I get `Error: General("inputs must have the same schema ...`
   
   The only diff is the naming of the schema, `schema` vs `arrow_schema`.
   
   ```diff
   --- a.txt       2024-05-27 09:32:48.409232203 +0200
   +++ b.txt       2024-05-27 09:32:55.073572763 +0200
   @@ -1,6 +1,6 @@
    GroupType {
        basic_info: BasicTypeInfo {
   -        name: \"schema\",
   +        name: \"arrow_schema\",
            repetition: None,
            converted_type: NONE,
            logical_type: None,
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org