You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mgrazianoc (via GitHub)" <gi...@apache.org> on 2023/09/25 20:44:52 UTC

[GitHub] [arrow] mgrazianoc opened a new issue, #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

mgrazianoc opened a new issue, #37726:
URL: https://github.com/apache/arrow/issues/37726

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   **Bug**
   
   During `FlightClient.doGet` call, `ArrowReader` process an empty `dataBody` on a valid flight data, raising `Fatal error: UnsafeRawBufferPointer.load with negative offset`. However, `dataHeader` is present.
   
   ---
   
   **Details**
   
   A valid flight call tested both in `python` and `rust` was also tested in the not yet released Swift package. The ticket is a simple `TEST` raw byte, in order for the server to start a simple stream of RecordBatch.
   
   ---
   
   **Error Messages**
   
   - Fatal error: UnsafeRawBufferPointer.load with negative offset
   
   ---
   
   **System details**
   
   - Using `Xcode Version 15.0 beta 8 (15A5229m)`
   - Building for **Apple Vision Pro Simulator**.
   - Swift-Driver version: 1.87.1 Apple Swift version 5.9 (swiftlang-5.9.0.128.106 clang-1500.0.40.1)
   
   ---
   
   **Screenshots**
   
   ![s1](https://github.com/apache/arrow/assets/22849928/7dd89957-4291-4f98-a369-1c9ba202efc3)
   ![s2](https://github.com/apache/arrow/assets/22849928/e6a60d57-99f0-4276-a68a-5eb9eb21d650)
   
   ### Component(s)
   
   Swift


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] abandy commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "abandy (via GitHub)" <gi...@apache.org>.
abandy commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1721175418

   @mgrazianoc Can you share the test case that hits this error?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] abandy commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "abandy (via GitHub)" <gi...@apache.org>.
abandy commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1721719949

   Hi @mgrazianoc, it looks like a test for calling doGet with a flightDataClosure is missing.  I will submit a PR to add the missing test case.  In the meantime, here is the test code for getting the data from the flightDataClosure into a RecordBatch.
   
   `    
   func doGetTestFlightData() async throws {
           let ticket = FlightTicket("flight_ticket test".data(using: .utf8)!)
           var num_call = 0
           try await client?.doGet(ticket, flightDataClosure: { flightData in
               let reader = ArrowReader();
               let result = reader.fromStream(flightData.dataBody)
               switch result {
               case .success(let rb):
                   XCTAssertEqual(rb.schema?.fields.count, 3)
                   XCTAssertEqual(rb.batches[0].length, 4)
                   num_call += 1
               case .failure(let error):
                   throw error
               }
           })
           
           XCTAssertEqual(num_call, 1)
       }
   `
   Once you have the record batches you should be able to call ArrowTable.from(recordBatchs: [RecordBatch]) to build the ArrowTable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mgrazianoc commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "mgrazianoc (via GitHub)" <gi...@apache.org>.
mgrazianoc commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1721595255

   Hi, I think the issue was due to the flat buffer data. I'm learning about the weeds, so here's what we've tried that worked:
   
   - Instead of using `readerResultClosure` which expects fileData format (maybe from IPC File Format extension, I'm not sure), we call `flightDataClosure`, which read the bytes successfully.
   
   Now, I'm trying to figure it out how to convert from an array of flight data into an arrow table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mgrazianoc commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "mgrazianoc (via GitHub)" <gi...@apache.org>.
mgrazianoc commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1736020301

   I've opened the other error in the #37884 issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mgrazianoc commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "mgrazianoc (via GitHub)" <gi...@apache.org>.
mgrazianoc commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1721783200

   Now I'm getting back the `Fatal error: UnsafeRawBufferPointer.load with negative offset`, raised inside the ArrowReader, with the code:
   
   ```swift
   func doGet() async throws {
       let ticket = FlightTicket("TEST".data(using: .utf8)!)
       var recordBatches = [RecordBatch]()
       try await client?.doGet(ticket, flightDataClosure: { flightData in
           let reader = ArrowReader();
           let result = reader.fromStream(flightData.dataBody)
           switch result {
           case .success(let readerResult):
               for rb in readerResult.batches {
                   recordBatches.append(rb)
               }
           case .failure(let error):
               throw error
           }
       })
       let table = ArrowTable.from(recordBatches: recordBatches)
       print(table)
   }
   ```
   Arrow table being sent (it raises with more complex ones), through https://arrow.apache.org/docs/python/generated/pyarrow.flight.RecordBatchStream.html:
   ```txt
   pyarrow.Table
   TEST_STRING: string
   ----
   TEST_STRING: [["testString"]]
   ```
   
   ![s3](https://github.com/apache/arrow/assets/22849928/120cc165-6845-4961-8d36-665fdff86048)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1732420843

   Could you open a new issue for the another bug?
   If we have one issue for one problem, we can fix one problem by one pull request. It's easy to review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1720434831

   @abandy Could you take a look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data. [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data. 
URL: https://github.com/apache/arrow/issues/37726


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mgrazianoc closed issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "mgrazianoc (via GitHub)" <gi...@apache.org>.
mgrazianoc closed issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data. 
URL: https://github.com/apache/arrow/issues/37726


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1734437251

   I reopen this because #37764 isn't merged yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] abandy commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "abandy (via GitHub)" <gi...@apache.org>.
abandy commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1721173124

   Yes, will do.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] abandy commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "abandy (via GitHub)" <gi...@apache.org>.
abandy commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1722589618

   @mgrazianoc Seems the behavior of the swift impl and existing flight implementations were different.  I have added a PR to update the swift behavior to match existing implementations.  Please test against it when you get a chance: https://github.com/apache/arrow/pull/37764.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mgrazianoc commented on issue #37726: [Swift][FlightSQL] Processing valid RecordBatch leads to invalid offset from body's data.

Posted by "mgrazianoc (via GitHub)" <gi...@apache.org>.
mgrazianoc commented on issue #37726:
URL: https://github.com/apache/arrow/issues/37726#issuecomment-1731477201

   I think just before we can close the issue, after the great work made by @abandy, there's one more bug relating to the arrow-flight `doGet`.
   
   Sending `utf-8` string seems to be crashing in some scenarios. I'm running the [rust implementation](https://github.com/apache/arrow-rs), which according with the documentation [is considered stable](https://github.com/apache/arrow-rs).
   
   Sending the `testString` will result in a `Swift/UnsafeRawPointer.swift:1203: Fatal error: load from misaligned raw pointer`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org