You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "no23reason (via GitHub)" <gi...@apache.org> on 2023/03/02 15:59:15 UTC
[GitHub] [arrow] no23reason opened a new issue, #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server
no23reason opened a new issue, #34415:
URL: https://github.com/apache/arrow/issues/34415
### Describe the bug, including details regarding any error messages, version, and platform.
When implementing a custom `do_put` operation using `FlightServerBase`, if the client does not call `done_writing()` on its writer before trying to read metadata sent from the server, both client and server hang and need to be killed.
Is the `done_writing()` required by design? In other words does the process always need to be
1. client writes all of its data
2. client calls `done_writing()`
3. client reads the metadata server sent back
Or is this a bug?
server.py
```python
import pyarrow as pa
import pyarrow.flight as flight
class Server(flight.FlightServerBase):
def __init__(self, location):
super(Server, self).__init__(location=location)
def do_put(self, context, descriptor, reader, writer):
for batch in reader:
print(f"got batch {str(batch.data)}")
# or even just reader.read_all()
print("sending metadata back")
writer.write(pa.py_buffer((42).to_bytes(8, byteorder="big")))
def main():
server = Server("grpc://localhost:16661")
server.serve()
if __name__ == "__main__":
main()
```
client.py
```python
import pandas as pd
import pyarrow as pa
import pyarrow.flight as flight
def main():
print("starting")
client = flight.connect("grpc://localhost:16661")
df = pd.DataFrame(
{
"n_legs": [None, 4, 5, None],
"animals": ["Flamingo", "Horse", None, "Centipede"],
}
)
table = pa.Table.from_pandas(df)
writer, reader = client.do_put(
flight.FlightDescriptor.for_path(b"fun/times"), table.schema
)
writer.write_table(table)
print("done putting")
# if this line is uncommented, everything starts working fine
# writer.done_writing()
meta = reader.read()
print(int.from_bytes(meta))
if __name__ == "__main__":
main()
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34415:
URL: https://github.com/apache/arrow/issues/34415#issuecomment-1452144741
Yeah, unfortunately I don't think the blocking APIs give you a lot of control over that right now :/ You can check for client cancellation, but the server itself can't cancel a read request right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] no23reason commented on issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "no23reason (via GitHub)" <gi...@apache.org>.
no23reason commented on issue #34415:
URL: https://github.com/apache/arrow/issues/34415#issuecomment-1452143011
Thank you for the reply! So to prevent some malicious request when having server like this is to implement some custom timeout mechanism, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34415:
URL: https://github.com/apache/arrow/issues/34415#issuecomment-1452129782
If you don't call done_writing, then the server will never break out of `for batch in reader`. The read and write streams are independent, so just reading on the client side isn't enough to tell the server that the writes are done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34415:
URL: https://github.com/apache/arrow/issues/34415#issuecomment-1452168966
I linked this to #34221 so we don't forget when we eventually get a chance to tackle this. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] no23reason closed issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "no23reason (via GitHub)" <gi...@apache.org>.
no23reason closed issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
URL: https://github.com/apache/arrow/issues/34415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34415:
URL: https://github.com/apache/arrow/issues/34415#issuecomment-1452131176
That said, the APIs here are arguably quite confusing. I have had in mind a larger API revamp someday, probably as part of introducing async APIs (as I don't want to break existing APIs), but that day is a while off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] no23reason commented on issue #34415: [Python]: Trying to read metadata without calling done_writing hangs both client and server thread
Posted by "no23reason (via GitHub)" <gi...@apache.org>.
no23reason commented on issue #34415:
URL: https://github.com/apache/arrow/issues/34415#issuecomment-1452166337
Got it, thank you again for the fast responses :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org