You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/06 10:54:09 UTC
[GitHub] [arrow] lilixiang opened a new pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
lilixiang opened a new pull request #10893:
URL: https://github.com/apache/arrow/pull/10893
try to fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r685172981
##########
File path: python/pyarrow/_flight.pyx
##########
@@ -949,7 +949,43 @@ cdef class MetadataRecordBatchWriter(_CRecordBatchWriter):
# individual batches to have control.
with nogil:
check_flight_status(
- self.writer.get().WriteRecordBatch(deref(batch.batch)))
+ self._writer().WriteRecordBatch(deref(batch.batch)))
+
+ def write_table(self, Table table, max_chunksize=None, **kwargs):
+ """
+ Write Table to stream in (contiguous) RecordBatch objects.
+
+ Parameters
+ ----------
+ table : Table
+ max_chunksize : int, default None
+ Maximum size for RecordBatch chunks. Individual chunks may be
+ smaller depending on the chunk layout of individual columns.
+ """
+ cdef:
+ # max_chunksize must be > 0 to have any impact
+ int64_t c_max_chunksize = -1
+
+ if 'chunksize' in kwargs:
+ max_chunksize = kwargs['chunksize']
+ msg = ('The parameter chunksize is deprecated for the write_table '
+ 'methods as of 0.15, please use parameter '
+ 'max_chunksize instead')
+ warnings.warn(msg, FutureWarning)
+
+ if max_chunksize is not None:
+ c_max_chunksize = max_chunksize
+
+ with nogil:
+ check_flight_status(
+ self._writer().WriteTable(table.table[0], c_max_chunksize))
Review comment:
This is consistent with what's in ipc.pxi so it's OK.
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1573,33 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ data = [pa.array([-10, -5, 0, 5, 10])]
+ table = pa.Table.from_arrays(data, names=['a'])
+
+ exceptions = {
+ 'internal': flight.FlightInternalError,
+ 'timedout':flight.FlightTimedOutError,
+ 'cancel':flight.FlightCancelledError,
+ 'unauthenticated':flight.FlightUnauthenticatedError,
+ 'unauthorized':flight.FlightUnauthorizedError,
+ 'after_read_internal':flight.FlightInternalError,
+ 'after_read_timedout':flight.FlightTimedOutError,
+ 'after_read_cancel':flight.FlightCancelledError,
+ 'after_read_unauthenticated':flight.FlightUnauthenticatedError,
+ 'after_read_unauthorized':flight.FlightUnauthorizedError,
+
+ }
+
+ for command, exception in exceptions.items():
+
+ with pytest.raises(exception, match=".*foo.*"):
Review comment:
I suggested also testing just `writer.close()` in a separate block by itself to ensure the error propagates even if you don't write data - does that also work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r686090588
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1573,33 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ data = [pa.array([-10, -5, 0, 5, 10])]
+ table = pa.Table.from_arrays(data, names=['a'])
+
+ exceptions = {
+ 'internal': flight.FlightInternalError,
+ 'timedout':flight.FlightTimedOutError,
+ 'cancel':flight.FlightCancelledError,
+ 'unauthenticated':flight.FlightUnauthenticatedError,
+ 'unauthorized':flight.FlightUnauthorizedError,
+ 'after_read_internal':flight.FlightInternalError,
+ 'after_read_timedout':flight.FlightTimedOutError,
+ 'after_read_cancel':flight.FlightCancelledError,
+ 'after_read_unauthenticated':flight.FlightUnauthenticatedError,
+ 'after_read_unauthorized':flight.FlightUnauthorizedError,
+
+ }
+
+ for command, exception in exceptions.items():
+
+ with pytest.raises(exception, match=".*foo.*"):
Review comment:
I have done this. Thanks a lot. 😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r684601302
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1578,48 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
Review comment:
change it with a dict, and it can precisely control of exception occurres, but i don't know if not call write_table before close will cause other exception, so I test it with same code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r684594487
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1578,48 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
+ writer, reader = client.do_put(flight.FlightDescriptor.for_command("internal"))
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r684602103
##########
File path: python/pyarrow/_flight.pyx
##########
@@ -949,7 +949,43 @@ cdef class MetadataRecordBatchWriter(_CRecordBatchWriter):
# individual batches to have control.
with nogil:
check_flight_status(
- self.writer.get().WriteRecordBatch(deref(batch.batch)))
+ self._writer().WriteRecordBatch(deref(batch.batch)))
+
+ def write_table(self, Table table, max_chunksize=None, **kwargs):
+ """
+ Write Table to stream in (contiguous) RecordBatch objects.
+
+ Parameters
+ ----------
+ table : Table
+ max_chunksize : int, default None
+ Maximum size for RecordBatch chunks. Individual chunks may be
+ smaller depending on the chunk layout of individual columns.
+ """
+ cdef:
+ # max_chunksize must be > 0 to have any impact
+ int64_t c_max_chunksize = -1
+
+ if 'chunksize' in kwargs:
+ max_chunksize = kwargs['chunksize']
+ msg = ('The parameter chunksize is deprecated for the write_table '
+ 'methods as of 0.15, please use parameter '
+ 'max_chunksize instead')
+ warnings.warn(msg, FutureWarning)
+
+ if max_chunksize is not None:
+ c_max_chunksize = max_chunksize
+
+ with nogil:
+ check_flight_status(
+ self._writer().WriteTable(table.table[0], c_max_chunksize))
Review comment:
don't much know about cython, and should I use
```python
self._writer().WriteTable(deref(table.table)), c_max_chunksize)
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r686058418
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1573,33 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ data = [pa.array([-10, -5, 0, 5, 10])]
+ table = pa.Table.from_arrays(data, names=['a'])
+
+ exceptions = {
+ 'internal': flight.FlightInternalError,
+ 'timedout':flight.FlightTimedOutError,
+ 'cancel':flight.FlightCancelledError,
+ 'unauthenticated':flight.FlightUnauthenticatedError,
+ 'unauthorized':flight.FlightUnauthorizedError,
+ 'after_read_internal':flight.FlightInternalError,
+ 'after_read_timedout':flight.FlightTimedOutError,
+ 'after_read_cancel':flight.FlightCancelledError,
+ 'after_read_unauthenticated':flight.FlightUnauthenticatedError,
+ 'after_read_unauthorized':flight.FlightUnauthorizedError,
+
+ }
+
+ for command, exception in exceptions.items():
+
+ with pytest.raises(exception, match=".*foo.*"):
Review comment:
Actually I see what you mean now. Yes, consolidating the two cases should work if you'd like to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang removed a comment on pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang removed a comment on pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#issuecomment-894614717
@github-actions autotune
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#issuecomment-894663961
@github-actions autotune
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r684238270
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1578,48 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
+ writer, reader = client.do_put(flight.FlightDescriptor.for_command("internal"))
Review comment:
do_put also needs the schema (so pass table.schema as the second argument)
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1578,48 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
Review comment:
Maybe we can put these errors in a list and loop over them, and then test both close() and write_table() separately? Something like
```python
exceptions = [flight.FlightInternalError, flight.FlightTimedOutError, ...]
for exception in exception:
with pytest.raises(exception, match=...):
writer, reader = client.do_put(...)
writer.close()
with pytest.raises(exception, match=...):
writer, reader = client.do_put(...)
writer.write_table(table)
writer.close()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r684594524
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1578,48 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
Review comment:
it's a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#issuecomment-894178374
https://issues.apache.org/jira/browse/ARROW-13577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r685997644
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1573,33 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ data = [pa.array([-10, -5, 0, 5, 10])]
+ table = pa.Table.from_arrays(data, names=['a'])
+
+ exceptions = {
+ 'internal': flight.FlightInternalError,
+ 'timedout':flight.FlightTimedOutError,
+ 'cancel':flight.FlightCancelledError,
+ 'unauthenticated':flight.FlightUnauthenticatedError,
+ 'unauthorized':flight.FlightUnauthorizedError,
+ 'after_read_internal':flight.FlightInternalError,
+ 'after_read_timedout':flight.FlightTimedOutError,
+ 'after_read_cancel':flight.FlightCancelledError,
+ 'after_read_unauthenticated':flight.FlightUnauthenticatedError,
+ 'after_read_unauthorized':flight.FlightUnauthorizedError,
+
+ }
+
+ for command, exception in exceptions.items():
+
+ with pytest.raises(exception, match=".*foo.*"):
Review comment:
I think it's fine as-is. I wanted to test that both `close` and `read_all` capture the exception properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on a change in pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on a change in pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#discussion_r685647708
##########
File path: python/pyarrow/tests/test_flight.py
##########
@@ -1545,6 +1573,33 @@ def test_roundtrip_errors():
with pytest.raises(flight.FlightInternalError, match=".*foo.*"):
list(client.list_flights())
+ data = [pa.array([-10, -5, 0, 5, 10])]
+ table = pa.Table.from_arrays(data, names=['a'])
+
+ exceptions = {
+ 'internal': flight.FlightInternalError,
+ 'timedout':flight.FlightTimedOutError,
+ 'cancel':flight.FlightCancelledError,
+ 'unauthenticated':flight.FlightUnauthenticatedError,
+ 'unauthorized':flight.FlightUnauthorizedError,
+ 'after_read_internal':flight.FlightInternalError,
+ 'after_read_timedout':flight.FlightTimedOutError,
+ 'after_read_cancel':flight.FlightCancelledError,
+ 'after_read_unauthenticated':flight.FlightUnauthenticatedError,
+ 'after_read_unauthorized':flight.FlightUnauthorizedError,
+
+ }
+
+ for command, exception in exceptions.items():
+
+ with pytest.raises(exception, match=".*foo.*"):
Review comment:
I see, I have fixed it. But a little problem about the define command, after_read_xx and xx, should i remove after_read? I think if call write_table, the error will raise before read_all, if just call close and don't use after_read_xx cmd, the exception will raise before read. So just write
``` python
class ErrorFlightServer(FlightServerBase):
....
def do_put(self, context, descriptor, reader, writer):
if descriptor.command == b"internal":
raise flight.FlightInternalError("foo")
elif descriptor.command == b"timedout":
raise flight.FlightTimedOutError("foo")
elif descriptor.command == b"cancel":
raise flight.FlightCancelledError("foo")
elif descriptor.command == b"unauthenticated":
raise flight.FlightUnauthenticatedError("foo")
elif descriptor.command == b"unauthorized":
raise flight.FlightUnauthorizedError("foo")
elif descriptor.command == b"protobuf":
err_msg = b'this is an error message'
raise flight.FlightUnauthorizedError("foo", err_msg)
```
without `reader.read_all()` ? And don't care about the server read or not, just raise error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #10893:
URL: https://github.com/apache/arrow/pull/10893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lilixiang commented on pull request #10893: ARROW-13577: [Python][FlightRPC] pyarrow client do_put close method after write_table did not throw flight error
Posted by GitBox <gi...@apache.org>.
lilixiang commented on pull request #10893:
URL: https://github.com/apache/arrow/pull/10893#issuecomment-894614717
@github-actions autotune
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org