You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/01 14:12:35 UTC
[GitHub] [arrow] lzmchina opened a new issue, #14288: pyarrow tranport Tensor type data to java arrow flight server
lzmchina opened a new issue, #14288:
URL: https://github.com/apache/arrow/issues/14288
Issue Description
I have some pyarrow vector data which is transported to java arrow flight server.
In python arrow flight client , my code is like this:
```python
writer,reader=client.do_exchange(descriptor=pa.flight.FlightDescriptor.for_path("streamed.parquet"))
start = time()
writer.begin(schema=schema)
with writer:
while True:
pop=q.get()
if pop is None:
break
# print("send")
batch = pa.record_batch([pa.array(q.get())], schema)
writer.write_batch(batch)
```
in java arrow flight server, my code is like this:
```java
VectorSchemaRoot vectorSchemaRootReceived = null;
FieldVector v = null;
try {
while (reader.next()) {
vectorSchemaRootReceived = reader.getRoot();
// arb = unloader.getRecordBatch();
rows += vectorSchemaRootReceived.getRowCount();
v = vectorSchemaRootReceived.getVector("tensor");
}
}catch (Exception e){
e.printStackTrace();
}
```
I known arrow flight can transport Vector data type, but I want to transport Tensor data type because I want to tansfer it into nd4j's INDArray type.
I have noticed that arrow in java does not have Tensor data type, so how can I realize it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow tranport Tensor type data to java arrow flight server [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #14288:
URL: https://github.com/apache/arrow/issues/14288#issuecomment-4485986929
The java code was moved to https://github.com/apache/arrow-java but unfortunately this issue was missed. If the issue is still relevant I suggest opening it in the java repo. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #14288: pyarrow tranport Tensor type data to java arrow flight server
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #14288:
URL: https://github.com/apache/arrow/issues/14288#issuecomment-1273681054
There's not an IPC format for the Tensor types yet. There is some discussion about providing an extension type for tensors, and then adding minimal Java support. See:
https://issues.apache.org/jira/browse/ARROW-1614
https://issues.apache.org/jira/browse/ARROW-8714
https://issues.apache.org/jira/browse/ARROW-10101
One of the open questions on the Java side was what libraries for tensors would be useful to interoperate with, so it is good to find out about nd4j.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow tranport Tensor type data to java arrow flight server [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #14288: pyarrow tranport Tensor type data to java arrow flight server
URL: https://github.com/apache/arrow/issues/14288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow tranport Tensor type data to java arrow flight server [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #14288: pyarrow tranport Tensor type data to java arrow flight server
URL: https://github.com/apache/arrow/issues/14288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org