[GitHub] [arrow] lhayhurst opened a new issue #7877: Adding local persistence to examples/flight/server.py

[GitBox <gi...@apache.org>]

lhayhurst opened a new issue #7877:
URL: https://github.com/apache/arrow/issues/7877


   Hi, thank you for providing [the sample client and server for pyarrow flight](https://github.com/apache/arrow/tree/apache-arrow-1.0.0/python/examples/flight) code--super useful for getting a bare-bones client-server setup.
   
   I'm demoing this tech for a colleague, and would love to be able to show off [feather](https://arrow.apache.org/docs/python/feather.html) or parquet as backing persistent local storage for the demo server.  Then, when I drop [the python server](https://github.com/apache/arrow/blob/apache-arrow-1.0.0/python/examples/flight/server.py) during demo, I can "just" restart it and continue to show off the client without having to re-run all the client's `do_puts` :fire: 
   
   From the server code:
   ```
       def do_put(self, context, descriptor, reader, writer):
           key = FlightServer.descriptor_to_key(descriptor)
           print(key)
           self.flights[key] = reader.read_all()
           print(self.flights[key])
   ```
   I see the `writer` isn't used above, an on inspection it is a `pyarrow._flight.FlightMetadataWriter` instance -- can it be used to implement local feather persistence? 
   
   Finally, for the [`do_get`](https://github.com/apache/arrow/blob/b0d623957db820de4f1ff0a5ebd3e888194a48f0/python/examples/flight/server.py#L88) will probably need to change, to lazy load the arrow into memory from disk when the client hits it for the first time. (And probably a change in _init__ to eager load `self.flights`?).
   
   Thanks in advance :pray: 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield closed issue #7877: Adding local persistence to examples/flight/server.py

[GitBox <gi...@apache.org>]

emkornfield closed issue #7877:
URL: https://github.com/apache/arrow/issues/7877


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] lhayhurst commented on issue #7877: Adding local persistence to examples/flight/server.py

[GitBox <gi...@apache.org>]

lhayhurst commented on issue #7877:
URL: https://github.com/apache/arrow/issues/7877#issuecomment-667626890


   Hacking around on this, I was able to implement local storage without too much work. Didn't need the writer (which wouldn't have helped here -- going through the c++ code I think its meant to pass messages back to the client). 
   
   Ended up having to store a pkl of the key along with the arrow file in order to get the 
   
   ```
   self.flights[key] = pyarrow.feather.read_feather(source=feather_file_path)
   ```
   load in __init__ working properly, which felt a little dirty, but ... :shrug: 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on issue #7877: Adding local persistence to examples/flight/server.py

[GitBox <gi...@apache.org>]

emkornfield commented on issue #7877:
URL: https://github.com/apache/arrow/issues/7877#issuecomment-667784599


   @lhayhurst glad you solved the issue.  these types of discussions are best had on user@ or dev@ arrow mailing lists (we don't use github issues to answer questions/track bugs).  If you have further questions feel free to send mail to one of the lists.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org