You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "amoeba (via GitHub)" <gi...@apache.org> on 2024/12/17 21:35:38 UTC

[I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

amoeba opened a new issue, #45056:
URL: https://github.com/apache/arrow/issues/45056

   ### Describe the enhancement requested
   
   As of writing Flight C++ currently only supports dictionary replacement in DoPut calls and not in DoGet or (partially) DoExchange.
   
   Sub-issues:
   
   - [ ] DoGet: https://github.com/apache/arrow/issues/45055
   - [ ] DoExchange: https://github.com/apache/arrow/issues/26727
   
   ### Component(s)
   
   C++, FlightRPC


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45056:
URL: https://github.com/apache/arrow/issues/45056#issuecomment-3200174136

   Is `DoExchange` also handled by https://github.com/apache/arrow/pull/47115 @raulcd ? If so, can we update this issue and the corresponding sub-issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45056:
URL: https://github.com/apache/arrow/issues/45056#issuecomment-3225037261

   Nice work @raulcd, thanks for taking this on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45056:
URL: https://github.com/apache/arrow/issues/45056#issuecomment-3200199814

   > Is `DoExchange` also handled by [#47115](https://github.com/apache/arrow/pull/47115)
   
   I'm not familiar with `DoExchange` so not sure if it reuses the same code path. I'll have to set up a test and try, I can work on that and will update with findings.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45056:
URL: https://github.com/apache/arrow/issues/45056#issuecomment-3200810933

   `DoExchange` doesn't seem to be using the same code path as the returned table is not allowing for dictionary replacement nor dictionary deltas. Test reproducer:
   ```python
   def test_flight_dictionary_deltas_do_exchange():
       class DeltaFlightServer(ConstantFlightServer):
           def do_exchange(self, context, descriptor, reader, writer):
               if descriptor.command == b'dict_deltas':
                   table = simple_dicts_table()
                   options = pa.ipc.IpcWriteOptions(emit_dictionary_deltas=True)
                   writer.begin(table.schema, options=options)
                   writer.write_table(table)
   
       with DeltaFlightServer() as server, \
               FlightClient(('localhost', server.port)) as client:
   
           descriptor = flight.FlightDescriptor.for_command(b"dict_deltas")
           writer, reader = client.do_exchange(descriptor,
                                               options=flight.FlightCallOptions(
                                                   write_options=pa.ipc.IpcWriteOptions(emit_dictionary_deltas=True)
                                               )
                                              )
   
           received_table = reader.read_all()
           expected_table = simple_dicts_table()
           assert received_table.equals(expected_table)
   ```
   The received table did not update the dictionary (neither deltas / or dictionary replacement)
   ```
   (Pdb) received_table
   pyarrow.Table
   some_dicts: dictionary<values=string, indices=int64, ordered=0>
   ----
   some_dicts: [  -- dictionary:
   ["foo","baz","quux"]  -- indices:
   [1,0,null],  -- dictionary:
   ["foo","baz","quux"]  -- indices:
   [2,1],  -- dictionary:
   ["foo","baz","quux"]  -- indices:
   [0,3]]
   (Pdb) expected_table
   pyarrow.Table
   some_dicts: dictionary<values=string, indices=int64, ordered=0>
   ----
   some_dicts: [  -- dictionary:
   ["foo","baz","quux"]  -- indices:
   [1,0,null],  -- dictionary:
   ["foo","baz","quux"]  -- indices:
   [2,1],  -- dictionary:
   ["foo","baz","quux","new"]  -- indices:
   [0,3]]
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #45056: [C++][Flight] Fully support dictionary replacement in Flight
URL: https://github.com/apache/arrow/issues/45056


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Flight] Fully support dictionary replacement in Flight [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45056:
URL: https://github.com/apache/arrow/issues/45056#issuecomment-3223447323

   Both `DoGet` and `DoExchange` have been implemented, see the sub-issues above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org