[GitHub] [arrow] hinxx commented on issue #34262: [C++] Support for union type in ORC writer

["hinxx (via GitHub)" <gi...@apache.org>]

hinxx commented on issue #34262:
URL: https://github.com/apache/arrow/issues/34262#issuecomment-1439573572

   Here is the script:
   ```
   import pandas as pd
   import pyarrow as pa
   from pyarrow import orc
   import json
   
   
   names = pd.Series(['row1', 'row2', 'row3', 'row4', 'row5', 'row6'], dtype='string', name='name')
   timestamps = pd.Series([pd.Timestamp('2022-07-15 12:40:13.439549952'), pd.Timestamp('2022-07-15 12:40:13.439546880'), pd.Timestamp('2023-02-08 09:13:32.287076352'), pd.Timestamp('2023-02-08 09:13:32.587076352'), pd.Timestamp('2022-07-07 14:23:10.092787968'), pd.Timestamp('2022-07-15 12:40:13.839546624')], dtype='datetime64[ns]', name='timestamp')
   tags = pd.Series([1, 1, 0, 0, 2, 1], dtype='uint8', name='tag')
   offsets = pd.Series([0, 1, 0, 1, 0, 2], dtype='uint32', name='offset')
   integers = pd.Series([5, 53], dtype='int64', name='integer')
   floats = pd.Series([0.011021, -32580.0, -33580.0], dtype='float64', name='float')
   strings = pd.Series(['3.10.0'], dtype='string', name='string')
   
   union_schema = pa.union([
       pa.field('int64', pa.int64()),
       pa.field('float64', pa.float64()),
       pa.field('string', pa.string())
       ], 'dense')
   schema = pa.schema([
       ('name', pa.string()),
       ('timestamp', pa.timestamp('ns')),
       ('value', union_schema)
       ])
   union = pa.UnionArray.from_dense(
       pa.array(tags, type='int8'),
       pa.array(offsets, type='int32'),
       [   pa.Array.from_pandas(integers),
           pa.Array.from_pandas(floats),
           pa.Array.from_pandas(strings)
       ],
       ['int64', 'float64', 'string']
       )
   table = pa.Table.from_arrays([
       pa.Array.from_pandas(names),
       pa.Array.from_pandas(timestamps),
       union
       ], schema=schema)
   
   print('table', table)
   writer = orc.ORCWriter('union1.orc', dictionary_key_size_threshold=1)
   writer.write(table)
   ```
   
   And this the output I get:
   ```
   $ python union1.py 
   table pyarrow.Table
   name: string
   timestamp: timestamp[ns]
   value: dense_union<int64: int64=0, float64: double=1, string: string=2>
     child 0, int64: int64
     child 1, float64: double
     child 2, string: string
   ----
   name: [["row1","row2","row3","row4","row5","row6"]]
   timestamp: [[2022-07-15 12:40:13.439549952,2022-07-15 12:40:13.439546880,2023-02-08 09:13:32.287076352,2023-02-08 09:13:32.587076352,2022-07-07 14:23:10.092787968,2022-07-15 12:40:13.839546624]]
   value: [  -- is_valid: all not null  -- type_ids: [1,1,0,0,2,1]  -- value_offsets: [0,1,0,1,0,2]
     -- child 0 type: int64
   [5,53]
     -- child 1 type: double
   [0.011021,-32580,-33580]
     -- child 2 type: string
   ["3.10.0"]]
   Traceback (most recent call last):
     File "union1.py", line 42, in <module>
       writer.write(table)
     File "/data/data/Code/orc/python/venv/lib/python3.8/site-packages/pyarrow/orc.py", line 289, in write
       self.writer.write(table)
     File "pyarrow/_orc.pyx", line 443, in pyarrow._orc.ORCWriter.write
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: Unknown or unsupported Arrow type: dense_union<int64: int64=0, float64: double=1, string: string=2>
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org