Re: [I] Streaming LIST data over ADBC [arrow-adbc]

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot commented on issue #2066:
URL: https://github.com/apache/arrow-adbc/issues/2066#issuecomment-2274793528

   I find installing the development ADBC Python packages rather difficult; however, I do have a build set up and ran your example (thanks!) against the postgres driver at main and it seems to fail! That error is coming from a previous implementation of ingestion that doesn't use COPY, and the COPY writer is where the feature was added in the PR you linked. It also may be that a recent PR I did resulted in the COPY path not getting picked up.
   
   ```python
   import os
   
   import pandas
   import pyarrow
   
   import adbc_driver_postgresql.dbapi
   
   # From the apache-adbc checkout, run docker compose up postgres-test
   conn = adbc_driver_postgresql.dbapi.connect("postgresql://localhost:5432/postgres?user=postgres&password=password")
   with conn.cursor() as cur:
       cur.execute("DROP TABLE IF EXISTS example")
   
   dfs = [
       pandas.DataFrame({"A": [[1, 2, 3], [1, 2], [1, 2, 3, 4]], "B": [10., 20., 30.]}),
       pandas.DataFrame({"A": [[4, 5, 6], [4, 5, 6, 7], [4]], "B": [40., 50., 60.]}),
   ]
   
   # Create a table and append to it.
   with conn.cursor() as cur:
       for df in dfs:
           table = pyarrow.Table.from_pandas(df)
           cur.adbc_ingest("example", table, mode="create_append")
           cur.execute("SELECT * FROM example")
           print(cur.fetchall())
   conn.close()
   #> Traceback
   #> ...
   #> NotSupportedError: NOT_IMPLEMENTED: [libpq] Field #1 ('A') has unsupported type for ingestion list
   ```
   
   Reproducer in R:
   
   ```r
   ``` r
   library(adbcdrivermanager)
   #> Warning: package 'adbcdrivermanager' was built under R version 4.3.3
   
   con <- adbc_database_init(
     adbcpostgresql::adbcpostgresql(),
     uri = "postgresql://localhost:5432/postgres?user=postgres&password=password"
   ) |> 
     adbc_connection_init()
   
   df <- tibble::tibble(
     A = vctrs::list_of(1:3, 4:5, 6:10),
     B = c(10.0, 20.0, 30.0)
   )
   
   con |> 
     execute_adbc("DROP TABLE IF EXISTS table_with_list")
   
   df |> 
     write_adbc(con, "table_with_list")
   #> Error in adbc_statement_execute_query(stmt): NOT_IMPLEMENTED: [libpq] Field #1 ('A') has unsupported type for ingestion list
   ```
   
   <sup>Created on 2024-08-07 with [reprex v2.1.0](https://reprex.tidyverse.org)</sup>
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org