You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "culpgrant (via GitHub)" <gi...@apache.org> on 2026/07/20 19:36:33 UTC
[I] Snowflake - Python - Pyarrow table input to adbc_ingest producing single parquet file and single PUT command [arrow-adbc]
culpgrant opened a new issue, #4548:
URL: https://github.com/apache/arrow-adbc/issues/4548
### What happened?
When passing a `pyarrow.Table` to `adbc_ingest` the diver always produces a single Parquet file, a single PUT and a single COPY INTO. Regardless of data size of any connection option
I would expected multiple parquet files with parallel PUTS and overlapping COPY INTO statements. I got just 1 parquet file, 1 PUT and 1 COPY INTO.
Here is a code snipped I made to showcase this:
```import pyarrow as pa
import adbc_driver_snowflake.dbapi as adbc_snowflake
import numpy as np
# Create a 80M row test table
n = 80_000_000
data = pa.table(
{
"ID": pa.array(range(n), type=pa.int64()),
"VALUE": pa.array(np.random.randn(n), type=pa.float64()),
"CATEGORY": pa.array(["A", "B", "C", "D"] * (n // 4), type=pa.string()),
}
)
print(
f"Table: {data.num_rows} rows, {round(data.nbytes / 1024**2)} MB in memory"
)
db_kwargs = {
"adbc.snowflake.sql.account": "**redacted**",
"adbc.snowflake.sql.role": "**redacted**",
"username": "**redacted**",
"adbc.snowflake.sql.warehouse": "**redacted**_SMALL",
"adbc.snowflake.sql.auth_type": "auth_ext_browser",
"adbc.snowflake.sql.client_option.tracing": "debug",
}
print("Connecting...")
with adbc_snowflake.connect(db_kwargs=db_kwargs) as conn:
with conn.cursor() as cursor:
cursor.execute("USE DATABASE **redacted**")
cursor.execute("USE SCHEMA **redacted**")
# Try setting options on the statement
cursor.adbc_statement.set_options(
**{
"adbc.snowflake.statement.ingest_target_file_size": "10485760",
"adbc.snowflake.statement.ingest_writer_concurrency": "4",
}
)
print("Ingesting...")
rows = cursor.adbc_ingest("ADBC_SPLIT_TEST", data, mode="create_append")
print(f"Ingested {rows} rows")
conn.commit()
```
### Stack Trace
I didnt want to upload the full log information but here is some snippets:
```
INFO[0016]file_transfer_agent.go:756 gosnowflake.(*snowflakeFileTransferAgent).upload uploading 1 large files LOG_SESSION_ID=4385957164815143
DEBU[0015]log.go:190 gosnowflake.(*defaultLogger).Debugf upload command initiated - file count: 1, query ID: 01c5d751-0f13-dca0-000f-95042301408b, encryption materials: 1
### How can we reproduce the bug?
Please use the script from above.
### Environment/Setup
`Python 3.12.9`
`adbc-driver-snowflake==1.11.0`
`adbc-driver-manager==1.11.0`
Using uv for package manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Snowflake - Python - Pyarrow table input to adbc_ingest producing single parquet file and single PUT command [arrow-adbc]
Posted by "culpgrant (via GitHub)" <gi...@apache.org>.
culpgrant commented on issue #4548:
URL: https://github.com/apache/arrow-adbc/issues/4548#issuecomment-5026461305
When I use `RecordBatchReader` the adbc `adbc_ingest` works as expeceted. I think `pa.Table` is going into the `ingestRecord` code path not `ingestStream`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Snowflake - Python - Pyarrow table input to adbc_ingest producing single parquet file and single PUT command [arrow-adbc]
Posted by "culpgrant (via GitHub)" <gi...@apache.org>.
culpgrant commented on issue #4548:
URL: https://github.com/apache/arrow-adbc/issues/4548#issuecomment-5026716540
Ahh thank you for the quick response, makes sense. I will go with the above approach. It might be worth adding this callout in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Snowflake - Python - Pyarrow table input to adbc_ingest producing single parquet file and single PUT command [arrow-adbc]
Posted by "culpgrant (via GitHub)" <gi...@apache.org>.
culpgrant closed issue #4548: Snowflake - Python - Pyarrow table input to adbc_ingest producing single parquet file and single PUT command
URL: https://github.com/apache/arrow-adbc/issues/4548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Snowflake - Python - Pyarrow table input to adbc_ingest producing single parquet file and single PUT command [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #4548:
URL: https://github.com/apache/arrow-adbc/issues/4548#issuecomment-5026585708
If the table is a single chunk, then by default pyarrow will convert the table into a single record batch and the driver will write it as is. If you want the driver to write multiple files in parallel and upload them separately, you need to also split the table as well. For example, by using `to_reader`:
```python
n = 80_000_000
data = pa.table(
{
"ID": pa.array(range(n), type=pa.int64()),
"VALUE": pa.array(np.random.randn(n), type=pa.float64()),
"CATEGORY": pa.array(["A", "B", "C", "D"] * (n // 4), type=pa.string()),
}
)
db_kwargs = {
"adbc.snowflake.sql.account": "**redacted**",
"adbc.snowflake.sql.role": "**redacted**",
"username": "**redacted**",
"adbc.snowflake.sql.warehouse": "**redacted**_SMALL",
"adbc.snowflake.sql.auth_type": "auth_ext_browser",
"adbc.snowflake.sql.client_option.tracing": "debug",
}
print("Connecting...")
with adbc_snowflake.connect(db_kwargs=db_kwargs) as conn:
with conn.cursor() as cursor:
cursor.execute("USE DATABASE **redacted**")
cursor.execute("USE SCHEMA **redacted**")
# Try setting options on the statement
cursor.adbc_statement.set_options(
**{
"adbc.snowflake.statement.ingest_target_file_size": "10485760",
"adbc.snowflake.statement.ingest_writer_concurrency": "4",
}
)
print("Ingesting...")
# rows = cursor.adbc_ingest("ADBC_SPLIT_TEST", data, mode="create_append")
max_chunksize = 10_000_000
rows = cursor.adbc_ingest("ADBC_SPLIT_TEST", data.to_reader(max_chunksize), mode="create_append")
print(f"Ingested {rows} rows")
conn.commit()
```
The above version using `to_reader` will split the table into 8 batches of 10M rows each and the driver will generate parquet files and upload them in parallel with multiple PUT commands. The parallelism is configurable via options like:
`adbc.snowflake.statement.ingest_writer_concurrency` --> number of concurrent parquet file writers (default is NUM_CORES)
`adbc.snowflake.statement.ingest_upload_concurrency` --> number of concurrent uploads (default is 8)
`adbc.snowflake.statement.ingest_copy_concurrency` --> number of concurrent COPY INTO commands (default is 4)
You can also split into more chunks and set `adbc.snowflake.statement.ingest_target_file_size` (default is 10MB) which will verify the size of the parquet file after each written record batch to attempt to limit the size. But because the driver currently only checks the size *after* writing a batch, when a table is passed that is not chunked already it will be sent as a single batch and written as a single file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org