You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "davlee1972 (via GitHub)" <gi...@apache.org> on 2025/05/07 18:32:25 UTC

[I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

davlee1972 opened a new issue, #2793:
URL: https://github.com/apache/arrow-adbc/issues/2793

### What happened?

According to the user docs...

**adbc.snowflake.statement.ingest_copy_concurrency**

Maximum number of COPY operations to run concurrently. Bulk ingestion performance is optimized by executing COPY queries as files are still being uploaded. Snowflake COPY speed scales with warehouse size, so smaller warehouses may benefit from setting this value higher to ensure long-running COPY queries do not block newly uploaded files from being loaded. Default is 4. **If set to 0, only a single COPY query will be executed as part of ingestion**, once all files have finished uploading. Cannot be negative

When set "0", the copy into operation never kicks off and adbc_ingest() seems to just hang with wait().

This is a bug with 1.5 and 1.6.

With the '1.5.0dev' driver I downloaded this wasn't a problem. Downgrading to 1.4 also works..

Current solution is to set adbc.snowflake.statement.ingest_copy_concurrency = "1"

### Stack Trace

_No response_

### How can we reproduce the bug?

_No response_

### Environment/Setup

_No response_

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

Posted by "davlee1972 (via GitHub)" <gi...@apache.org>.

davlee1972 commented on issue #2793:
URL: https://github.com/apache/arrow-adbc/issues/2793#issuecomment-2859944244

   The difference is coming from here..
   
   ![Image](https://github.com/user-attachments/assets/5264559d-fab7-40aa-a51f-a66470b4ab67)
   
   ![Image](https://github.com/user-attachments/assets/55b07bf9-7e84-4530-8806-0671e44eb985)
   
   ![Image](https://github.com/user-attachments/assets/10685f44-4838-4ea2-b8ea-fb5705a1473e)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.

zeroshade commented on issue #2793:
URL: https://github.com/apache/arrow-adbc/issues/2793#issuecomment-2860607885

   Okay, thanks. Pretty sure I see the cause of the hang, will test and put up a fix tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

Posted by "davlee1972 (via GitHub)" <gi...@apache.org>.

davlee1972 commented on issue #2793:
URL: https://github.com/apache/arrow-adbc/issues/2793#issuecomment-2860578763

   I opened a support ticket with Snowflake already..
   
   This bug is happening in 1.6 too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.

lidavidm closed issue #2793: adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into
URL: https://github.com/apache/arrow-adbc/issues/2793


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

Posted by "davlee1972 (via GitHub)" <gi...@apache.org>.

davlee1972 commented on issue #2793:
URL: https://github.com/apache/arrow-adbc/issues/2793#issuecomment-2859884250

   The record counts are also off..
   
   Setting it to "1" spawns more than a single copy into operation, but it looks like it is running one at a time.. Not sure if it will run into the existing "select count(*)" copy into termination problem..
   
   With ADBC 1.5.0 driver with adbc.snowflake.statement.ingest_copy_concurrency = "1"
   
   ![Image](https://github.com/user-attachments/assets/fef6d76b-7eaf-4d44-83d6-56af530f4c1a)
   
   ![Image](https://github.com/user-attachments/assets/913b6cda-e854-452a-b997-881292d7aa6d)
   
   ![Image](https://github.com/user-attachments/assets/d7a0c852-14f6-46d7-b6ef-62be6ece8ccc)
   
   The total record counts are correct, but they don't match the rows being reported by copy into..
   
   98,848,888 from the 3 screenshots above vs the correct 98,752,339 total..
   
   ![Image](https://github.com/user-attachments/assets/ff12fba3-a4fd-457e-9995-0201698dc328)
   
   With ADBC 1.5.0 driver with adbc.snowflake.statement.ingest_copy_concurrency = "0". Copy Into never executes.
   
   ![Image](https://github.com/user-attachments/assets/6680b497-7d0c-4ac8-9aa7-f8de2701f5e2)
   
   With ADBC 1.5.0dev driver with adbc.snowflake.statement.ingest_copy_concurrency = "0"
   
   ![Image](https://github.com/user-attachments/assets/cb9b157b-6a28-4661-a671-4540c9e08134)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] adbc.snowflake.statement.ingest_copy_concurrency = "0" blocks copy into [arrow-adbc]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.

zeroshade commented on issue #2793:
URL: https://github.com/apache/arrow-adbc/issues/2793#issuecomment-2860534081

   The difference between what `COPY INTO` reports and what is actually loaded is something we should bring up to Snowflake as I'm not sure what we can do about it in the ADBC driver as we can only report what COPY INTO returns. 
   
   For the hanging, I'll take a look. I'm not sure what changed between that dev build and the release 1.5.0, as we just did a release of 1.6.0 can you just quickly double check whether the issue persists in 1.6.0?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org