You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues-all@impala.apache.org by "David Rorke (Jira)" <ji...@apache.org> on 2022/05/04 22:57:00 UTC

[jira] [Assigned] (IMPALA-2607) S3: need to determine the right default for fs.s3a.connection.maximum

     [ https://issues.apache.org/jira/browse/IMPALA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Rorke reassigned IMPALA-2607:
-----------------------------------

    Assignee: David Rorke

> S3: need to determine the right default for fs.s3a.connection.maximum
> ---------------------------------------------------------------------
>
>                 Key: IMPALA-2607
>                 URL: https://issues.apache.org/jira/browse/IMPALA-2607
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.3.0
>            Reporter: Daniel Hecht
>            Assignee: David Rorke
>            Priority: Major
>              Labels: s3, usability
>
> When Impala is scanning parquet files off S3, it appears we can get into a deadlock if fs.s3a.connection.maximum is not set high enough, because we can spread the connection use across files but unless we have a connection for all selected columns of a file, we can't make progress scanning that file.  So, we need to figure out the right default for fs.s3a.connection.maximum for Impala, and also determine whether we need to put an upper bound on scanner concurrency given the number of connections.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org