You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2022/11/14 22:46:00 UTC
[jira] [Comment Edited] (IMPALA-9437) Cannot get Ozone file block size after opening the file
[ https://issues.apache.org/jira/browse/IMPALA-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17574946#comment-17574946 ]
Michael Smith edited comment on IMPALA-9437 at 11/14/22 10:45 PM:
------------------------------------------------------------------
-I don't think we can change Ozone's block size from 256MB. It looks like a property the client sends, and we don't support sending it.-
Ozone's SCM UI reports when I change {{ozone.scm.block.size}}, so it's having some effect. Hard to see any other impacts from changing it, and for some reason the Ozone client in Impala doesn't seem to be picking up my configuration change.
Update: filed HDDS-7325 as Ozone only reports the right size if we explicitly pass an OzoneConfiguration object. That's not going to be very feasible to do.
was (Author: JIRAUSER288956):
-I don't think we can change Ozone's block size from 256MB. It looks like a property the client sends, and we don't support sending it.-
Ozone's SCM UI reports when I change {{ozone.scm.block.size}}, so it's having some effect. Hard to see any other impacts from changing it, and for some reason the Ozone client in Impala doesn't seem to be picking up my configuration change.
> Cannot get Ozone file block size after opening the file
> -------------------------------------------------------
>
> Key: IMPALA-9437
> URL: https://issues.apache.org/jira/browse/IMPALA-9437
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0
> Reporter: Sahil Takiar
> Assignee: Michael Smith
> Priority: Minor
>
> When creating a tmp file on HDFS, {{HdfsTableSink::CreateNewTmpFile}} first opens the file, and then stats the file ({{hdfsGetPathInfo}}) before actually writing any data to the file or closing the file. HDFS seems to allow this behavior. However, Ozone, S3A, and ABFS do not. Impala does this for HDFS in order to get the block size of the opened file. According to {{HdfsTableSink}} it is possible for HDFS to create a block size with a different one than requested by Impala. So in order to track the correct block size for a file, the file needs to be stat'ed after opening it. For S3A and ABFS this isn't a big deal, because they aren't block based filesystem, but Ozone is. So we should investigate the impact of not having this capability and consider adding it to the Ozone client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org