You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Manish Maheshwari (JIRA)" <ji...@apache.org> on 2017/06/06 03:29:18 UTC
[jira] [Created] (IMPALA-5442) Allow HDFS replication factor to be
set
Manish Maheshwari created IMPALA-5442:
-----------------------------------------
Summary: Allow HDFS replication factor to be set
Key: IMPALA-5442
URL: https://issues.apache.org/jira/browse/IMPALA-5442
Project: IMPALA
Issue Type: New Feature
Components: Backend, Clients
Affects Versions: Impala 2.8.0
Reporter: Manish Maheshwari
Currently for small tables with just 1 HDFS file, the file is potentially available on default 3 DN's. If the file content was static, we would run a hdfs dfs -setrep and then issue a refresh table/partition command to cache the newly replicated block locations.
However since hdfs dfs -setrep is not synchronous, we don't really know when to issue a setrep command.
HDFS caching helps this a bit, but has the same issue that we need to do a setrep and then refresh the table to get the cached and on disk block locations into the catalog.
A good feature for small tables would be to allow HDFS replication factor be specified as part of the Impala INSERT INTO clause, (Cannot use this as part of create table as Hive would not support this.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)