You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Hudson (Jira)" <ji...@apache.org> on 2021/01/05 19:44:01 UTC
[jira] [Commented] (HBASE-25318) Configure where
IntegrationTestImportTsv generates HFiles
[ https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259167#comment-17259167 ]
Hudson commented on HBASE-25318:
--------------------------------
Results for branch branch-2.2
[build #144 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/144/]: (x) *{color:red}-1 overall{color}*
----
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/144//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/144//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/144//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/144//artifact/output-integration/hadoop-2.log]. (note that this means we didn't run on Hadoop 3)
> Configure where IntegrationTestImportTsv generates HFiles
> ---------------------------------------------------------
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
> Reporter: Mate Szalay-Beko
> Assignee: Mate Szalay-Beko
> Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.1
>
>
> Currently IntegrationTestImportTsv is generating HFiles under the working directory of the hdfs user executing the tool, before bulkloading it into HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS Transparent Encryption), you can bulkload HFiles only if they sit in the same encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster and the working directory of the current user (e.g. /user/hbase) is not in the same encryption zone as the HBase root directory (e.g. /hbase/data) then you will get an exception:
>
> {code:java}
> ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc can't be moved into an encryption zone.
> {code}
>
> In this ticket I make it configurable where the IntegrationTestImportTsv generates the HFiles. From now, one can execute this integration test on HDFS Transparent Encryption enabled clusters, like:
> {code:java}
> ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D IntegrationTestImportTsv.generatedHFileFolder=/<my hbase encryption zone path>/testdata{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)