You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Laszlo Gaal (JIRA)" <ji...@apache.org> on 2017/04/10 12:02:41 UTC
[jira] [Resolved] (IMPALA-4088) HDFS data nodes pick HTTP server
ports at random, sometimes stealing HBase master's port
[ https://issues.apache.org/jira/browse/IMPALA-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Laszlo Gaal resolved IMPALA-4088.
---------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.9.0
https://git-wip-us.apache.org/repos/asf?p=incubator-impala.git;a=commit;h=9e7fb830fd047ed6445ea82280993e56ea51e15b
IMPALA-4088: Assign fix values to the minicluster server ports
The minicluster setup logic assigned fixed port numbers to several
but not all listening sockets of the data nodes. This change
assigns similar port ranges to all the listening ports that were
so far allowed to pick their own port numbers, interfering with
other components, e.g. HBase.
Change-Id: Iecf312873b7026c52b0ac0e71adbecab181925a0
Reviewed-on: http://gerrit.cloudera.org:8080/6531
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Impala Public Jenkins
> HDFS data nodes pick HTTP server ports at random, sometimes stealing HBase master's port
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-4088
> URL: https://issues.apache.org/jira/browse/IMPALA-4088
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 2.7.0
> Reporter: Alexander Behm
> Assignee: Laszlo Gaal
> Priority: Critical
> Labels: broken-build
> Fix For: Impala 2.9.0
>
> Attachments: logs.tar.gz
>
>
> Michael, can you take a first look since you've dabbled with HBase startup before? It looks like HBase may not be in a good state. I will not cancel the build so you can log into the machine if necessary. Feel free to cancel the build if you have all the info you need.
> There are several issues here:
> * figure out why data loading did not succeed
> * the build should not hang even if there are errors in data loading
> This is where it hangs:
> {code}
> ...
> 00:35:24.252 [INFO] BUILD SUCCESS
> 00:35:24.253 ------------------------------------------------------------------------
> 00:35:24.253
> 00:35:24.260 ========================================================================
> 00:35:24.260 Running mvn package
> 00:35:24.262 Directory: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata
> 00:35:24.262 ========================================================================
> 00:35:32.241 [INFO] /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata/src/main/java/com/cloudera/impala/datagenerator/HBaseTestDataRegionAssigment.java: Some input files use or override a deprecated API.
> 00:35:32.242 [INFO] /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata/src/main/java/com/cloudera/impala/datagenerator/HBaseTestDataRegionAssigment.java: Recompile with -Xlint:deprecation for details.
> 00:35:32.242 [INFO] BUILD SUCCESS
> 00:35:32.242 ------------------------------------------------------------------------
> 00:35:32.243
> 00:35:32.249 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata/target
> 00:35:34.029 SUCCESS, data generated into /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata/target
> 00:45:44.231 Loading HDFS data from snapshot: /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh5-59-SNAPSHOT.tar.gz (logging to load-test-warehouse-snapshot.log)... OK
> 00:46:24.762 Starting Impala cluster (logging to start-impala-cluster.log)... OK
> 00:46:44.765 Setting up HDFS environment (logging to setup-hdfs-env.log)... OK
> 00:46:44.765 Skipped loading the metadata.
> <does not proceed beyond here>
> {code}
> From the hive logs (repeats very often):
> {code}
> org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for 126 in functional_hbase.alltypes after 35 tries.
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1329)
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1199)
> at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:395)
> at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:238)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:146)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:113)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1084)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat$MyRecordWriter.write(HiveHBaseTableOutputFormat.java:146)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat$MyRecordWriter.write(HiveHBaseTableOutputFormat.java:117)
> at org.apache.hadoop.hive.ql.io.HivePassThroughRecordWriter.write(HivePassThroughRecordWriter.java:40)
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Build:
> http://sandbox.jenkins.cloudera.com/job/impala-umbrella-build-and-test/4465/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)