You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Hui Zheng (JIRA)" <ji...@apache.org> on 2018/02/28 02:22:00 UTC
[jira] [Commented] (PHOENIX-4631) PhoenixInputFormat should close
connection after generateSplits()
[ https://issues.apache.org/jira/browse/PHOENIX-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379674#comment-16379674 ]
Hui Zheng commented on PHOENIX-4631:
------------------------------------
phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java
I think the connection should be cloesd after splits get generated here.
{code:java}
private List<InputSplit> generateSplits(final QueryPlan qplan, final List<KeyRange> splits, Configuration config) throws IOException {
Preconditions.checkNotNull(qplan);
Preconditions.checkNotNull(splits);
// Get the RegionSizeCalculator
try(org.apache.hadoop.hbase.client.Connection connection =
HBaseFactoryProvider.getHConnectionFactory().createConnection(config)) {
RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
.getTableRef().getTable().getPhysicalName().toString()));
RegionSizeCalculator sizeCalculator = new RegionSizeCalculator(regionLocator, connection
.getAdmin());
{code}
> PhoenixInputFormat should close connection after generateSplits()
> -----------------------------------------------------------------
>
> Key: PHOENIX-4631
> URL: https://issues.apache.org/jira/browse/PHOENIX-4631
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.10.0
> Reporter: Hui Zheng
> Priority: Major
>
> In our sparkstreaming usecase which loads a phoenix table as a Dataset, it will leak zookeeper connection(3 connections per batch) and lead to OOM exception in its driver process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)