You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Alexandre Rafalovitch (Jira)" <ji...@apache.org> on 2020/08/29 20:30:00 UTC
[jira] [Resolved] (SOLR-6644) DataImportHandler holds on to each DB
connection until the end
[ https://issues.apache.org/jira/browse/SOLR-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexandre Rafalovitch resolved SOLR-6644.
-----------------------------------------
Resolution: Won't Fix
> DataImportHandler holds on to each DB connection until the end
> --------------------------------------------------------------
>
> Key: SOLR-6644
> URL: https://issues.apache.org/jira/browse/SOLR-6644
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 4.7, 4.10.1
> Reporter: Gunnlaugur Thor Briem
> Priority: Major
> Labels: dataimport, jdbc
>
> DataImportHandler with a JDBC data source opens one DB connection per entity, and then holds on to that DB connection with an open transaction after it's finished processing that entity ... right until the whole DataImportHandler operation is finished.
> So this can mean dozens of DB connections tied up for hours, unnecessarily --- with each connection staying in "idle in transaction" state, holding (in PostgreSQL) an AccessShareLock on each relation it has looked at. Not ideal for production operations, of course.
> Here are the connections from Solr to the DB when a large import has been running for a while:
> {code}
> backend_start | xact_start | query_start | state | minutes idle
> ---------------+------------+-------------+---------------------+--------------
> 20:03:20 | 20:03:20 | 20:03:21 | idle in transaction | 32
> 20:03:22 | 20:03:22 | 20:03:22 | idle in transaction | 32
> 20:03:22 | 20:03:22 | 20:03:22 | idle in transaction | 32
> 20:03:22 | 20:03:22 | 20:03:23 | idle in transaction | 32
> 20:03:21 | 20:03:21 | 20:16:35 | idle in transaction | 19
> 20:03:21 | 20:03:21 | 20:16:35 | idle in transaction | 19
> 20:03:22 | 20:03:22 | 20:16:35 | idle in transaction | 19
> 20:03:22 | 20:03:22 | 20:16:35 | idle in transaction | 19
> 20:03:22 | 20:03:22 | 20:16:35 | idle in transaction | 19
> 20:16:37 | 20:16:37 | 20:16:38 | idle in transaction | 19
> 20:03:21 | 20:03:21 | 20:16:35 | idle in transaction | 19
> 20:03:21 | 20:03:21 | 20:16:35 | idle in transaction | 19
> 20:03:21 | 20:03:21 | 20:16:35 | idle in transaction | 19
> 20:16:36 | 20:16:36 | 20:16:37 | idle in transaction | 19
> 20:03:20 | 20:03:20 | 20:16:35 | idle in transaction | 19
> 20:16:36 | 20:16:36 | 20:35:49 | idle in transaction | 0
> 20:16:36 | 20:16:36 | 20:35:49 | idle in transaction | 0
> 20:16:37 | 20:16:37 | 20:35:49 | idle in transaction | 0
> 20:16:35 | 20:16:35 | 20:35:41 | idle in transaction | 0
> 20:16:36 | 20:16:36 | 20:35:49 | idle in transaction | 0
> 20:16:37 | 20:16:37 | 20:35:49 | active | 0
> {code}
> Most of these haven't been touched for a long time, and will not be needed again (because DataImportHandler is done with that top-level entity). They should be released as soon as possible.
> Noticed in production in Solr 4.7.0, then reproduced in 4.10.1 (so probably also true of all versions inbetween).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org