You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Cheng Zeng <ze...@hotmail.co.uk> on 2017/03/08 15:43:12 UTC
The job got stuck when JDBC Connector got
org.postgresql.util.PSQLException
Hi all,
I was trying to use JDBC connector to index records from MS SQLServer 2012. I used ManifoldCF 2.3.
I created a job which need index over 80,000 records in the table. However, The job worker stopped processing more docs once org.postgresql.util.PSQLException was thrown.
Any help would be appreciated.
Thanks,
Cheng
Here is the track.
DEBUG 2017-03-08 22:46:14,738 (Document delete stuffer thread) - Document delete stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Deleting {}
DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Hopcount removal {}
DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Rescanning documents {}
ERROR 2017-03-08 22:46:14,834 (Worker thread '21') - Worker thread aborting and restarting due to database connection reset: Database exception: SQLException doing query (22021): ERROR: invalid byte sequence for encoding "UTF8": 0x00
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database exception: SQLException doing query (22021): ERROR: invalid byte sequence for encoding "UTF8": 0x00
at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)
at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)
at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:784)
at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:661)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performUpdate(DBInterfacePostgreSQL.java:254)
at org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.noteDocumentIngest(IncrementalIngester.java:2101)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$OutputAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3403)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3072)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2706)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548)
at org.apache.manifoldcf.crawler.connectors.jdbc.JDBCConnector.processDocuments(JDBCConnector.java:826)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:916)
at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696)
DEBUG 2017-03-08 22:46:15,581 (Set priority thread) - Done reprioritizing because exceeded cycle count
DEBUG 2017-03-08 22:46:15,581 (Set priority thread) - Set priority thread woke up
DEBUG 2017-03-08 22:46:15,586 (Stuffer thread) - Stuffer thread: Found 105 documents to queue
DEBUG 2017-03-08 22:46:15,738 (Document delete stuffer thread) - Document delete stuffer thread woke up
DEBUG 2017-03-08 22:46:15,738 (Document cleanup stuffer thread) - Document cleanup stuffer thread woke up
DEBUG 2017-03-08 22:46:15,738 (Document delete stuffer thread) - Document delete stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:15,738 (Document cleanup stuffer thread) - Document cleanup stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:15,762 (Delete startup thread) - Checking for deleting jobs
DEBUG 2017-03-08 22:46:15,763 (Job delete thread) - Deleting jobs that need cleanup
DEBUG 2017-03-08 22:46:15,763 (Startup thread) - Checking for starting jobs
DEBUG 2017-03-08 22:46:17,742 (Document cleanup stuffer thread) - Document cleanup stuffer thread woke up
DEBUG 2017-03-08 22:46:17,742 (Document delete stuffer thread) - Document delete stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:17,742 (Document cleanup stuffer thread) - Document cleanup stuffer thread found nothing to do
Re: The job got stuck when JDBC Connector got
org.postgresql.util.PSQLException
Posted by Cheng Zeng <ze...@hotmail.co.uk>.
Hi Karl,
Thank you very much for your reply. I am not sure why there is a character zero returned by MS SQL server. I use a simple version query and there is no PSQLException thrown though. The job thread works fine now.
Thanks,
Cheng
________________________________
From: Karl Wright <da...@gmail.com>
Sent: 08 March 2017 16:04
To: user@manifoldcf.apache.org
Subject: Re: The job got stuck when JDBC Connector got org.postgresql.util.PSQLException
Hi Cheng,
The issue is that your JDBC connection is generating a version string that has a character zero (0x0) in it, and postgresql doesn't allow that.
You get to specify the version string query as part of the job definition -- can you look at that and see how you are getting this back? It might be due to the JDBC driver doing something funny, or it might be that your supplied version query is querying for something that's really binary, and you'll need to do something different to make it work.
Thanks,
Karl
On Wed, Mar 8, 2017 at 10:43 AM, Cheng Zeng <ze...@hotmail.co.uk>> wrote:
Hi all,
I was trying to use JDBC connector to index records from MS SQLServer 2012. I used ManifoldCF 2.3.
I created a job which need index over 80,000 records in the table. However, The job worker stopped processing more docs once org.postgresql.util.PSQLException was thrown.
Any help would be appreciated.
Thanks,
Cheng
Here is the track.
DEBUG 2017-03-08 22:46:14,738 (Document delete stuffer thread) - Document delete stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Deleting {}
DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Hopcount removal {}
DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Rescanning documents {}
ERROR 2017-03-08 22:46:14,834 (Worker thread '21') - Worker thread aborting and restarting due to database connection reset: Database exception: SQLException doing query (22021): ERROR: invalid byte sequence for encoding "UTF8": 0x00
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database exception: SQLException doing query (22021): ERROR: invalid byte sequence for encoding "UTF8": 0x00
at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)
at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)
at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:784)
at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:661)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performUpdate(DBInterfacePostgreSQL.java:254)
at org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.noteDocumentIngest(IncrementalIngester.java:2101)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$OutputAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3403)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3072)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2706)
at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548)
at org.apache.manifoldcf.crawler.connectors.jdbc.JDBCConnector.processDocuments(JDBCConnector.java:826)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:916)
at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696)
DEBUG 2017-03-08 22:46:15,581 (Set priority thread) - Done reprioritizing because exceeded cycle count
DEBUG 2017-03-08 22:46:15,581 (Set priority thread) - Set priority thread woke up
DEBUG 2017-03-08 22:46:15,586 (Stuffer thread) - Stuffer thread: Found 105 documents to queue
DEBUG 2017-03-08 22:46:15,738 (Document delete stuffer thread) - Document delete stuffer thread woke up
DEBUG 2017-03-08 22:46:15,738 (Document cleanup stuffer thread) - Document cleanup stuffer thread woke up
DEBUG 2017-03-08 22:46:15,738 (Document delete stuffer thread) - Document delete stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:15,738 (Document cleanup stuffer thread) - Document cleanup stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:15,762 (Delete startup thread) - Checking for deleting jobs
DEBUG 2017-03-08 22:46:15,763 (Job delete thread) - Deleting jobs that need cleanup
DEBUG 2017-03-08 22:46:15,763 (Startup thread) - Checking for starting jobs
DEBUG 2017-03-08 22:46:17,742 (Document cleanup stuffer thread) - Document cleanup stuffer thread woke up
DEBUG 2017-03-08 22:46:17,742 (Document delete stuffer thread) - Document delete stuffer thread found nothing to do
DEBUG 2017-03-08 22:46:17,742 (Document cleanup stuffer thread) - Document cleanup stuffer thread found nothing to do
Re: The job got stuck when JDBC Connector got org.postgresql.util.PSQLException
Posted by Karl Wright <da...@gmail.com>.
Hi Cheng,
The issue is that your JDBC connection is generating a version string that
has a character zero (0x0) in it, and postgresql doesn't allow that.
You get to specify the version string query as part of the job definition
-- can you look at that and see how you are getting this back? It might be
due to the JDBC driver doing something funny, or it might be that your
supplied version query is querying for something that's really binary, and
you'll need to do something different to make it work.
Thanks,
Karl
On Wed, Mar 8, 2017 at 10:43 AM, Cheng Zeng <ze...@hotmail.co.uk> wrote:
> Hi all,
>
>
> I was trying to use JDBC connector to index records from MS SQLServer
> 2012. I used ManifoldCF 2.3.
>
>
> I created a job which need index over 80,000 records in the table.
> However, The job worker stopped processing more docs once
> org.postgresql.util.PSQLException was thrown.
>
>
> Any help would be appreciated.
>
>
> Thanks,
>
> Cheng
>
>
> Here is the track.
>
>
> DEBUG 2017-03-08 22:46:14,738 (Document delete stuffer thread) - Document
> delete stuffer thread found nothing to do
> DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Deleting {}
> DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Hopcount removal {}
> DEBUG 2017-03-08 22:46:14,745 (Worker thread '28') - Rescanning documents
> {}
> ERROR 2017-03-08 22:46:14,834 (Worker thread '21') - Worker thread
> aborting and restarting due to database connection reset: Database
> exception: SQLException doing query (22021): ERROR: invalid byte sequence
> for encoding "UTF8": 0x00
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (22021): ERROR: invalid byte sequence
> for encoding "UTF8": 0x00
> at org.apache.manifoldcf.core.database.Database$
> ExecuteQueryThread.finishUp(Database.java:715)
> at org.apache.manifoldcf.core.database.Database.
> executeViaThread(Database.java:741)
> at org.apache.manifoldcf.core.database.Database.
> executeUncachedQuery(Database.java:784)
> at org.apache.manifoldcf.core.database.Database$
> QueryCacheExecutor.create(Database.java:1457)
> at org.apache.manifoldcf.core.cachemanager.CacheManager.
> findObjectsAndExecute(CacheManager.java:146)
> at org.apache.manifoldcf.core.database.Database.
> executeQuery(Database.java:204)
> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.
> performModification(DBInterfacePostgreSQL.java:661)
> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.
> performUpdate(DBInterfacePostgreSQL.java:254)
> at org.apache.manifoldcf.core.database.BaseTable.
> performUpdate(BaseTable.java:80)
> at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.
> noteDocumentIngest(IncrementalIngester.java:2101)
> at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$
> OutputAddEntryPoint.addOrReplaceDocumentWithExcept
> ion(IncrementalIngester.java:3403)
> at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$
> PipelineAddFanout.sendDocument(IncrementalIngester.java:3072)
> at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$
> PipelineObjectWithVersions.addOrReplaceDocumentWithExcept
> ion(IncrementalIngester.java:2706)
> at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.
> documentIngest(IncrementalIngester.java:756)
> at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.
> ingestDocumentWithException(WorkerThread.java:1583)
> at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.
> ingestDocumentWithException(WorkerThread.java:1548)
> at org.apache.manifoldcf.crawler.connectors.jdbc.JDBCConnector.
> processDocuments(JDBCConnector.java:826)
> at org.apache.manifoldcf.crawler.system.WorkerThread.run(
> WorkerThread.java:399)
> Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte
> sequence for encoding "UTF8": 0x00
> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(
> QueryExecutorImpl.java:2102)
> at org.postgresql.core.v3.QueryExecutorImpl.processResults(
> QueryExecutorImpl.java:1835)
> at org.postgresql.core.v3.QueryExecutorImpl.execute(
> QueryExecutorImpl.java:257)
> at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(
> AbstractJdbc2Statement.java:500)
> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(
> AbstractJdbc2Statement.java:388)
> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(
> AbstractJdbc2Statement.java:334)
> at org.apache.manifoldcf.core.database.Database.execute(
> Database.java:916)
> at org.apache.manifoldcf.core.database.Database$
> ExecuteQueryThread.run(Database.java:696)
> DEBUG 2017-03-08 22:46:15,581 (Set priority thread) - Done reprioritizing
> because exceeded cycle count
> DEBUG 2017-03-08 22:46:15,581 (Set priority thread) - Set priority thread
> woke up
> DEBUG 2017-03-08 22:46:15,586 (Stuffer thread) - Stuffer thread: Found 105
> documents to queue
> DEBUG 2017-03-08 22:46:15,738 (Document delete stuffer thread) - Document
> delete stuffer thread woke up
> DEBUG 2017-03-08 22:46:15,738 (Document cleanup stuffer thread) - Document
> cleanup stuffer thread woke up
> DEBUG 2017-03-08 22:46:15,738 (Document delete stuffer thread) - Document
> delete stuffer thread found nothing to do
> DEBUG 2017-03-08 22:46:15,738 (Document cleanup stuffer thread) - Document
> cleanup stuffer thread found nothing to do
> DEBUG 2017-03-08 22:46:15,762 (Delete startup thread) - Checking for
> deleting jobs
> DEBUG 2017-03-08 22:46:15,763 (Job delete thread) - Deleting jobs that
> need cleanup
> DEBUG 2017-03-08 22:46:15,763 (Startup thread) - Checking for starting jobs
> DEBUG 2017-03-08 22:46:17,742 (Document cleanup stuffer thread) - Document
> cleanup stuffer thread woke up
> DEBUG 2017-03-08 22:46:17,742 (Document delete stuffer thread) - Document
> delete stuffer thread found nothing to do
> DEBUG 2017-03-08 22:46:17,742 (Document cleanup stuffer thread) - Document
> cleanup stuffer thread found nothing to do
>
>