You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Erlend Garåsen <e....@usit.uio.no> on 2012/04/23 14:57:31 UTC
Importing old configuration data
I'm trying to import my old configuration files into my freshly
installed MCF version, but I'm afraid that these configuration data is
too old (from version 0.3, I think). I get the following error:
"Database exception: Exception doing query: ERROR: insert or update on
table "jobs" violates foreign key constraint "jobs_outputname_fkey"
Detail: Key (outputname)=(Solr) is not present in table
"outputconnections"".
Here's what I tried to do. The last step fails:
./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
./processes/script/executecommand.sh
org.apache.manifoldcf.agents.RegisterOutput
org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
./processes/script/executecommand.sh
org.apache.manifoldcf.agents.Register
org.apache.manifoldcf.crawler.system.CrawlerAgent
./processes/script/executecommand.sh
org.apache.manifoldcf.crawler.Register
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
WebCrawler
./processes/script/executecommand.sh
org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
Complete stack trace:
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
exception: Exception doing query: ERROR: insert or update on table
"jobs" violates foreign key constraint "jobs_outputname_fkey"
Detail: Key (outputname)=(Solr) is not present in table
"outputconnections".
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
at org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
at
org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
at
org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
at
org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
at
org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
at
org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
table "jobs" violates foreign key constraint "jobs_outputname_fkey"
Detail: Key (outputname)=(Solr) is not present in table
"outputconnections".
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Importing old configuration data
Posted by Erlend Garåsen <e....@usit.uio.no>.
Thanks a lot, Karl. Yes, I messed with the file (unzipped, changed a
password and zipped it again). I did not think the order of the content
was relevant, but I was wrong. Here is the order after I changed it:
erlendfg$ jar tf crawler_solr-test02_config.zip
Which outputs:
authorities
connections
jobs
outputs
Thanks for your helo!
Erlend
On 24.04.12 07.34, Karl Wright wrote:
> Oh, wait - you messed with the file, right? Probably you changed the
> order of the entries around by doing so...
>
> The current 0.5 code outputs the right order:
>
> java.util.zip.ZipEntry outputEntry = new
> java.util.zip.ZipEntry("outputs");
> zos.putNextEntry(outputEntry);
> outputManager.exportConfiguration(zos);
> zos.closeEntry();
>
> java.util.zip.ZipEntry authEntry = new
> java.util.zip.ZipEntry("authorities");
> zos.putNextEntry(authEntry);
> authManager.exportConfiguration(zos);
> zos.closeEntry();
>
> java.util.zip.ZipEntry connEntry = new
> java.util.zip.ZipEntry("connections");
> zos.putNextEntry(connEntry);
> connManager.exportConfiguration(zos);
> zos.closeEntry();
>
> java.util.zip.ZipEntry jobsEntry = new
> java.util.zip.ZipEntry("jobs");
> zos.putNextEntry(jobsEntry);
> jobManager.exportConfiguration(zos);
> zos.closeEntry();
>
>
> ... so there's no *current* problem with this. If you've messed with
> the order though and still want to use the file, try this:
>
> - unpack
> - pack each individual entry in its own zip
> - import them, one by one, in the order above (outputs, authorities,
> connections, jobs)
>
> I bet that will work for you...
>
> Karl
>
>
> On Mon, Apr 23, 2012 at 1:33 PM, Karl Wright<da...@gmail.com> wrote:
>> The problem is not with the current version; it's with the ManifoldCF
>> 0.3 version, which did not output things in the correct order.
>> Therefore the outputs do not get processed before the jobs do, leading
>> to this problem.
>>
>> There are two possible solutions. The first is to hack up your copy
>> of 0.5 to do TWO passes over the data file, looking for "outputs" on
>> the first pass, and looking for everything on the second. The second
>> is create the "Solr" output connection by hand; running the script
>> after that should populate everything.
>>
>> Thanks,
>> Karl
>>
>>
>> On Mon, Apr 23, 2012 at 9:25 AM, Karl Wright<da...@gmail.com> wrote:
>>> Hmm. Your output shows that the output connector is indeed registered:
>>>
>>> ./processes/script/executecommand.sh
>>> org.apache.manifoldcf.agents.RegisterOutput
>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>>
>>> If you have no passwords in the config, you can send me the file and
>>> I'll debug it to see what's going on.
>>>
>>> Karl
>>>
>>> On Mon, Apr 23, 2012 at 9:04 AM, Karl Wright<da...@gmail.com> wrote:
>>>> Hmm. There have been no changes to the format or meaning of the
>>>> configuration file AFAIK for quite a while.
>>>>
>>>> This is complaining about there not being an output connection called
>>>> "Solr" in place at the time it tries to create a job. In theory, on
>>>> import, the creation of connections takes place before the creation of
>>>> jobs. Is there any possibility that the connectors have not been
>>>> registered? I think they would need to be before you attempt the
>>>> import.
>>>>
>>>> Karl
>>>>
>>>> On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen<e....@usit.uio.no> wrote:
>>>>>
>>>>> I'm trying to import my old configuration files into my freshly installed
>>>>> MCF version, but I'm afraid that these configuration data is too old (from
>>>>> version 0.3, I think). I get the following error: "Database exception:
>>>>> Exception doing query: ERROR: insert or update on table "jobs" violates
>>>>> foreign key constraint "jobs_outputname_fkey"
>>>>> Detail: Key (outputname)=(Solr) is not present in table
>>>>> "outputconnections"".
>>>>>
>>>>> Here's what I tried to do. The last step fails:
>>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
>>>>> ./processes/script/executecommand.sh
>>>>> org.apache.manifoldcf.agents.RegisterOutput
>>>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
>>>>> org.apache.manifoldcf.crawler.system.CrawlerAgent
>>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
>>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>>>>> WebCrawler
>>>>> ./processes/script/executecommand.sh
>>>>> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>>>>>
>>>>> Complete stack trace:
>>>>>
>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>>>>> exception: Exception doing query: ERROR: insert or update on table "jobs"
>>>>> violates foreign key constraint "jobs_outputname_fkey"
>>>>> Detail: Key (outputname)=(Solr) is not present in table
>>>>> "outputconnections".
>>>>> at
>>>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
>>>>> at
>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>>>>> at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
>>>>> at
>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
>>>>> at
>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
>>>>> at
>>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
>>>>> at
>>>>> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
>>>>> at
>>>>> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
>>>>> at
>>>>> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
>>>>> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
>>>>> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
>>>>> Detail: Key (outputname)=(Solr) is not present in table
>>>>> "outputconnections".
>>>>> at
>>>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
>>>>> at
>>>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
>>>>> at
>>>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>>>>> at
>>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
>>>>> at
>>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>>>>> at
>>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
>>>>> at
>>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Importing old configuration data
Posted by Karl Wright <da...@gmail.com>.
Oh, wait - you messed with the file, right? Probably you changed the
order of the entries around by doing so...
The current 0.5 code outputs the right order:
java.util.zip.ZipEntry outputEntry = new
java.util.zip.ZipEntry("outputs");
zos.putNextEntry(outputEntry);
outputManager.exportConfiguration(zos);
zos.closeEntry();
java.util.zip.ZipEntry authEntry = new
java.util.zip.ZipEntry("authorities");
zos.putNextEntry(authEntry);
authManager.exportConfiguration(zos);
zos.closeEntry();
java.util.zip.ZipEntry connEntry = new
java.util.zip.ZipEntry("connections");
zos.putNextEntry(connEntry);
connManager.exportConfiguration(zos);
zos.closeEntry();
java.util.zip.ZipEntry jobsEntry = new
java.util.zip.ZipEntry("jobs");
zos.putNextEntry(jobsEntry);
jobManager.exportConfiguration(zos);
zos.closeEntry();
... so there's no *current* problem with this. If you've messed with
the order though and still want to use the file, try this:
- unpack
- pack each individual entry in its own zip
- import them, one by one, in the order above (outputs, authorities,
connections, jobs)
I bet that will work for you...
Karl
On Mon, Apr 23, 2012 at 1:33 PM, Karl Wright <da...@gmail.com> wrote:
> The problem is not with the current version; it's with the ManifoldCF
> 0.3 version, which did not output things in the correct order.
> Therefore the outputs do not get processed before the jobs do, leading
> to this problem.
>
> There are two possible solutions. The first is to hack up your copy
> of 0.5 to do TWO passes over the data file, looking for "outputs" on
> the first pass, and looking for everything on the second. The second
> is create the "Solr" output connection by hand; running the script
> after that should populate everything.
>
> Thanks,
> Karl
>
>
> On Mon, Apr 23, 2012 at 9:25 AM, Karl Wright <da...@gmail.com> wrote:
>> Hmm. Your output shows that the output connector is indeed registered:
>>
>> ./processes/script/executecommand.sh
>> org.apache.manifoldcf.agents.RegisterOutput
>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>
>> If you have no passwords in the config, you can send me the file and
>> I'll debug it to see what's going on.
>>
>> Karl
>>
>> On Mon, Apr 23, 2012 at 9:04 AM, Karl Wright <da...@gmail.com> wrote:
>>> Hmm. There have been no changes to the format or meaning of the
>>> configuration file AFAIK for quite a while.
>>>
>>> This is complaining about there not being an output connection called
>>> "Solr" in place at the time it tries to create a job. In theory, on
>>> import, the creation of connections takes place before the creation of
>>> jobs. Is there any possibility that the connectors have not been
>>> registered? I think they would need to be before you attempt the
>>> import.
>>>
>>> Karl
>>>
>>> On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>>>
>>>> I'm trying to import my old configuration files into my freshly installed
>>>> MCF version, but I'm afraid that these configuration data is too old (from
>>>> version 0.3, I think). I get the following error: "Database exception:
>>>> Exception doing query: ERROR: insert or update on table "jobs" violates
>>>> foreign key constraint "jobs_outputname_fkey"
>>>> Detail: Key (outputname)=(Solr) is not present in table
>>>> "outputconnections"".
>>>>
>>>> Here's what I tried to do. The last step fails:
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
>>>> ./processes/script/executecommand.sh
>>>> org.apache.manifoldcf.agents.RegisterOutput
>>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
>>>> org.apache.manifoldcf.crawler.system.CrawlerAgent
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>>>> WebCrawler
>>>> ./processes/script/executecommand.sh
>>>> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>>>>
>>>> Complete stack trace:
>>>>
>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>>>> exception: Exception doing query: ERROR: insert or update on table "jobs"
>>>> violates foreign key constraint "jobs_outputname_fkey"
>>>> Detail: Key (outputname)=(Solr) is not present in table
>>>> "outputconnections".
>>>> at
>>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
>>>> at
>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
>>>> at
>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
>>>> at
>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>> at
>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
>>>> at
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>>>> at
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>>>> at
>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>>>> at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
>>>> at
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
>>>> at
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
>>>> at
>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
>>>> at
>>>> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
>>>> at
>>>> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
>>>> at
>>>> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
>>>> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
>>>> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
>>>> Detail: Key (outputname)=(Solr) is not present in table
>>>> "outputconnections".
>>>> at
>>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
>>>> at
>>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
>>>> at
>>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>>>> at
>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
>>>> at
>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>>>> at
>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>>>> at
>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
>>>> at
>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Importing old configuration data
Posted by Karl Wright <da...@gmail.com>.
The problem is not with the current version; it's with the ManifoldCF
0.3 version, which did not output things in the correct order.
Therefore the outputs do not get processed before the jobs do, leading
to this problem.
There are two possible solutions. The first is to hack up your copy
of 0.5 to do TWO passes over the data file, looking for "outputs" on
the first pass, and looking for everything on the second. The second
is create the "Solr" output connection by hand; running the script
after that should populate everything.
Thanks,
Karl
On Mon, Apr 23, 2012 at 9:25 AM, Karl Wright <da...@gmail.com> wrote:
> Hmm. Your output shows that the output connector is indeed registered:
>
> ./processes/script/executecommand.sh
> org.apache.manifoldcf.agents.RegisterOutput
> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>
> If you have no passwords in the config, you can send me the file and
> I'll debug it to see what's going on.
>
> Karl
>
> On Mon, Apr 23, 2012 at 9:04 AM, Karl Wright <da...@gmail.com> wrote:
>> Hmm. There have been no changes to the format or meaning of the
>> configuration file AFAIK for quite a while.
>>
>> This is complaining about there not being an output connection called
>> "Solr" in place at the time it tries to create a job. In theory, on
>> import, the creation of connections takes place before the creation of
>> jobs. Is there any possibility that the connectors have not been
>> registered? I think they would need to be before you attempt the
>> import.
>>
>> Karl
>>
>> On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>>
>>> I'm trying to import my old configuration files into my freshly installed
>>> MCF version, but I'm afraid that these configuration data is too old (from
>>> version 0.3, I think). I get the following error: "Database exception:
>>> Exception doing query: ERROR: insert or update on table "jobs" violates
>>> foreign key constraint "jobs_outputname_fkey"
>>> Detail: Key (outputname)=(Solr) is not present in table
>>> "outputconnections"".
>>>
>>> Here's what I tried to do. The last step fails:
>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
>>> ./processes/script/executecommand.sh
>>> org.apache.manifoldcf.agents.RegisterOutput
>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
>>> org.apache.manifoldcf.crawler.system.CrawlerAgent
>>> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>>> WebCrawler
>>> ./processes/script/executecommand.sh
>>> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>>>
>>> Complete stack trace:
>>>
>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>>> exception: Exception doing query: ERROR: insert or update on table "jobs"
>>> violates foreign key constraint "jobs_outputname_fkey"
>>> Detail: Key (outputname)=(Solr) is not present in table
>>> "outputconnections".
>>> at
>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
>>> at
>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
>>> at
>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
>>> at
>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>> at
>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
>>> at
>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>>> at
>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>>> at
>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>>> at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
>>> at
>>> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
>>> at
>>> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
>>> at
>>> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
>>> at
>>> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
>>> at
>>> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
>>> at
>>> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
>>> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
>>> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
>>> Detail: Key (outputname)=(Solr) is not present in table
>>> "outputconnections".
>>> at
>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
>>> at
>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
>>> at
>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>>> at
>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
>>> at
>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Importing old configuration data
Posted by Karl Wright <da...@gmail.com>.
Hmm. Your output shows that the output connector is indeed registered:
./processes/script/executecommand.sh
org.apache.manifoldcf.agents.RegisterOutput
org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
If you have no passwords in the config, you can send me the file and
I'll debug it to see what's going on.
Karl
On Mon, Apr 23, 2012 at 9:04 AM, Karl Wright <da...@gmail.com> wrote:
> Hmm. There have been no changes to the format or meaning of the
> configuration file AFAIK for quite a while.
>
> This is complaining about there not being an output connection called
> "Solr" in place at the time it tries to create a job. In theory, on
> import, the creation of connections takes place before the creation of
> jobs. Is there any possibility that the connectors have not been
> registered? I think they would need to be before you attempt the
> import.
>
> Karl
>
> On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> I'm trying to import my old configuration files into my freshly installed
>> MCF version, but I'm afraid that these configuration data is too old (from
>> version 0.3, I think). I get the following error: "Database exception:
>> Exception doing query: ERROR: insert or update on table "jobs" violates
>> foreign key constraint "jobs_outputname_fkey"
>> Detail: Key (outputname)=(Solr) is not present in table
>> "outputconnections"".
>>
>> Here's what I tried to do. The last step fails:
>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
>> ./processes/script/executecommand.sh
>> org.apache.manifoldcf.agents.RegisterOutput
>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
>> org.apache.manifoldcf.crawler.system.CrawlerAgent
>> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>> WebCrawler
>> ./processes/script/executecommand.sh
>> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>>
>> Complete stack trace:
>>
>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>> exception: Exception doing query: ERROR: insert or update on table "jobs"
>> violates foreign key constraint "jobs_outputname_fkey"
>> Detail: Key (outputname)=(Solr) is not present in table
>> "outputconnections".
>> at
>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
>> at
>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
>> at
>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
>> at
>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>> at
>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
>> at
>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>> at
>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>> at
>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>> at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
>> at
>> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
>> at
>> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
>> at
>> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
>> at
>> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
>> at
>> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
>> at
>> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
>> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
>> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
>> Detail: Key (outputname)=(Solr) is not present in table
>> "outputconnections".
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>> at
>> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
>> at
>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Importing old configuration data
Posted by Karl Wright <da...@gmail.com>.
Hmm. There have been no changes to the format or meaning of the
configuration file AFAIK for quite a while.
This is complaining about there not being an output connection called
"Solr" in place at the time it tries to create a job. In theory, on
import, the creation of connections takes place before the creation of
jobs. Is there any possibility that the connectors have not been
registered? I think they would need to be before you attempt the
import.
Karl
On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I'm trying to import my old configuration files into my freshly installed
> MCF version, but I'm afraid that these configuration data is too old (from
> version 0.3, I think). I get the following error: "Database exception:
> Exception doing query: ERROR: insert or update on table "jobs" violates
> foreign key constraint "jobs_outputname_fkey"
> Detail: Key (outputname)=(Solr) is not present in table
> "outputconnections"".
>
> Here's what I tried to do. The last step fails:
> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
> ./processes/script/executecommand.sh
> org.apache.manifoldcf.agents.RegisterOutput
> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
> org.apache.manifoldcf.crawler.system.CrawlerAgent
> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
> WebCrawler
> ./processes/script/executecommand.sh
> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>
> Complete stack trace:
>
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: Exception doing query: ERROR: insert or update on table "jobs"
> violates foreign key constraint "jobs_outputname_fkey"
> Detail: Key (outputname)=(Solr) is not present in table
> "outputconnections".
> at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
> at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
> at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
> at
> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
> at
> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
> at
> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
> at
> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
> Detail: Key (outputname)=(Solr) is not present in table
> "outputconnections".
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
> at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
> at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050