Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

I have placed the three war files into Resin's webapps directory and set 
the following system variable inside resin.xml:

<system-property 
org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml" 
/>

I have also created my own home directory for ManifoldCF and copied the 
necessary files from the example directory. When I start Resin and goes 
to ... mcf-crawler-ui/, the start page shows up, but the page looks ugly 
(I guess the path to CSS files etc. are broken). And when I try to click 
on a link, e.g. "List Output Connections", I get the following error:
java.lang.IllegalStateException: can't forward after writing HTTP headers

I find nothing in manifoldcf.log.

BTW: PostgreSQL is not configured. I will start using Derby until it 
runs stable on Resin.

E

-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Ok, thanks for the update!
Karl

On Mon, Mar 7, 2011 at 11:21 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> It seems that the reason why the CSS file is not processed correctly is due
> to a bug in Resin:
> http://bugs.caucho.com/view.php?id=4180
>
> It's caused by the MacRoman char set which prevents Firefox to read the CSS
> file. This bug is fixed in the latest version of Resin, 4.0.15, which was
> released for a month ago.
>
> Erlend
>
> On 07.03.11 12.13, Karl Wright wrote:
>>
>> Hi Erlend,
>>
>> I'd originally proposed 4/15 for 0.2 release.  And yes, please open a
>> new ticket if there are problems running under Resin; it would be
>> great to figure out what the issue was.  I'll triage this problem is
>> required for 0.2 release.
>>
>> We do have to get the release candidate together by 3/22 or so if
>> we're going to make that proposed release schedule, because we need to
>> get people to sign off, and that takes time.  So I hope you'd be able
>> to do a little research on it between now and then.
>>
>> Thanks,
>> Karl
>>
>> On Mon, Mar 7, 2011 at 5:24 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> I'm sorry about the late reply, I have been out of office.
>>>
>>> The reason why it didn't work is because I didn't read the documentation
>>> properly. I simply forgot to run the following command after I started
>>> Resin:
>>> ./processes/script/executecommand.sh
>>> org.apache.manifoldcf.agents.AgentRun
>>>
>>> I thought it was sufficient to start my application server.
>>>
>>> And I'm sorry about the import expansions in the Java classes. I think I
>>> need a separate version of Eclipse for the Apache projects without all
>>> the
>>> settings I have for the other Java projects I'm working with at the
>>> university. It will not happen again.
>>>
>>> I think I will create another ticket if you don't mind. It can be
>>> discussed
>>> whether it is a bug, but the ManifoldCF pages cannot be displayed
>>> properly
>>> by Firefox if the application is deployed on Resin. Opera and Safari
>>> works.
>>> I have mentioned this before, and the problem is that the CSS file is
>>> probably not loaded. This is the only problem I have with ManifoldCF and
>>> Resin.
>>>
>>> BTW, do you think the 0.2-incubating version will be released before June
>>> this year?
>>>
>>> And thanks for the quick reply and for resolving this issue.
>>>
>>> Erlend
>>>
>>>
>>> On 04.03.11 16.32, Karl Wright wrote:
>>>>
>>>> I ran the patch against the standard postgresql tests, which do a
>>>> number of file-system crawls, and all was well.
>>>> So I am still very curious as to why you are seeing the hang.  Let me
>>>> know what I can do to help chase this down.
>>>>
>>>> Thanks!
>>>> Karl
>>>>
>>>> On Fri, Mar 4, 2011 at 9:40 AM, Karl Wright<da...@gmail.com>
>>>>  wrote:
>>>>>
>>>>> Yes, please submit the patch.
>>>>>
>>>>> Can you give me more details of the start up issue?  I'd like to try
>>>>> it here; are these pages I can get at?
>>>>>
>>>>> The crawler leaves the "starting up" state when the seeds have been
>>>>> added to the queue.  The fact that this seeding process does not seem
>>>>> to be completing means that something is pretty badly broken
>>>>> somewhere. For the web connector (which from your history I am
>>>>> presuming you are using), the seeding phase doesn't do much else than
>>>>> write records to the database.
>>>>>
>>>>> I think you may be misinterpreting how to use the Simple History
>>>>> report.  You are supposed to select what activities you want to see
>>>>> from the pulldown in the upper right.  By default, all of them are
>>>>> selected, and the initial report displayed is based on that.  If it is
>>>>> empty (that is, you see column headers but nothing else), then NO
>>>>> activity at all has occurred in the time window given for that
>>>>> connection.
>>>>>
>>>>> Thanks,
>>>>> Karl
>>>>>
>>>>> On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> Maybe I should create the PostgreSQL patch without any more testing. I
>>>>>> have
>>>>>> some problems to run the jobs inside the application, but I doubt the
>>>>>> reason
>>>>>> is my PostgreSQL code changes since the same happens with Apache
>>>>>> Derby.
>>>>>> The
>>>>>> status of the job says "Starting up" and nothing shows up in the
>>>>>> manifoldcf.log nor in the history logs available inside the
>>>>>> application.
>>>>>>
>>>>>> If I select "Simple History", the list is empty, but in the activities
>>>>>> list
>>>>>> (in the upper right corner) I see that "robots parse" is the last
>>>>>> activity
>>>>>> run.
>>>>>>
>>>>>> This is not something Resin specific since the same happens under
>>>>>> Tomcat.
>>>>>>
>>>>>> My contribution is clean and I have only made a few changes in the
>>>>>> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
>>>>>> Database super class.
>>>>>>
>>>>>> I haven't run "ant test-pg" since it seems to need a local Postgresql
>>>>>> server
>>>>>> running. But "ant test" reports no errors.
>>>>>>
>>>>>> I have done an svn up before I created the patch in order to include
>>>>>> the
>>>>>> latest changes from trunk.
>>>>>>
>>>>>> The documentation needs to be updated and I can suggest what to add
>>>>>> later.
>>>>>>
>>>>>> BTW, it should be interesting to know what causes the problems
>>>>>> described
>>>>>> above since I really want to run my jobs.
>>>>>>
>>>>>> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even
>>>>>> if
>>>>>> I'm
>>>>>> logging in, so I had to turn of formatting in Eclipse in order to keep
>>>>>> the
>>>>>> existing formatting (I get the following error when I click on the
>>>>>> link
>>>>>> to
>>>>>> the xml file: "You are not allowed to do AttachFile on this page.")
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>>
>>>>>> I don't think it is something PG specific either,
>>>>>>
>>>>>> On 25.02.11 16.15, Karl Wright wrote:
>>>>>>>
>>>>>>> The DBCreate command requires privs necessary to create the database
>>>>>>> instance in the first place.  It uses the admin credentials that you
>>>>>>> supply to it as command args.  These are NOT the same credentials
>>>>>>> used
>>>>>>> to access the database instance, after it has been created.
>>>>>>>
>>>>>>> Thus, you can skip the DBCreate step if you have already created the
>>>>>>> database by hand.
>>>>>>>
>>>>>>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>>>>>>> Java trust store.  Since there is only one database active for any
>>>>>>> installation, it is perfectly OK for the global trust store to be
>>>>>>> used
>>>>>>> for this purpose.  It will, however, probably require the addition of
>>>>>>> some documentation as to how to set it up properly on various
>>>>>>> platforms.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Karl
>>>>>>>
>>>>>>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> I think we need another parameter as well:
>>>>>>>> org.apache.manifoldcf.postgresql.port
>>>>>>>>
>>>>>>>> I was going to test my code today, but figured out that the
>>>>>>>> following
>>>>>>>> command class needs admin privileges to the PostgreSQL database,
>>>>>>>> which
>>>>>>>> I
>>>>>>>> don't and should never have:
>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>>
>>>>>>>> So I skipped that part. Anyway, the DBCreate method is also supposed
>>>>>>>> to
>>>>>>>> work
>>>>>>>> with my contribution if I had the sufficient privileges.
>>>>>>>>
>>>>>>>> I went on by using the Install method class instead, but ended up
>>>>>>>> with
>>>>>>>> a
>>>>>>>> Handshake exception:
>>>>>>>> javax.net.ssl.SSLHandshakeException:
>>>>>>>> sun.security.validator.ValidatorException: PKIX path building
>>>>>>>> failed:
>>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable
>>>>>>>> to
>>>>>>>> find
>>>>>>>> valid certification path to requested target.
>>>>>>>>
>>>>>>>> After I added our PG server's certificate to my local keystore,
>>>>>>>> things
>>>>>>>> began
>>>>>>>> to work as it should:
>>>>>>>> ->      Agent tables installed
>>>>>>>>
>>>>>>>> To be continued. I have a lot of things to do at the moment, and I
>>>>>>>> cannot
>>>>>>>> test my contribution properly yet since I'm at home with a slower
>>>>>>>> bandwidth.
>>>>>>>>
>>>>>>>> To be continued. :)
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>>
>>>>>>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> Sure, feel free to create a ticket and try to come up with a patch.
>>>>>>>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>>>>>>>> communication, I would make those parameters local to the
>>>>>>>>> postgresql
>>>>>>>>> implementation.  So the parameters would be something like:
>>>>>>>>>
>>>>>>>>> org.apache.manifoldcf.postgresql.hostname
>>>>>>>>> org.apache.manifoldcf.postgresql.ssl
>>>>>>>>>
>>>>>>>>> or some such.  Then, you'd want to load the necessary parameters in
>>>>>>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>>>>>>>> since that's not officially supported at this time.)
>>>>>>>>>
>>>>>>>>> The only problem I see is that the constructor architecture of
>>>>>>>>> Database.java may make it a challenge to load the appropriate
>>>>>>>>> parameters in the DBInterfacePostgreSQL constructor.  But I think
>>>>>>>>> you
>>>>>>>>> could make it work, and if not, we should probably fix this anyway
>>>>>>>>> to
>>>>>>>>> make it easier.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> You're right! I didn't read the command section in the
>>>>>>>>>> documentation
>>>>>>>>>> properly. ManifoldCF does also run on Resin now. The only problem
>>>>>>>>>> with
>>>>>>>>>> Resin
>>>>>>>>>> is that the necessary resources, such as CSS, are not loaded. All
>>>>>>>>>> the
>>>>>>>>>> other
>>>>>>>>>> errors have disappeared, which means that it should be possible to
>>>>>>>>>> run
>>>>>>>>>> the
>>>>>>>>>> application on Resin.
>>>>>>>>>>
>>>>>>>>>> Do you think I should create a ticket about running an external
>>>>>>>>>> PostgreSQL
>>>>>>>>>> database as well? I think I know which classes to modify. I see
>>>>>>>>>> there
>>>>>>>>>> are
>>>>>>>>>> classes for MySQL as well, so perhaps the issue should be
>>>>>>>>>> generally,
>>>>>>>>>> i.e.
>>>>>>>>>> not PG specific.
>>>>>>>>>>
>>>>>>>>>> I was thinking about extending the property list like this:
>>>>>>>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>>>>>>>> localhost)
>>>>>>>>>> org.apache.manifoldcf.database.ssl (not required, defaults to
>>>>>>>>>> false)
>>>>>>>>>>
>>>>>>>>>> For PostgreSQL I think it should be quite easy to implement this,
>>>>>>>>>> but
>>>>>>>>>> I'm
>>>>>>>>>> not sure when it comes to MySQL. Haven't touched that database
>>>>>>>>>> system
>>>>>>>>>> for
>>>>>>>>>> years.
>>>>>>>>>>
>>>>>>>>>> Erlend
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>>>>>>> installation in how-to-build-and-deploy.html?
>>>>>>>>>>> You will need to create the database (using DBCreate), and
>>>>>>>>>>> install
>>>>>>>>>>> the
>>>>>>>>>>> schema (using Install), and register the crawler agent (using
>>>>>>>>>>> Register).  You will also need to create the properties.xml and
>>>>>>>>>>> logging.ini files by hand.
>>>>>>>>>>>
>>>>>>>>>>> Let me know if this helps.
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>  wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I guess the problem is that the database does not exist. When I
>>>>>>>>>>>> installed
>>>>>>>>>>>> it
>>>>>>>>>>>> on Tomcat, I got an error message telling me that the schema
>>>>>>>>>>>> TEST
>>>>>>>>>>>> does
>>>>>>>>>>>> not
>>>>>>>>>>>> exist. I tried to build the database with the username and
>>>>>>>>>>>> password
>>>>>>>>>>>> "test"
>>>>>>>>>>>> by the following command:
>>>>>>>>>>>> ./processes/script/executecommand.sh
>>>>>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>>>>>> test test
>>>>>>>>>>>>
>>>>>>>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>>>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>>>>>>>>> value="test"/>
>>>>>>>>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>>>>>>>>> value="test"/>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm using the Derby database since I cannot see how I could
>>>>>>>>>>>> contact
>>>>>>>>>>>> an
>>>>>>>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>>>>>>>> database
>>>>>>>>>>>> systems the university supports. I guess there should be an
>>>>>>>>>>>> option
>>>>>>>>>>>> for
>>>>>>>>>>>> configuring external hosts and optionally turn on SSL
>>>>>>>>>>>> encryption.
>>>>>>>>>>>> Maybe
>>>>>>>>>>>> another ticket to create.
>>>>>>>>>>>>
>>>>>>>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then
>>>>>>>>>>>> deploy
>>>>>>>>>>>> it
>>>>>>>>>>>> on
>>>>>>>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>>>>>>>
>>>>>>>>>>>> Erlend
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sure, if you want, create a ticket, and see if you can figure
>>>>>>>>>>>>> out
>>>>>>>>>>>>> what
>>>>>>>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>>>>>>>> stylesheet, actually.  The "redirection" error sound like the
>>>>>>>>>>>>> crawler
>>>>>>>>>>>>> UI is trying to present an error and resin is handling one of
>>>>>>>>>>>>> the
>>>>>>>>>>>>> java
>>>>>>>>>>>>> standard tags in a non-standard way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Resin is de facto application server at the university, but we
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>> Tomcat.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>>>>>>>> message
>>>>>>>>>>>>>> comes
>>>>>>>>>>>>>> from Resin, probably caused by a redirect after the http
>>>>>>>>>>>>>> header
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> written.
>>>>>>>>>>>>>> I can notify you, eventually create a patch, if I manage to
>>>>>>>>>>>>>> run
>>>>>>>>>>>>>> ManifoldCF
>>>>>>>>>>>>>> on Resin.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Erlend
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
>>>>>>>>>>>>>>>  Clearly
>>>>>>>>>>>>>>> Resin as an application server is having difficulty doing
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>> basic stuff.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> FWIW, when we first started the Jetty development, I had to
>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it
>>>>>>>>>>>>>>> work.
>>>>>>>>>>>>>>>  It
>>>>>>>>>>>>>>> just worked.  Your experiences with Resin are making me
>>>>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have placed the three war files into Resin's webapps
>>>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <system-property
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>>>>>>> />
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have also created my own home directory for ManifoldCF and
>>>>>>>>>>>>>>>> copied
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> necessary files from the example directory. When I start
>>>>>>>>>>>>>>>> Resin
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page
>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>> ugly
>>>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> click
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> link, e.g. "List Output Connections", I get the following
>>>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing
>>>>>>>>>>>>>>>> HTTP
>>>>>>>>>>>>>>>> headers
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby
>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> runs
>>>>>>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47)
>>>>>>>>>>>>>>>> 91380968,
>>>>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47)
>>>>>>>>>>>>>> 91380968,
>>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>>> VIP:
>>>>>>>>>>>> 31050
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>> VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

It seems that the reason why the CSS file is not processed correctly is 
due to a bug in Resin:
http://bugs.caucho.com/view.php?id=4180

It's caused by the MacRoman char set which prevents Firefox to read the 
CSS file. This bug is fixed in the latest version of Resin, 4.0.15, 
which was released for a month ago.

Erlend

On 07.03.11 12.13, Karl Wright wrote:
> Hi Erlend,
>
> I'd originally proposed 4/15 for 0.2 release.  And yes, please open a
> new ticket if there are problems running under Resin; it would be
> great to figure out what the issue was.  I'll triage this problem is
> required for 0.2 release.
>
> We do have to get the release candidate together by 3/22 or so if
> we're going to make that proposed release schedule, because we need to
> get people to sign off, and that takes time.  So I hope you'd be able
> to do a little research on it between now and then.
>
> Thanks,
> Karl
>
> On Mon, Mar 7, 2011 at 5:24 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> I'm sorry about the late reply, I have been out of office.
>>
>> The reason why it didn't work is because I didn't read the documentation
>> properly. I simply forgot to run the following command after I started
>> Resin:
>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.AgentRun
>>
>> I thought it was sufficient to start my application server.
>>
>> And I'm sorry about the import expansions in the Java classes. I think I
>> need a separate version of Eclipse for the Apache projects without all the
>> settings I have for the other Java projects I'm working with at the
>> university. It will not happen again.
>>
>> I think I will create another ticket if you don't mind. It can be discussed
>> whether it is a bug, but the ManifoldCF pages cannot be displayed properly
>> by Firefox if the application is deployed on Resin. Opera and Safari works.
>> I have mentioned this before, and the problem is that the CSS file is
>> probably not loaded. This is the only problem I have with ManifoldCF and
>> Resin.
>>
>> BTW, do you think the 0.2-incubating version will be released before June
>> this year?
>>
>> And thanks for the quick reply and for resolving this issue.
>>
>> Erlend
>>
>>
>> On 04.03.11 16.32, Karl Wright wrote:
>>>
>>> I ran the patch against the standard postgresql tests, which do a
>>> number of file-system crawls, and all was well.
>>> So I am still very curious as to why you are seeing the hang.  Let me
>>> know what I can do to help chase this down.
>>>
>>> Thanks!
>>> Karl
>>>
>>> On Fri, Mar 4, 2011 at 9:40 AM, Karl Wright<da...@gmail.com>    wrote:
>>>>
>>>> Yes, please submit the patch.
>>>>
>>>> Can you give me more details of the start up issue?  I'd like to try
>>>> it here; are these pages I can get at?
>>>>
>>>> The crawler leaves the "starting up" state when the seeds have been
>>>> added to the queue.  The fact that this seeding process does not seem
>>>> to be completing means that something is pretty badly broken
>>>> somewhere. For the web connector (which from your history I am
>>>> presuming you are using), the seeding phase doesn't do much else than
>>>> write records to the database.
>>>>
>>>> I think you may be misinterpreting how to use the Simple History
>>>> report.  You are supposed to select what activities you want to see
>>>> from the pulldown in the upper right.  By default, all of them are
>>>> selected, and the initial report displayed is based on that.  If it is
>>>> empty (that is, you see column headers but nothing else), then NO
>>>> activity at all has occurred in the time window given for that
>>>> connection.
>>>>
>>>> Thanks,
>>>> Karl
>>>>
>>>> On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>   wrote:
>>>>>
>>>>> Maybe I should create the PostgreSQL patch without any more testing. I
>>>>> have
>>>>> some problems to run the jobs inside the application, but I doubt the
>>>>> reason
>>>>> is my PostgreSQL code changes since the same happens with Apache Derby.
>>>>> The
>>>>> status of the job says "Starting up" and nothing shows up in the
>>>>> manifoldcf.log nor in the history logs available inside the application.
>>>>>
>>>>> If I select "Simple History", the list is empty, but in the activities
>>>>> list
>>>>> (in the upper right corner) I see that "robots parse" is the last
>>>>> activity
>>>>> run.
>>>>>
>>>>> This is not something Resin specific since the same happens under
>>>>> Tomcat.
>>>>>
>>>>> My contribution is clean and I have only made a few changes in the
>>>>> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
>>>>> Database super class.
>>>>>
>>>>> I haven't run "ant test-pg" since it seems to need a local Postgresql
>>>>> server
>>>>> running. But "ant test" reports no errors.
>>>>>
>>>>> I have done an svn up before I created the patch in order to include the
>>>>> latest changes from trunk.
>>>>>
>>>>> The documentation needs to be updated and I can suggest what to add
>>>>> later.
>>>>>
>>>>> BTW, it should be interesting to know what causes the problems described
>>>>> above since I really want to run my jobs.
>>>>>
>>>>> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if
>>>>> I'm
>>>>> logging in, so I had to turn of formatting in Eclipse in order to keep
>>>>> the
>>>>> existing formatting (I get the following error when I click on the link
>>>>> to
>>>>> the xml file: "You are not allowed to do AttachFile on this page.")
>>>>>
>>>>> Erlend
>>>>>
>>>>>
>>>>>
>>>>> I don't think it is something PG specific either,
>>>>>
>>>>> On 25.02.11 16.15, Karl Wright wrote:
>>>>>>
>>>>>> The DBCreate command requires privs necessary to create the database
>>>>>> instance in the first place.  It uses the admin credentials that you
>>>>>> supply to it as command args.  These are NOT the same credentials used
>>>>>> to access the database instance, after it has been created.
>>>>>>
>>>>>> Thus, you can skip the DBCreate step if you have already created the
>>>>>> database by hand.
>>>>>>
>>>>>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>>>>>> Java trust store.  Since there is only one database active for any
>>>>>> installation, it is perfectly OK for the global trust store to be used
>>>>>> for this purpose.  It will, however, probably require the addition of
>>>>>> some documentation as to how to set it up properly on various
>>>>>> platforms.
>>>>>>
>>>>>> Thanks,
>>>>>> Karl
>>>>>>
>>>>>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend
>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>   wrote:
>>>>>>>
>>>>>>> I think we need another parameter as well:
>>>>>>> org.apache.manifoldcf.postgresql.port
>>>>>>>
>>>>>>> I was going to test my code today, but figured out that the following
>>>>>>> command class needs admin privileges to the PostgreSQL database, which
>>>>>>> I
>>>>>>> don't and should never have:
>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>
>>>>>>> So I skipped that part. Anyway, the DBCreate method is also supposed
>>>>>>> to
>>>>>>> work
>>>>>>> with my contribution if I had the sufficient privileges.
>>>>>>>
>>>>>>> I went on by using the Install method class instead, but ended up with
>>>>>>> a
>>>>>>> Handshake exception:
>>>>>>> javax.net.ssl.SSLHandshakeException:
>>>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>>>>> find
>>>>>>> valid certification path to requested target.
>>>>>>>
>>>>>>> After I added our PG server's certificate to my local keystore, things
>>>>>>> began
>>>>>>> to work as it should:
>>>>>>> ->      Agent tables installed
>>>>>>>
>>>>>>> To be continued. I have a lot of things to do at the moment, and I
>>>>>>> cannot
>>>>>>> test my contribution properly yet since I'm at home with a slower
>>>>>>> bandwidth.
>>>>>>>
>>>>>>> To be continued. :)
>>>>>>>
>>>>>>> Erlend
>>>>>>>
>>>>>>>
>>>>>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>>>>>
>>>>>>>> Sure, feel free to create a ticket and try to come up with a patch.
>>>>>>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>>>>>>> communication, I would make those parameters local to the postgresql
>>>>>>>> implementation.  So the parameters would be something like:
>>>>>>>>
>>>>>>>> org.apache.manifoldcf.postgresql.hostname
>>>>>>>> org.apache.manifoldcf.postgresql.ssl
>>>>>>>>
>>>>>>>> or some such.  Then, you'd want to load the necessary parameters in
>>>>>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>>>>>>> since that's not officially supported at this time.)
>>>>>>>>
>>>>>>>> The only problem I see is that the constructor architecture of
>>>>>>>> Database.java may make it a challenge to load the appropriate
>>>>>>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>>>>>>> could make it work, and if not, we should probably fix this anyway to
>>>>>>>> make it easier.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Karl
>>>>>>>>
>>>>>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend
>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>   wrote:
>>>>>>>>>
>>>>>>>>> You're right! I didn't read the command section in the documentation
>>>>>>>>> properly. ManifoldCF does also run on Resin now. The only problem
>>>>>>>>> with
>>>>>>>>> Resin
>>>>>>>>> is that the necessary resources, such as CSS, are not loaded. All
>>>>>>>>> the
>>>>>>>>> other
>>>>>>>>> errors have disappeared, which means that it should be possible to
>>>>>>>>> run
>>>>>>>>> the
>>>>>>>>> application on Resin.
>>>>>>>>>
>>>>>>>>> Do you think I should create a ticket about running an external
>>>>>>>>> PostgreSQL
>>>>>>>>> database as well? I think I know which classes to modify. I see
>>>>>>>>> there
>>>>>>>>> are
>>>>>>>>> classes for MySQL as well, so perhaps the issue should be generally,
>>>>>>>>> i.e.
>>>>>>>>> not PG specific.
>>>>>>>>>
>>>>>>>>> I was thinking about extending the property list like this:
>>>>>>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>>>>>>> localhost)
>>>>>>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>>>>>>
>>>>>>>>> For PostgreSQL I think it should be quite easy to implement this,
>>>>>>>>> but
>>>>>>>>> I'm
>>>>>>>>> not sure when it comes to MySQL. Haven't touched that database
>>>>>>>>> system
>>>>>>>>> for
>>>>>>>>> years.
>>>>>>>>>
>>>>>>>>> Erlend
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>>>>>
>>>>>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>>>>>> installation in how-to-build-and-deploy.html?
>>>>>>>>>> You will need to create the database (using DBCreate), and install
>>>>>>>>>> the
>>>>>>>>>> schema (using Install), and register the crawler agent (using
>>>>>>>>>> Register).  You will also need to create the properties.xml and
>>>>>>>>>> logging.ini files by hand.
>>>>>>>>>>
>>>>>>>>>> Let me know if this helps.
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>   wrote:
>>>>>>>>>>>
>>>>>>>>>>> I guess the problem is that the database does not exist. When I
>>>>>>>>>>> installed
>>>>>>>>>>> it
>>>>>>>>>>> on Tomcat, I got an error message telling me that the schema TEST
>>>>>>>>>>> does
>>>>>>>>>>> not
>>>>>>>>>>> exist. I tried to build the database with the username and
>>>>>>>>>>> password
>>>>>>>>>>> "test"
>>>>>>>>>>> by the following command:
>>>>>>>>>>> ./processes/script/executecommand.sh
>>>>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>>>>> test test
>>>>>>>>>>>
>>>>>>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>>>>>>   <property name="org.apache.manifoldcf.database.username"
>>>>>>>>>>> value="test"/>
>>>>>>>>>>>   <property name="org.apache.manifoldcf.database.password"
>>>>>>>>>>> value="test"/>
>>>>>>>>>>>
>>>>>>>>>>> I'm using the Derby database since I cannot see how I could
>>>>>>>>>>> contact
>>>>>>>>>>> an
>>>>>>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>>>>>>> database
>>>>>>>>>>> systems the university supports. I guess there should be an option
>>>>>>>>>>> for
>>>>>>>>>>> configuring external hosts and optionally turn on SSL encryption.
>>>>>>>>>>> Maybe
>>>>>>>>>>> another ticket to create.
>>>>>>>>>>>
>>>>>>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then
>>>>>>>>>>> deploy
>>>>>>>>>>> it
>>>>>>>>>>> on
>>>>>>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>>>>>>
>>>>>>>>>>> Erlend
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Sure, if you want, create a ticket, and see if you can figure out
>>>>>>>>>>>> what
>>>>>>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>>>>>>> stylesheet, actually.  The "redirection" error sound like the
>>>>>>>>>>>> crawler
>>>>>>>>>>>> UI is trying to present an error and resin is handling one of the
>>>>>>>>>>>> java
>>>>>>>>>>>> standard tags in a non-standard way.
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>>   wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Resin is de facto application server at the university, but we
>>>>>>>>>>>>> also
>>>>>>>>>>>>> support
>>>>>>>>>>>>> Tomcat.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>>>>>>> message
>>>>>>>>>>>>> comes
>>>>>>>>>>>>> from Resin, probably caused by a redirect after the http header
>>>>>>>>>>>>> is
>>>>>>>>>>>>> written.
>>>>>>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>>>>>>> ManifoldCF
>>>>>>>>>>>>> on Resin.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erlend
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
>>>>>>>>>>>>>>   Clearly
>>>>>>>>>>>>>> Resin as an application server is having difficulty doing some
>>>>>>>>>>>>>> very
>>>>>>>>>>>>>> basic stuff.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it
>>>>>>>>>>>>>> work.
>>>>>>>>>>>>>>   It
>>>>>>>>>>>>>> just worked.  Your experiences with Resin are making me wonder
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>>>>   wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have placed the three war files into Resin's webapps
>>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <system-property
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>>>>>> />
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have also created my own home directory for ManifoldCF and
>>>>>>>>>>>>>>> copied
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> necessary files from the example directory. When I start Resin
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page
>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>> ugly
>>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> click
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> link, e.g. "List Output Connections", I get the following
>>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing
>>>>>>>>>>>>>>> HTTP
>>>>>>>>>>>>>>> headers
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby
>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> runs
>>>>>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47)
>>>>>>>>>>>>>>> 91380968,
>>>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>> University of Oslo
>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>> VIP:
>>>>>>>>>>> 31050
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Erlend Garåsen
>>>>>>>>> Center for Information Technology Services
>>>>>>>>> University of Oslo
>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>> VIP:
>>>>>>>>> 31050
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Hi Erlend,

I'd originally proposed 4/15 for 0.2 release.  And yes, please open a
new ticket if there are problems running under Resin; it would be
great to figure out what the issue was.  I'll triage this problem is
required for 0.2 release.

We do have to get the release candidate together by 3/22 or so if
we're going to make that proposed release schedule, because we need to
get people to sign off, and that takes time.  So I hope you'd be able
to do a little research on it between now and then.

Thanks,
Karl

On Mon, Mar 7, 2011 at 5:24 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I'm sorry about the late reply, I have been out of office.
>
> The reason why it didn't work is because I didn't read the documentation
> properly. I simply forgot to run the following command after I started
> Resin:
> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.AgentRun
>
> I thought it was sufficient to start my application server.
>
> And I'm sorry about the import expansions in the Java classes. I think I
> need a separate version of Eclipse for the Apache projects without all the
> settings I have for the other Java projects I'm working with at the
> university. It will not happen again.
>
> I think I will create another ticket if you don't mind. It can be discussed
> whether it is a bug, but the ManifoldCF pages cannot be displayed properly
> by Firefox if the application is deployed on Resin. Opera and Safari works.
> I have mentioned this before, and the problem is that the CSS file is
> probably not loaded. This is the only problem I have with ManifoldCF and
> Resin.
>
> BTW, do you think the 0.2-incubating version will be released before June
> this year?
>
> And thanks for the quick reply and for resolving this issue.
>
> Erlend
>
>
> On 04.03.11 16.32, Karl Wright wrote:
>>
>> I ran the patch against the standard postgresql tests, which do a
>> number of file-system crawls, and all was well.
>> So I am still very curious as to why you are seeing the hang.  Let me
>> know what I can do to help chase this down.
>>
>> Thanks!
>> Karl
>>
>> On Fri, Mar 4, 2011 at 9:40 AM, Karl Wright<da...@gmail.com>  wrote:
>>>
>>> Yes, please submit the patch.
>>>
>>> Can you give me more details of the start up issue?  I'd like to try
>>> it here; are these pages I can get at?
>>>
>>> The crawler leaves the "starting up" state when the seeds have been
>>> added to the queue.  The fact that this seeding process does not seem
>>> to be completing means that something is pretty badly broken
>>> somewhere. For the web connector (which from your history I am
>>> presuming you are using), the seeding phase doesn't do much else than
>>> write records to the database.
>>>
>>> I think you may be misinterpreting how to use the Simple History
>>> report.  You are supposed to select what activities you want to see
>>> from the pulldown in the upper right.  By default, all of them are
>>> selected, and the initial report displayed is based on that.  If it is
>>> empty (that is, you see column headers but nothing else), then NO
>>> activity at all has occurred in the time window given for that
>>> connection.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>  wrote:
>>>>
>>>> Maybe I should create the PostgreSQL patch without any more testing. I
>>>> have
>>>> some problems to run the jobs inside the application, but I doubt the
>>>> reason
>>>> is my PostgreSQL code changes since the same happens with Apache Derby.
>>>> The
>>>> status of the job says "Starting up" and nothing shows up in the
>>>> manifoldcf.log nor in the history logs available inside the application.
>>>>
>>>> If I select "Simple History", the list is empty, but in the activities
>>>> list
>>>> (in the upper right corner) I see that "robots parse" is the last
>>>> activity
>>>> run.
>>>>
>>>> This is not something Resin specific since the same happens under
>>>> Tomcat.
>>>>
>>>> My contribution is clean and I have only made a few changes in the
>>>> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
>>>> Database super class.
>>>>
>>>> I haven't run "ant test-pg" since it seems to need a local Postgresql
>>>> server
>>>> running. But "ant test" reports no errors.
>>>>
>>>> I have done an svn up before I created the patch in order to include the
>>>> latest changes from trunk.
>>>>
>>>> The documentation needs to be updated and I can suggest what to add
>>>> later.
>>>>
>>>> BTW, it should be interesting to know what causes the problems described
>>>> above since I really want to run my jobs.
>>>>
>>>> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if
>>>> I'm
>>>> logging in, so I had to turn of formatting in Eclipse in order to keep
>>>> the
>>>> existing formatting (I get the following error when I click on the link
>>>> to
>>>> the xml file: "You are not allowed to do AttachFile on this page.")
>>>>
>>>> Erlend
>>>>
>>>>
>>>>
>>>> I don't think it is something PG specific either,
>>>>
>>>> On 25.02.11 16.15, Karl Wright wrote:
>>>>>
>>>>> The DBCreate command requires privs necessary to create the database
>>>>> instance in the first place.  It uses the admin credentials that you
>>>>> supply to it as command args.  These are NOT the same credentials used
>>>>> to access the database instance, after it has been created.
>>>>>
>>>>> Thus, you can skip the DBCreate step if you have already created the
>>>>> database by hand.
>>>>>
>>>>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>>>>> Java trust store.  Since there is only one database active for any
>>>>> installation, it is perfectly OK for the global trust store to be used
>>>>> for this purpose.  It will, however, probably require the addition of
>>>>> some documentation as to how to set it up properly on various
>>>>> platforms.
>>>>>
>>>>> Thanks,
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend
>>>>> Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> I think we need another parameter as well:
>>>>>> org.apache.manifoldcf.postgresql.port
>>>>>>
>>>>>> I was going to test my code today, but figured out that the following
>>>>>> command class needs admin privileges to the PostgreSQL database, which
>>>>>> I
>>>>>> don't and should never have:
>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>
>>>>>> So I skipped that part. Anyway, the DBCreate method is also supposed
>>>>>> to
>>>>>> work
>>>>>> with my contribution if I had the sufficient privileges.
>>>>>>
>>>>>> I went on by using the Install method class instead, but ended up with
>>>>>> a
>>>>>> Handshake exception:
>>>>>> javax.net.ssl.SSLHandshakeException:
>>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>>>> find
>>>>>> valid certification path to requested target.
>>>>>>
>>>>>> After I added our PG server's certificate to my local keystore, things
>>>>>> began
>>>>>> to work as it should:
>>>>>> ->    Agent tables installed
>>>>>>
>>>>>> To be continued. I have a lot of things to do at the moment, and I
>>>>>> cannot
>>>>>> test my contribution properly yet since I'm at home with a slower
>>>>>> bandwidth.
>>>>>>
>>>>>> To be continued. :)
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, feel free to create a ticket and try to come up with a patch.
>>>>>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>>>>>> communication, I would make those parameters local to the postgresql
>>>>>>> implementation.  So the parameters would be something like:
>>>>>>>
>>>>>>> org.apache.manifoldcf.postgresql.hostname
>>>>>>> org.apache.manifoldcf.postgresql.ssl
>>>>>>>
>>>>>>> or some such.  Then, you'd want to load the necessary parameters in
>>>>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>>>>>> since that's not officially supported at this time.)
>>>>>>>
>>>>>>> The only problem I see is that the constructor architecture of
>>>>>>> Database.java may make it a challenge to load the appropriate
>>>>>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>>>>>> could make it work, and if not, we should probably fix this anyway to
>>>>>>> make it easier.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Karl
>>>>>>>
>>>>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> You're right! I didn't read the command section in the documentation
>>>>>>>> properly. ManifoldCF does also run on Resin now. The only problem
>>>>>>>> with
>>>>>>>> Resin
>>>>>>>> is that the necessary resources, such as CSS, are not loaded. All
>>>>>>>> the
>>>>>>>> other
>>>>>>>> errors have disappeared, which means that it should be possible to
>>>>>>>> run
>>>>>>>> the
>>>>>>>> application on Resin.
>>>>>>>>
>>>>>>>> Do you think I should create a ticket about running an external
>>>>>>>> PostgreSQL
>>>>>>>> database as well? I think I know which classes to modify. I see
>>>>>>>> there
>>>>>>>> are
>>>>>>>> classes for MySQL as well, so perhaps the issue should be generally,
>>>>>>>> i.e.
>>>>>>>> not PG specific.
>>>>>>>>
>>>>>>>> I was thinking about extending the property list like this:
>>>>>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>>>>>> localhost)
>>>>>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>>>>>
>>>>>>>> For PostgreSQL I think it should be quite easy to implement this,
>>>>>>>> but
>>>>>>>> I'm
>>>>>>>> not sure when it comes to MySQL. Haven't touched that database
>>>>>>>> system
>>>>>>>> for
>>>>>>>> years.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>>
>>>>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>>>>> installation in how-to-build-and-deploy.html?
>>>>>>>>> You will need to create the database (using DBCreate), and install
>>>>>>>>> the
>>>>>>>>> schema (using Install), and register the crawler agent (using
>>>>>>>>> Register).  You will also need to create the properties.xml and
>>>>>>>>> logging.ini files by hand.
>>>>>>>>>
>>>>>>>>> Let me know if this helps.
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> I guess the problem is that the database does not exist. When I
>>>>>>>>>> installed
>>>>>>>>>> it
>>>>>>>>>> on Tomcat, I got an error message telling me that the schema TEST
>>>>>>>>>> does
>>>>>>>>>> not
>>>>>>>>>> exist. I tried to build the database with the username and
>>>>>>>>>> password
>>>>>>>>>> "test"
>>>>>>>>>> by the following command:
>>>>>>>>>> ./processes/script/executecommand.sh
>>>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>>>> test test
>>>>>>>>>>
>>>>>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>>>>>>> value="test"/>
>>>>>>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>>>>>>> value="test"/>
>>>>>>>>>>
>>>>>>>>>> I'm using the Derby database since I cannot see how I could
>>>>>>>>>> contact
>>>>>>>>>> an
>>>>>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>>>>>> database
>>>>>>>>>> systems the university supports. I guess there should be an option
>>>>>>>>>> for
>>>>>>>>>> configuring external hosts and optionally turn on SSL encryption.
>>>>>>>>>> Maybe
>>>>>>>>>> another ticket to create.
>>>>>>>>>>
>>>>>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then
>>>>>>>>>> deploy
>>>>>>>>>> it
>>>>>>>>>> on
>>>>>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>>>>>
>>>>>>>>>> Erlend
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>>>>
>>>>>>>>>>> Sure, if you want, create a ticket, and see if you can figure out
>>>>>>>>>>> what
>>>>>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>>>>>> stylesheet, actually.  The "redirection" error sound like the
>>>>>>>>>>> crawler
>>>>>>>>>>> UI is trying to present an error and resin is handling one of the
>>>>>>>>>>> java
>>>>>>>>>>> standard tags in a non-standard way.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>  wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Resin is de facto application server at the university, but we
>>>>>>>>>>>> also
>>>>>>>>>>>> support
>>>>>>>>>>>> Tomcat.
>>>>>>>>>>>>
>>>>>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>>>>>> message
>>>>>>>>>>>> comes
>>>>>>>>>>>> from Resin, probably caused by a redirect after the http header
>>>>>>>>>>>> is
>>>>>>>>>>>> written.
>>>>>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>>>>>> ManifoldCF
>>>>>>>>>>>> on Resin.
>>>>>>>>>>>>
>>>>>>>>>>>> Erlend
>>>>>>>>>>>>
>>>>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
>>>>>>>>>>>>>  Clearly
>>>>>>>>>>>>> Resin as an application server is having difficulty doing some
>>>>>>>>>>>>> very
>>>>>>>>>>>>> basic stuff.
>>>>>>>>>>>>>
>>>>>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>>>>>> nothing
>>>>>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it
>>>>>>>>>>>>> work.
>>>>>>>>>>>>>  It
>>>>>>>>>>>>> just worked.  Your experiences with Resin are making me wonder
>>>>>>>>>>>>> if
>>>>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have placed the three war files into Resin's webapps
>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <system-property
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>>>>> />
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have also created my own home directory for ManifoldCF and
>>>>>>>>>>>>>> copied
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> necessary files from the example directory. When I start Resin
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page
>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>> ugly
>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> click
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> link, e.g. "List Output Connections", I get the following
>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing
>>>>>>>>>>>>>> HTTP
>>>>>>>>>>>>>> headers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby
>>>>>>>>>>>>>> until
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> runs
>>>>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47)
>>>>>>>>>>>>>> 91380968,
>>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>>> VIP:
>>>>>>>>>>>> 31050
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>> VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

I'm sorry about the late reply, I have been out of office.

The reason why it didn't work is because I didn't read the documentation 
properly. I simply forgot to run the following command after I started 
Resin:
./processes/script/executecommand.sh org.apache.manifoldcf.agents.AgentRun

I thought it was sufficient to start my application server.

And I'm sorry about the import expansions in the Java classes. I think I 
need a separate version of Eclipse for the Apache projects without all 
the settings I have for the other Java projects I'm working with at the 
university. It will not happen again.

I think I will create another ticket if you don't mind. It can be 
discussed whether it is a bug, but the ManifoldCF pages cannot be 
displayed properly by Firefox if the application is deployed on Resin. 
Opera and Safari works. I have mentioned this before, and the problem is 
that the CSS file is probably not loaded. This is the only problem I 
have with ManifoldCF and Resin.

BTW, do you think the 0.2-incubating version will be released before 
June this year?

And thanks for the quick reply and for resolving this issue.

Erlend


On 04.03.11 16.32, Karl Wright wrote:
> I ran the patch against the standard postgresql tests, which do a
> number of file-system crawls, and all was well.
> So I am still very curious as to why you are seeing the hang.  Let me
> know what I can do to help chase this down.
>
> Thanks!
> Karl
>
> On Fri, Mar 4, 2011 at 9:40 AM, Karl Wright<da...@gmail.com>  wrote:
>> Yes, please submit the patch.
>>
>> Can you give me more details of the start up issue?  I'd like to try
>> it here; are these pages I can get at?
>>
>> The crawler leaves the "starting up" state when the seeds have been
>> added to the queue.  The fact that this seeding process does not seem
>> to be completing means that something is pretty badly broken
>> somewhere. For the web connector (which from your history I am
>> presuming you are using), the seeding phase doesn't do much else than
>> write records to the database.
>>
>> I think you may be misinterpreting how to use the Simple History
>> report.  You are supposed to select what activities you want to see
>> from the pulldown in the upper right.  By default, all of them are
>> selected, and the initial report displayed is based on that.  If it is
>> empty (that is, you see column headers but nothing else), then NO
>> activity at all has occurred in the time window given for that
>> connection.
>>
>> Thanks,
>> Karl
>>
>> On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>>
>>> Maybe I should create the PostgreSQL patch without any more testing. I have
>>> some problems to run the jobs inside the application, but I doubt the reason
>>> is my PostgreSQL code changes since the same happens with Apache Derby. The
>>> status of the job says "Starting up" and nothing shows up in the
>>> manifoldcf.log nor in the history logs available inside the application.
>>>
>>> If I select "Simple History", the list is empty, but in the activities list
>>> (in the upper right corner) I see that "robots parse" is the last activity
>>> run.
>>>
>>> This is not something Resin specific since the same happens under Tomcat.
>>>
>>> My contribution is clean and I have only made a few changes in the
>>> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
>>> Database super class.
>>>
>>> I haven't run "ant test-pg" since it seems to need a local Postgresql server
>>> running. But "ant test" reports no errors.
>>>
>>> I have done an svn up before I created the patch in order to include the
>>> latest changes from trunk.
>>>
>>> The documentation needs to be updated and I can suggest what to add later.
>>>
>>> BTW, it should be interesting to know what causes the problems described
>>> above since I really want to run my jobs.
>>>
>>> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if I'm
>>> logging in, so I had to turn of formatting in Eclipse in order to keep the
>>> existing formatting (I get the following error when I click on the link to
>>> the xml file: "You are not allowed to do AttachFile on this page.")
>>>
>>> Erlend
>>>
>>>
>>>
>>> I don't think it is something PG specific either,
>>>
>>> On 25.02.11 16.15, Karl Wright wrote:
>>>>
>>>> The DBCreate command requires privs necessary to create the database
>>>> instance in the first place.  It uses the admin credentials that you
>>>> supply to it as command args.  These are NOT the same credentials used
>>>> to access the database instance, after it has been created.
>>>>
>>>> Thus, you can skip the DBCreate step if you have already created the
>>>> database by hand.
>>>>
>>>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>>>> Java trust store.  Since there is only one database active for any
>>>> installation, it is perfectly OK for the global trust store to be used
>>>> for this purpose.  It will, however, probably require the addition of
>>>> some documentation as to how to set it up properly on various
>>>> platforms.
>>>>
>>>> Thanks,
>>>> Karl
>>>>
>>>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>   wrote:
>>>>>
>>>>> I think we need another parameter as well:
>>>>> org.apache.manifoldcf.postgresql.port
>>>>>
>>>>> I was going to test my code today, but figured out that the following
>>>>> command class needs admin privileges to the PostgreSQL database, which I
>>>>> don't and should never have:
>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>
>>>>> So I skipped that part. Anyway, the DBCreate method is also supposed to
>>>>> work
>>>>> with my contribution if I had the sufficient privileges.
>>>>>
>>>>> I went on by using the Install method class instead, but ended up with a
>>>>> Handshake exception:
>>>>> javax.net.ssl.SSLHandshakeException:
>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>>> find
>>>>> valid certification path to requested target.
>>>>>
>>>>> After I added our PG server's certificate to my local keystore, things
>>>>> began
>>>>> to work as it should:
>>>>> ->    Agent tables installed
>>>>>
>>>>> To be continued. I have a lot of things to do at the moment, and I cannot
>>>>> test my contribution properly yet since I'm at home with a slower
>>>>> bandwidth.
>>>>>
>>>>> To be continued. :)
>>>>>
>>>>> Erlend
>>>>>
>>>>>
>>>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>>>
>>>>>> Sure, feel free to create a ticket and try to come up with a patch.
>>>>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>>>>> communication, I would make those parameters local to the postgresql
>>>>>> implementation.  So the parameters would be something like:
>>>>>>
>>>>>> org.apache.manifoldcf.postgresql.hostname
>>>>>> org.apache.manifoldcf.postgresql.ssl
>>>>>>
>>>>>> or some such.  Then, you'd want to load the necessary parameters in
>>>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>>>>> since that's not officially supported at this time.)
>>>>>>
>>>>>> The only problem I see is that the constructor architecture of
>>>>>> Database.java may make it a challenge to load the appropriate
>>>>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>>>>> could make it work, and if not, we should probably fix this anyway to
>>>>>> make it easier.
>>>>>>
>>>>>> Thanks,
>>>>>> Karl
>>>>>>
>>>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>>   wrote:
>>>>>>>
>>>>>>> You're right! I didn't read the command section in the documentation
>>>>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>>>>>> Resin
>>>>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>>>>>> other
>>>>>>> errors have disappeared, which means that it should be possible to run
>>>>>>> the
>>>>>>> application on Resin.
>>>>>>>
>>>>>>> Do you think I should create a ticket about running an external
>>>>>>> PostgreSQL
>>>>>>> database as well? I think I know which classes to modify. I see there
>>>>>>> are
>>>>>>> classes for MySQL as well, so perhaps the issue should be generally,
>>>>>>> i.e.
>>>>>>> not PG specific.
>>>>>>>
>>>>>>> I was thinking about extending the property list like this:
>>>>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>>>>> localhost)
>>>>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>>>>
>>>>>>> For PostgreSQL I think it should be quite easy to implement this, but
>>>>>>> I'm
>>>>>>> not sure when it comes to MySQL. Haven't touched that database system
>>>>>>> for
>>>>>>> years.
>>>>>>>
>>>>>>> Erlend
>>>>>>>
>>>>>>>
>>>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>>>
>>>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>>>> installation in how-to-build-and-deploy.html?
>>>>>>>> You will need to create the database (using DBCreate), and install the
>>>>>>>> schema (using Install), and register the crawler agent (using
>>>>>>>> Register).  You will also need to create the properties.xml and
>>>>>>>> logging.ini files by hand.
>>>>>>>>
>>>>>>>> Let me know if this helps.
>>>>>>>> Karl
>>>>>>>>
>>>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>   wrote:
>>>>>>>>>
>>>>>>>>> I guess the problem is that the database does not exist. When I
>>>>>>>>> installed
>>>>>>>>> it
>>>>>>>>> on Tomcat, I got an error message telling me that the schema TEST
>>>>>>>>> does
>>>>>>>>> not
>>>>>>>>> exist. I tried to build the database with the username and password
>>>>>>>>> "test"
>>>>>>>>> by the following command:
>>>>>>>>> ./processes/script/executecommand.sh
>>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>>> test test
>>>>>>>>>
>>>>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>>>>   <property name="org.apache.manifoldcf.database.username"
>>>>>>>>> value="test"/>
>>>>>>>>>   <property name="org.apache.manifoldcf.database.password"
>>>>>>>>> value="test"/>
>>>>>>>>>
>>>>>>>>> I'm using the Derby database since I cannot see how I could contact
>>>>>>>>> an
>>>>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>>>>> database
>>>>>>>>> systems the university supports. I guess there should be an option
>>>>>>>>> for
>>>>>>>>> configuring external hosts and optionally turn on SSL encryption.
>>>>>>>>> Maybe
>>>>>>>>> another ticket to create.
>>>>>>>>>
>>>>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>>>>>> it
>>>>>>>>> on
>>>>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>>>>
>>>>>>>>> Erlend
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>>>
>>>>>>>>>> Sure, if you want, create a ticket, and see if you can figure out
>>>>>>>>>> what
>>>>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>>>>> stylesheet, actually.  The "redirection" error sound like the
>>>>>>>>>> crawler
>>>>>>>>>> UI is trying to present an error and resin is handling one of the
>>>>>>>>>> java
>>>>>>>>>> standard tags in a non-standard way.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>   wrote:
>>>>>>>>>>>
>>>>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>>>>> support
>>>>>>>>>>> Tomcat.
>>>>>>>>>>>
>>>>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>>>>> message
>>>>>>>>>>> comes
>>>>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>>>>> written.
>>>>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>>>>> ManifoldCF
>>>>>>>>>>> on Resin.
>>>>>>>>>>>
>>>>>>>>>>> Erlend
>>>>>>>>>>>
>>>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
>>>>>>>>>>>>   Clearly
>>>>>>>>>>>> Resin as an application server is having difficulty doing some
>>>>>>>>>>>> very
>>>>>>>>>>>> basic stuff.
>>>>>>>>>>>>
>>>>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>>>>> nothing
>>>>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>>>>>   It
>>>>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>>   wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>>>>>> and
>>>>>>>>>>>>> set
>>>>>>>>>>>>> the
>>>>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>>>>
>>>>>>>>>>>>> <system-property
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>>>> />
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have also created my own home directory for ManifoldCF and
>>>>>>>>>>>>> copied
>>>>>>>>>>>>> the
>>>>>>>>>>>>> necessary files from the example directory. When I start Resin
>>>>>>>>>>>>> and
>>>>>>>>>>>>> goes
>>>>>>>>>>>>> to
>>>>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>>>>>> ugly
>>>>>>>>>>>>> (I
>>>>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>>>>>> click
>>>>>>>>>>>>> on
>>>>>>>>>>>>> a
>>>>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>>>>> headers
>>>>>>>>>>>>>
>>>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>>>>>> it
>>>>>>>>>>>>> runs
>>>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>>>
>>>>>>>>>>>>> E
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>> University of Oslo
>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>> VIP:
>>>>>>>>>>> 31050
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Erlend Garåsen
>>>>>>>>> Center for Information Technology Services
>>>>>>>>> University of Oslo
>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>>> 31050
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>>
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

I ran the patch against the standard postgresql tests, which do a
number of file-system crawls, and all was well.
So I am still very curious as to why you are seeing the hang.  Let me
know what I can do to help chase this down.

Thanks!
Karl

On Fri, Mar 4, 2011 at 9:40 AM, Karl Wright <da...@gmail.com> wrote:
> Yes, please submit the patch.
>
> Can you give me more details of the start up issue?  I'd like to try
> it here; are these pages I can get at?
>
> The crawler leaves the "starting up" state when the seeds have been
> added to the queue.  The fact that this seeding process does not seem
> to be completing means that something is pretty badly broken
> somewhere. For the web connector (which from your history I am
> presuming you are using), the seeding phase doesn't do much else than
> write records to the database.
>
> I think you may be misinterpreting how to use the Simple History
> report.  You are supposed to select what activities you want to see
> from the pulldown in the upper right.  By default, all of them are
> selected, and the initial report displayed is based on that.  If it is
> empty (that is, you see column headers but nothing else), then NO
> activity at all has occurred in the time window given for that
> connection.
>
> Thanks,
> Karl
>
> On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> Maybe I should create the PostgreSQL patch without any more testing. I have
>> some problems to run the jobs inside the application, but I doubt the reason
>> is my PostgreSQL code changes since the same happens with Apache Derby. The
>> status of the job says "Starting up" and nothing shows up in the
>> manifoldcf.log nor in the history logs available inside the application.
>>
>> If I select "Simple History", the list is empty, but in the activities list
>> (in the upper right corner) I see that "robots parse" is the last activity
>> run.
>>
>> This is not something Resin specific since the same happens under Tomcat.
>>
>> My contribution is clean and I have only made a few changes in the
>> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
>> Database super class.
>>
>> I haven't run "ant test-pg" since it seems to need a local Postgresql server
>> running. But "ant test" reports no errors.
>>
>> I have done an svn up before I created the patch in order to include the
>> latest changes from trunk.
>>
>> The documentation needs to be updated and I can suggest what to add later.
>>
>> BTW, it should be interesting to know what causes the problems described
>> above since I really want to run my jobs.
>>
>> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if I'm
>> logging in, so I had to turn of formatting in Eclipse in order to keep the
>> existing formatting (I get the following error when I click on the link to
>> the xml file: "You are not allowed to do AttachFile on this page.")
>>
>> Erlend
>>
>>
>>
>> I don't think it is something PG specific either,
>>
>> On 25.02.11 16.15, Karl Wright wrote:
>>>
>>> The DBCreate command requires privs necessary to create the database
>>> instance in the first place.  It uses the admin credentials that you
>>> supply to it as command args.  These are NOT the same credentials used
>>> to access the database instance, after it has been created.
>>>
>>> Thus, you can skip the DBCreate step if you have already created the
>>> database by hand.
>>>
>>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>>> Java trust store.  Since there is only one database active for any
>>> installation, it is perfectly OK for the global trust store to be used
>>> for this purpose.  It will, however, probably require the addition of
>>> some documentation as to how to set it up properly on various
>>> platforms.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend Garåsen<e....@usit.uio.no>
>>>  wrote:
>>>>
>>>> I think we need another parameter as well:
>>>> org.apache.manifoldcf.postgresql.port
>>>>
>>>> I was going to test my code today, but figured out that the following
>>>> command class needs admin privileges to the PostgreSQL database, which I
>>>> don't and should never have:
>>>> org.apache.manifoldcf.core.DBCreate
>>>>
>>>> So I skipped that part. Anyway, the DBCreate method is also supposed to
>>>> work
>>>> with my contribution if I had the sufficient privileges.
>>>>
>>>> I went on by using the Install method class instead, but ended up with a
>>>> Handshake exception:
>>>> javax.net.ssl.SSLHandshakeException:
>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>> find
>>>> valid certification path to requested target.
>>>>
>>>> After I added our PG server's certificate to my local keystore, things
>>>> began
>>>> to work as it should:
>>>> ->  Agent tables installed
>>>>
>>>> To be continued. I have a lot of things to do at the moment, and I cannot
>>>> test my contribution properly yet since I'm at home with a slower
>>>> bandwidth.
>>>>
>>>> To be continued. :)
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>>
>>>>> Sure, feel free to create a ticket and try to come up with a patch.
>>>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>>>> communication, I would make those parameters local to the postgresql
>>>>> implementation.  So the parameters would be something like:
>>>>>
>>>>> org.apache.manifoldcf.postgresql.hostname
>>>>> org.apache.manifoldcf.postgresql.ssl
>>>>>
>>>>> or some such.  Then, you'd want to load the necessary parameters in
>>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>>>> since that's not officially supported at this time.)
>>>>>
>>>>> The only problem I see is that the constructor architecture of
>>>>> Database.java may make it a challenge to load the appropriate
>>>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>>>> could make it work, and if not, we should probably fix this anyway to
>>>>> make it easier.
>>>>>
>>>>> Thanks,
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> You're right! I didn't read the command section in the documentation
>>>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>>>>> Resin
>>>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>>>>> other
>>>>>> errors have disappeared, which means that it should be possible to run
>>>>>> the
>>>>>> application on Resin.
>>>>>>
>>>>>> Do you think I should create a ticket about running an external
>>>>>> PostgreSQL
>>>>>> database as well? I think I know which classes to modify. I see there
>>>>>> are
>>>>>> classes for MySQL as well, so perhaps the issue should be generally,
>>>>>> i.e.
>>>>>> not PG specific.
>>>>>>
>>>>>> I was thinking about extending the property list like this:
>>>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>>>> localhost)
>>>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>>>
>>>>>> For PostgreSQL I think it should be quite easy to implement this, but
>>>>>> I'm
>>>>>> not sure when it comes to MySQL. Haven't touched that database system
>>>>>> for
>>>>>> years.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>>
>>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>>> installation in how-to-build-and-deploy.html?
>>>>>>> You will need to create the database (using DBCreate), and install the
>>>>>>> schema (using Install), and register the crawler agent (using
>>>>>>> Register).  You will also need to create the properties.xml and
>>>>>>> logging.ini files by hand.
>>>>>>>
>>>>>>> Let me know if this helps.
>>>>>>> Karl
>>>>>>>
>>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> I guess the problem is that the database does not exist. When I
>>>>>>>> installed
>>>>>>>> it
>>>>>>>> on Tomcat, I got an error message telling me that the schema TEST
>>>>>>>> does
>>>>>>>> not
>>>>>>>> exist. I tried to build the database with the username and password
>>>>>>>> "test"
>>>>>>>> by the following command:
>>>>>>>> ./processes/script/executecommand.sh
>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>> test test
>>>>>>>>
>>>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>>>>> value="test"/>
>>>>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>>>>> value="test"/>
>>>>>>>>
>>>>>>>> I'm using the Derby database since I cannot see how I could contact
>>>>>>>> an
>>>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>>>> database
>>>>>>>> systems the university supports. I guess there should be an option
>>>>>>>> for
>>>>>>>> configuring external hosts and optionally turn on SSL encryption.
>>>>>>>> Maybe
>>>>>>>> another ticket to create.
>>>>>>>>
>>>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>>>>> it
>>>>>>>> on
>>>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> Sure, if you want, create a ticket, and see if you can figure out
>>>>>>>>> what
>>>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>>>> stylesheet, actually.  The "redirection" error sound like the
>>>>>>>>> crawler
>>>>>>>>> UI is trying to present an error and resin is handling one of the
>>>>>>>>> java
>>>>>>>>> standard tags in a non-standard way.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>>>> support
>>>>>>>>>> Tomcat.
>>>>>>>>>>
>>>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>>>> message
>>>>>>>>>> comes
>>>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>>>> written.
>>>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>>>> ManifoldCF
>>>>>>>>>> on Resin.
>>>>>>>>>>
>>>>>>>>>> Erlend
>>>>>>>>>>
>>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>>
>>>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
>>>>>>>>>>>  Clearly
>>>>>>>>>>> Resin as an application server is having difficulty doing some
>>>>>>>>>>> very
>>>>>>>>>>> basic stuff.
>>>>>>>>>>>
>>>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>>>> nothing
>>>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>>>>  It
>>>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>>  wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>>>>> and
>>>>>>>>>>>> set
>>>>>>>>>>>> the
>>>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>>>
>>>>>>>>>>>> <system-property
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>>> />
>>>>>>>>>>>>
>>>>>>>>>>>> I have also created my own home directory for ManifoldCF and
>>>>>>>>>>>> copied
>>>>>>>>>>>> the
>>>>>>>>>>>> necessary files from the example directory. When I start Resin
>>>>>>>>>>>> and
>>>>>>>>>>>> goes
>>>>>>>>>>>> to
>>>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>>>>> ugly
>>>>>>>>>>>> (I
>>>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>>>>> click
>>>>>>>>>>>> on
>>>>>>>>>>>> a
>>>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>>>> headers
>>>>>>>>>>>>
>>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>>
>>>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>>>>> it
>>>>>>>>>>>> runs
>>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>>
>>>>>>>>>>>> E
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>>> VIP:
>>>>>>>>>>>> 31050
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>
>

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Yes, please submit the patch.

Can you give me more details of the start up issue?  I'd like to try
it here; are these pages I can get at?

The crawler leaves the "starting up" state when the seeds have been
added to the queue.  The fact that this seeding process does not seem
to be completing means that something is pretty badly broken
somewhere. For the web connector (which from your history I am
presuming you are using), the seeding phase doesn't do much else than
write records to the database.

I think you may be misinterpreting how to use the Simple History
report.  You are supposed to select what activities you want to see
from the pulldown in the upper right.  By default, all of them are
selected, and the initial report displayed is based on that.  If it is
empty (that is, you see column headers but nothing else), then NO
activity at all has occurred in the time window given for that
connection.

Thanks,
Karl

On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> Maybe I should create the PostgreSQL patch without any more testing. I have
> some problems to run the jobs inside the application, but I doubt the reason
> is my PostgreSQL code changes since the same happens with Apache Derby. The
> status of the job says "Starting up" and nothing shows up in the
> manifoldcf.log nor in the history logs available inside the application.
>
> If I select "Simple History", the list is empty, but in the activities list
> (in the upper right corner) I see that "robots parse" is the last activity
> run.
>
> This is not something Resin specific since the same happens under Tomcat.
>
> My contribution is clean and I have only made a few changes in the
> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
> Database super class.
>
> I haven't run "ant test-pg" since it seems to need a local Postgresql server
> running. But "ant test" reports no errors.
>
> I have done an svn up before I created the patch in order to include the
> latest changes from trunk.
>
> The documentation needs to be updated and I can suggest what to add later.
>
> BTW, it should be interesting to know what causes the problems described
> above since I really want to run my jobs.
>
> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if I'm
> logging in, so I had to turn of formatting in Eclipse in order to keep the
> existing formatting (I get the following error when I click on the link to
> the xml file: "You are not allowed to do AttachFile on this page.")
>
> Erlend
>
>
>
> I don't think it is something PG specific either,
>
> On 25.02.11 16.15, Karl Wright wrote:
>>
>> The DBCreate command requires privs necessary to create the database
>> instance in the first place.  It uses the admin credentials that you
>> supply to it as command args.  These are NOT the same credentials used
>> to access the database instance, after it has been created.
>>
>> Thus, you can skip the DBCreate step if you have already created the
>> database by hand.
>>
>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>> Java trust store.  Since there is only one database active for any
>> installation, it is perfectly OK for the global trust store to be used
>> for this purpose.  It will, however, probably require the addition of
>> some documentation as to how to set it up properly on various
>> platforms.
>>
>> Thanks,
>> Karl
>>
>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> I think we need another parameter as well:
>>> org.apache.manifoldcf.postgresql.port
>>>
>>> I was going to test my code today, but figured out that the following
>>> command class needs admin privileges to the PostgreSQL database, which I
>>> don't and should never have:
>>> org.apache.manifoldcf.core.DBCreate
>>>
>>> So I skipped that part. Anyway, the DBCreate method is also supposed to
>>> work
>>> with my contribution if I had the sufficient privileges.
>>>
>>> I went on by using the Install method class instead, but ended up with a
>>> Handshake exception:
>>> javax.net.ssl.SSLHandshakeException:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find
>>> valid certification path to requested target.
>>>
>>> After I added our PG server's certificate to my local keystore, things
>>> began
>>> to work as it should:
>>> ->  Agent tables installed
>>>
>>> To be continued. I have a lot of things to do at the moment, and I cannot
>>> test my contribution properly yet since I'm at home with a slower
>>> bandwidth.
>>>
>>> To be continued. :)
>>>
>>> Erlend
>>>
>>>
>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>
>>>> Sure, feel free to create a ticket and try to come up with a patch.
>>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>>> communication, I would make those parameters local to the postgresql
>>>> implementation.  So the parameters would be something like:
>>>>
>>>> org.apache.manifoldcf.postgresql.hostname
>>>> org.apache.manifoldcf.postgresql.ssl
>>>>
>>>> or some such.  Then, you'd want to load the necessary parameters in
>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>>> since that's not officially supported at this time.)
>>>>
>>>> The only problem I see is that the constructor architecture of
>>>> Database.java may make it a challenge to load the appropriate
>>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>>> could make it work, and if not, we should probably fix this anyway to
>>>> make it easier.
>>>>
>>>> Thanks,
>>>> Karl
>>>>
>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> You're right! I didn't read the command section in the documentation
>>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>>>> Resin
>>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>>>> other
>>>>> errors have disappeared, which means that it should be possible to run
>>>>> the
>>>>> application on Resin.
>>>>>
>>>>> Do you think I should create a ticket about running an external
>>>>> PostgreSQL
>>>>> database as well? I think I know which classes to modify. I see there
>>>>> are
>>>>> classes for MySQL as well, so perhaps the issue should be generally,
>>>>> i.e.
>>>>> not PG specific.
>>>>>
>>>>> I was thinking about extending the property list like this:
>>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>>> localhost)
>>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>>
>>>>> For PostgreSQL I think it should be quite easy to implement this, but
>>>>> I'm
>>>>> not sure when it comes to MySQL. Haven't touched that database system
>>>>> for
>>>>> years.
>>>>>
>>>>> Erlend
>>>>>
>>>>>
>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>
>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>> installation in how-to-build-and-deploy.html?
>>>>>> You will need to create the database (using DBCreate), and install the
>>>>>> schema (using Install), and register the crawler agent (using
>>>>>> Register).  You will also need to create the properties.xml and
>>>>>> logging.ini files by hand.
>>>>>>
>>>>>> Let me know if this helps.
>>>>>> Karl
>>>>>>
>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>  wrote:
>>>>>>>
>>>>>>> I guess the problem is that the database does not exist. When I
>>>>>>> installed
>>>>>>> it
>>>>>>> on Tomcat, I got an error message telling me that the schema TEST
>>>>>>> does
>>>>>>> not
>>>>>>> exist. I tried to build the database with the username and password
>>>>>>> "test"
>>>>>>> by the following command:
>>>>>>> ./processes/script/executecommand.sh
>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>> test test
>>>>>>>
>>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>>>> value="test"/>
>>>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>>>> value="test"/>
>>>>>>>
>>>>>>> I'm using the Derby database since I cannot see how I could contact
>>>>>>> an
>>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>>> database
>>>>>>> systems the university supports. I guess there should be an option
>>>>>>> for
>>>>>>> configuring external hosts and optionally turn on SSL encryption.
>>>>>>> Maybe
>>>>>>> another ticket to create.
>>>>>>>
>>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>>>> it
>>>>>>> on
>>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>>
>>>>>>> Erlend
>>>>>>>
>>>>>>>
>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>
>>>>>>>> Sure, if you want, create a ticket, and see if you can figure out
>>>>>>>> what
>>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>>> stylesheet, actually.  The "redirection" error sound like the
>>>>>>>> crawler
>>>>>>>> UI is trying to present an error and resin is handling one of the
>>>>>>>> java
>>>>>>>> standard tags in a non-standard way.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>>> support
>>>>>>>>> Tomcat.
>>>>>>>>>
>>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>>> message
>>>>>>>>> comes
>>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>>> written.
>>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>>> ManifoldCF
>>>>>>>>> on Resin.
>>>>>>>>>
>>>>>>>>> Erlend
>>>>>>>>>
>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>
>>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
>>>>>>>>>>  Clearly
>>>>>>>>>> Resin as an application server is having difficulty doing some
>>>>>>>>>> very
>>>>>>>>>> basic stuff.
>>>>>>>>>>
>>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>>> nothing
>>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>>>  It
>>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>>  wrote:
>>>>>>>>>>>
>>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>>>> and
>>>>>>>>>>> set
>>>>>>>>>>> the
>>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>>
>>>>>>>>>>> <system-property
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>> />
>>>>>>>>>>>
>>>>>>>>>>> I have also created my own home directory for ManifoldCF and
>>>>>>>>>>> copied
>>>>>>>>>>> the
>>>>>>>>>>> necessary files from the example directory. When I start Resin
>>>>>>>>>>> and
>>>>>>>>>>> goes
>>>>>>>>>>> to
>>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>>>> ugly
>>>>>>>>>>> (I
>>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>>>> click
>>>>>>>>>>> on
>>>>>>>>>>> a
>>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>>> headers
>>>>>>>>>>>
>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>
>>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>>>> it
>>>>>>>>>>> runs
>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>
>>>>>>>>>>> E
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>> University of Oslo
>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>>> VIP:
>>>>>>>>>>> 31050
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Erlend Garåsen
>>>>>>>>> Center for Information Technology Services
>>>>>>>>> University of Oslo
>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>> VIP:
>>>>>>>>> 31050
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

Maybe I should create the PostgreSQL patch without any more testing. I 
have some problems to run the jobs inside the application, but I doubt 
the reason is my PostgreSQL code changes since the same happens with 
Apache Derby. The status of the job says "Starting up" and nothing shows 
up in the manifoldcf.log nor in the history logs available inside the 
application.

If I select "Simple History", the list is empty, but in the activities 
list (in the upper right corner) I see that "robots parse" is the last 
activity run.

This is not something Resin specific since the same happens under Tomcat.

My contribution is clean and I have only made a few changes in the 
DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the 
Database super class.

I haven't run "ant test-pg" since it seems to need a local Postgresql 
server running. But "ant test" reports no errors.

I have done an svn up before I created the patch in order to include the 
latest changes from trunk.

The documentation needs to be updated and I can suggest what to add later.

BTW, it should be interesting to know what causes the problems described 
above since I really want to run my jobs.

PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if 
I'm logging in, so I had to turn of formatting in Eclipse in order to 
keep the existing formatting (I get the following error when I click on 
the link to the xml file: "You are not allowed to do AttachFile on this 
page.")

Erlend



I don't think it is something PG specific either,

On 25.02.11 16.15, Karl Wright wrote:
> The DBCreate command requires privs necessary to create the database
> instance in the first place.  It uses the admin credentials that you
> supply to it as command args.  These are NOT the same credentials used
> to access the database instance, after it has been created.
>
> Thus, you can skip the DBCreate step if you have already created the
> database by hand.
>
> It does not surprise me that PostgreSQL's JDBC driver uses the global
> Java trust store.  Since there is only one database active for any
> installation, it is perfectly OK for the global trust store to be used
> for this purpose.  It will, however, probably require the addition of
> some documentation as to how to set it up properly on various
> platforms.
>
> Thanks,
> Karl
>
> On Fri, Feb 25, 2011 at 9:46 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> I think we need another parameter as well:
>> org.apache.manifoldcf.postgresql.port
>>
>> I was going to test my code today, but figured out that the following
>> command class needs admin privileges to the PostgreSQL database, which I
>> don't and should never have:
>> org.apache.manifoldcf.core.DBCreate
>>
>> So I skipped that part. Anyway, the DBCreate method is also supposed to work
>> with my contribution if I had the sufficient privileges.
>>
>> I went on by using the Install method class instead, but ended up with a
>> Handshake exception:
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target.
>>
>> After I added our PG server's certificate to my local keystore, things began
>> to work as it should:
>> ->  Agent tables installed
>>
>> To be continued. I have a lot of things to do at the moment, and I cannot
>> test my contribution properly yet since I'm at home with a slower bandwidth.
>>
>> To be continued. :)
>>
>> Erlend
>>
>>
>> On 04.02.11 16.30, Karl Wright wrote:
>>>
>>> Sure, feel free to create a ticket and try to come up with a patch.
>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>> communication, I would make those parameters local to the postgresql
>>> implementation.  So the parameters would be something like:
>>>
>>> org.apache.manifoldcf.postgresql.hostname
>>> org.apache.manifoldcf.postgresql.ssl
>>>
>>> or some such.  Then, you'd want to load the necessary parameters in
>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>> since that's not officially supported at this time.)
>>>
>>> The only problem I see is that the constructor architecture of
>>> Database.java may make it a challenge to load the appropriate
>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>> could make it work, and if not, we should probably fix this anyway to
>>> make it easier.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>>   wrote:
>>>>
>>>> You're right! I didn't read the command section in the documentation
>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>>> Resin
>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>>> other
>>>> errors have disappeared, which means that it should be possible to run
>>>> the
>>>> application on Resin.
>>>>
>>>> Do you think I should create a ticket about running an external
>>>> PostgreSQL
>>>> database as well? I think I know which classes to modify. I see there are
>>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>>> not PG specific.
>>>>
>>>> I was thinking about extending the property list like this:
>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>> localhost)
>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>
>>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>>> not sure when it comes to MySQL. Haven't touched that database system for
>>>> years.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>
>>>>> Ok, have you followed the instructions for a multi-process
>>>>> installation in how-to-build-and-deploy.html?
>>>>> You will need to create the database (using DBCreate), and install the
>>>>> schema (using Install), and register the crawler agent (using
>>>>> Register).  You will also need to create the properties.xml and
>>>>> logging.ini files by hand.
>>>>>
>>>>> Let me know if this helps.
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>   wrote:
>>>>>>
>>>>>> I guess the problem is that the database does not exist. When I
>>>>>> installed
>>>>>> it
>>>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>>>> not
>>>>>> exist. I tried to build the database with the username and password
>>>>>> "test"
>>>>>> by the following command:
>>>>>> ./processes/script/executecommand.sh
>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>> test test
>>>>>>
>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>   <property name="org.apache.manifoldcf.database.username"
>>>>>> value="test"/>
>>>>>>   <property name="org.apache.manifoldcf.database.password"
>>>>>> value="test"/>
>>>>>>
>>>>>> I'm using the Derby database since I cannot see how I could contact an
>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>> database
>>>>>> systems the university supports. I guess there should be an option for
>>>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>>>> another ticket to create.
>>>>>>
>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>>> it
>>>>>> on
>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>>>> UI is trying to present an error and resin is handling one of the java
>>>>>>> standard tags in a non-standard way.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>   wrote:
>>>>>>>>
>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>> support
>>>>>>>> Tomcat.
>>>>>>>>
>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>> message
>>>>>>>> comes
>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>> written.
>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>> ManifoldCF
>>>>>>>> on Resin.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>>>> basic stuff.
>>>>>>>>>
>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>> nothing
>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>>   It
>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>   wrote:
>>>>>>>>>>
>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>>> and
>>>>>>>>>> set
>>>>>>>>>> the
>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>
>>>>>>>>>> <system-property
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>> />
>>>>>>>>>>
>>>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>>>> the
>>>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>>>> goes
>>>>>>>>>> to
>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>>> ugly
>>>>>>>>>> (I
>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>>> click
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>> headers
>>>>>>>>>>
>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>
>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>>> it
>>>>>>>>>> runs
>>>>>>>>>> stable on Resin.
>>>>>>>>>>
>>>>>>>>>> E
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

The DBCreate command requires privs necessary to create the database
instance in the first place.  It uses the admin credentials that you
supply to it as command args.  These are NOT the same credentials used
to access the database instance, after it has been created.

Thus, you can skip the DBCreate step if you have already created the
database by hand.

It does not surprise me that PostgreSQL's JDBC driver uses the global
Java trust store.  Since there is only one database active for any
installation, it is perfectly OK for the global trust store to be used
for this purpose.  It will, however, probably require the addition of
some documentation as to how to set it up properly on various
platforms.

Thanks,
Karl

On Fri, Feb 25, 2011 at 9:46 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I think we need another parameter as well:
> org.apache.manifoldcf.postgresql.port
>
> I was going to test my code today, but figured out that the following
> command class needs admin privileges to the PostgreSQL database, which I
> don't and should never have:
> org.apache.manifoldcf.core.DBCreate
>
> So I skipped that part. Anyway, the DBCreate method is also supposed to work
> with my contribution if I had the sufficient privileges.
>
> I went on by using the Install method class instead, but ended up with a
> Handshake exception:
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target.
>
> After I added our PG server's certificate to my local keystore, things began
> to work as it should:
> -> Agent tables installed
>
> To be continued. I have a lot of things to do at the moment, and I cannot
> test my contribution properly yet since I'm at home with a slower bandwidth.
>
> To be continued. :)
>
> Erlend
>
>
> On 04.02.11 16.30, Karl Wright wrote:
>>
>> Sure, feel free to create a ticket and try to come up with a patch.
>> Since the DBInterfacePostgreSQL class is responsible for any remote
>> communication, I would make those parameters local to the postgresql
>> implementation.  So the parameters would be something like:
>>
>> org.apache.manifoldcf.postgresql.hostname
>> org.apache.manifoldcf.postgresql.ssl
>>
>> or some such.  Then, you'd want to load the necessary parameters in
>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>> since that's not officially supported at this time.)
>>
>> The only problem I see is that the constructor architecture of
>> Database.java may make it a challenge to load the appropriate
>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>> could make it work, and if not, we should probably fix this anyway to
>> make it easier.
>>
>> Thanks,
>> Karl
>>
>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> You're right! I didn't read the command section in the documentation
>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>> Resin
>>> is that the necessary resources, such as CSS, are not loaded. All the
>>> other
>>> errors have disappeared, which means that it should be possible to run
>>> the
>>> application on Resin.
>>>
>>> Do you think I should create a ticket about running an external
>>> PostgreSQL
>>> database as well? I think I know which classes to modify. I see there are
>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>> not PG specific.
>>>
>>> I was thinking about extending the property list like this:
>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>> localhost)
>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>
>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>> not sure when it comes to MySQL. Haven't touched that database system for
>>> years.
>>>
>>> Erlend
>>>
>>>
>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>
>>>> Ok, have you followed the instructions for a multi-process
>>>> installation in how-to-build-and-deploy.html?
>>>> You will need to create the database (using DBCreate), and install the
>>>> schema (using Install), and register the crawler agent (using
>>>> Register).  You will also need to create the properties.xml and
>>>> logging.ini files by hand.
>>>>
>>>> Let me know if this helps.
>>>> Karl
>>>>
>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> I guess the problem is that the database does not exist. When I
>>>>> installed
>>>>> it
>>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>>> not
>>>>> exist. I tried to build the database with the username and password
>>>>> "test"
>>>>> by the following command:
>>>>> ./processes/script/executecommand.sh
>>>>> org.apache.manifoldcf.core.DBCreate
>>>>> test test
>>>>>
>>>>> And I also configured these two lines in my properties.xml file:
>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>> value="test"/>
>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>> value="test"/>
>>>>>
>>>>> I'm using the Derby database since I cannot see how I could contact an
>>>>> external PostgreSQL server on another host. PG is one of several
>>>>> database
>>>>> systems the university supports. I guess there should be an option for
>>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>>> another ticket to create.
>>>>>
>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>> it
>>>>> on
>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>
>>>>> Erlend
>>>>>
>>>>>
>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>
>>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>>> UI is trying to present an error and resin is handling one of the java
>>>>>> standard tags in a non-standard way.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>  wrote:
>>>>>>>
>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>> support
>>>>>>> Tomcat.
>>>>>>>
>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>> message
>>>>>>> comes
>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>> written.
>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>> ManifoldCF
>>>>>>> on Resin.
>>>>>>>
>>>>>>> Erlend
>>>>>>>
>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>
>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>>> basic stuff.
>>>>>>>>
>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>> nothing
>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>  It
>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>> and
>>>>>>>>> set
>>>>>>>>> the
>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>
>>>>>>>>> <system-property
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>> />
>>>>>>>>>
>>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>>> the
>>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>>> goes
>>>>>>>>> to
>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>> ugly
>>>>>>>>> (I
>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>> click
>>>>>>>>> on
>>>>>>>>> a
>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>> headers
>>>>>>>>>
>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>
>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>> it
>>>>>>>>> runs
>>>>>>>>> stable on Resin.
>>>>>>>>>
>>>>>>>>> E
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Erlend Garåsen
>>>>>>>>> Center for Information Technology Services
>>>>>>>>> University of Oslo
>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>> VIP:
>>>>>>>>> 31050
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

I think we need another parameter as well:
org.apache.manifoldcf.postgresql.port

I was going to test my code today, but figured out that the following 
command class needs admin privileges to the PostgreSQL database, which I 
don't and should never have:
org.apache.manifoldcf.core.DBCreate

So I skipped that part. Anyway, the DBCreate method is also supposed to 
work with my contribution if I had the sufficient privileges.

I went on by using the Install method class instead, but ended up with a 
Handshake exception:
javax.net.ssl.SSLHandshakeException: 
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to 
find valid certification path to requested target.

After I added our PG server's certificate to my local keystore, things 
began to work as it should:
-> Agent tables installed

To be continued. I have a lot of things to do at the moment, and I 
cannot test my contribution properly yet since I'm at home with a slower 
bandwidth.

To be continued. :)

Erlend


On 04.02.11 16.30, Karl Wright wrote:
> Sure, feel free to create a ticket and try to come up with a patch.
> Since the DBInterfacePostgreSQL class is responsible for any remote
> communication, I would make those parameters local to the postgresql
> implementation.  So the parameters would be something like:
>
> org.apache.manifoldcf.postgresql.hostname
> org.apache.manifoldcf.postgresql.ssl
>
> or some such.  Then, you'd want to load the necessary parameters in
> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
> since that's not officially supported at this time.)
>
> The only problem I see is that the constructor architecture of
> Database.java may make it a challenge to load the appropriate
> parameters in the DBInterfacePostgreSQL constructor.  But I think you
> could make it work, and if not, we should probably fix this anyway to
> make it easier.
>
> Thanks,
> Karl
>
> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> You're right! I didn't read the command section in the documentation
>> properly. ManifoldCF does also run on Resin now. The only problem with Resin
>> is that the necessary resources, such as CSS, are not loaded. All the other
>> errors have disappeared, which means that it should be possible to run the
>> application on Resin.
>>
>> Do you think I should create a ticket about running an external PostgreSQL
>> database as well? I think I know which classes to modify. I see there are
>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>> not PG specific.
>>
>> I was thinking about extending the property list like this:
>> org.apache.manifoldcf.database.hostname (not required, defaults to
>> localhost)
>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>
>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>> not sure when it comes to MySQL. Haven't touched that database system for
>> years.
>>
>> Erlend
>>
>>
>> On 04.02.11 15.42, Karl Wright wrote:
>>>
>>> Ok, have you followed the instructions for a multi-process
>>> installation in how-to-build-and-deploy.html?
>>> You will need to create the database (using DBCreate), and install the
>>> schema (using Install), and register the crawler agent (using
>>> Register).  You will also need to create the properties.xml and
>>> logging.ini files by hand.
>>>
>>> Let me know if this helps.
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>   wrote:
>>>>
>>>> I guess the problem is that the database does not exist. When I installed
>>>> it
>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>> not
>>>> exist. I tried to build the database with the username and password
>>>> "test"
>>>> by the following command:
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
>>>> test test
>>>>
>>>> And I also configured these two lines in my properties.xml file:
>>>>   <property name="org.apache.manifoldcf.database.username" value="test"/>
>>>>   <property name="org.apache.manifoldcf.database.password" value="test"/>
>>>>
>>>> I'm using the Derby database since I cannot see how I could contact an
>>>> external PostgreSQL server on another host. PG is one of several database
>>>> systems the university supports. I guess there should be an option for
>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>> another ticket to create.
>>>>
>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it
>>>> on
>>>> Resin and then try to configure an external PostgreSQL server.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>
>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>> the problems are.  I'd start with not being able to reach the
>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>> UI is trying to present an error and resin is handling one of the java
>>>>> standard tags in a non-standard way.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>   wrote:
>>>>>>
>>>>>> Resin is de facto application server at the university, but we also
>>>>>> support
>>>>>> Tomcat.
>>>>>>
>>>>>> I can try to deploy the war files on Tomcat instead. The error message
>>>>>> comes
>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>> written.
>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>> ManifoldCF
>>>>>> on Resin.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>
>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>> basic stuff.
>>>>>>>
>>>>>>> FWIW, when we first started the Jetty development, I had to do nothing
>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.  It
>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>   wrote:
>>>>>>>>
>>>>>>>> I have placed the three war files into Resin's webapps directory and
>>>>>>>> set
>>>>>>>> the
>>>>>>>> following system variable inside resin.xml:
>>>>>>>>
>>>>>>>> <system-property
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>> />
>>>>>>>>
>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>> the
>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>> goes
>>>>>>>> to
>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly
>>>>>>>> (I
>>>>>>>> guess the path to CSS files etc. are broken). And when I try to click
>>>>>>>> on
>>>>>>>> a
>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>> headers
>>>>>>>>
>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>
>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>>>>> runs
>>>>>>>> stable on Resin.
>>>>>>>>
>>>>>>>> E
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Oh, if you are using Derby, you're screwed, because Derby is
single-process in embedded mode.  Sorry for not catching that sooner!

Karl

On Fri, Feb 4, 2011 at 11:25 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> The log entry I sent is not related to the job, but when I forgot to start
> Solr when I created the Solr connection.
>
> I have created another job and exactly the same thing happens. It just won't
> start. Solr is up and running. The connection status for the Solr connector
> says "Connection working".
>
> BTW, I have just started the following agents/connectors:
> org.apache.manifoldcf.crawler.system.CrawlerAgent
> org.apache.manifoldcf.agents.output.solr.SolrConnector
> org.apache.manifoldcf.agents.output.nullconnector.NullConnector
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>
> But I can try to use PostgreSQL instead of Derby next week in case it helps.
>
> Erlend
>
>
> On 04.02.11 17.12, Karl Wright wrote:
>>
>> You are seeing this on the job status screen, right?  That means that
>> the job is aborting because it can't talk to Solr.  This is exactly
>> the error I was expecting it to be, so no surprises.
>>
>> If you think the behavior should be for ManifoldCF to retry if it
>> can't reach Solr, then we can change the Solr connector accordingly.
>> But based on the way it is coded now, that is why your job doesn't
>> last long.
>>
>> Karl
>>
>>
>> On Fri, Feb 4, 2011 at 11:08 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> On 04.02.11 16.58, Karl Wright wrote:
>>>>
>>>> Do you have a full stack trace with that message?  I would expect one
>>>> in manifoldcf.log.
>>>
>>> Perhaps this log entry is caused by a mistake I did when I forgot to
>>> start
>>> Solr before I configured the Solr connector. So the real problem may not
>>> be
>>> logged at all.
>>>
>>> java.net.ConnectException: Connection refused
>>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>>        at
>>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>>>        at java.net.Socket.connect(Socket.java:529)
>>>        at java.net.Socket.connect(Socket.java:478)
>>>        at java.net.Socket.<init>(Socket.java:375)
>>>        at java.net.Socket.<init>(Socket.java:189)
>>>        at
>>>
>>> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:733)
>>>        at
>>>
>>> org.apache.manifoldcf.agents.output.solr.HttpPoster$StatusThread.run(HttpPoster.java:1752)
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

The log entry I sent is not related to the job, but when I forgot to 
start Solr when I created the Solr connection.

I have created another job and exactly the same thing happens. It just 
won't start. Solr is up and running. The connection status for the Solr 
connector says "Connection working".

BTW, I have just started the following agents/connectors:
org.apache.manifoldcf.crawler.system.CrawlerAgent
org.apache.manifoldcf.agents.output.solr.SolrConnector
org.apache.manifoldcf.agents.output.nullconnector.NullConnector
org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector

But I can try to use PostgreSQL instead of Derby next week in case it helps.

Erlend


On 04.02.11 17.12, Karl Wright wrote:
> You are seeing this on the job status screen, right?  That means that
> the job is aborting because it can't talk to Solr.  This is exactly
> the error I was expecting it to be, so no surprises.
>
> If you think the behavior should be for ManifoldCF to retry if it
> can't reach Solr, then we can change the Solr connector accordingly.
> But based on the way it is coded now, that is why your job doesn't
> last long.
>
> Karl
>
>
> On Fri, Feb 4, 2011 at 11:08 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>> On 04.02.11 16.58, Karl Wright wrote:
>>>
>>> Do you have a full stack trace with that message?  I would expect one
>>> in manifoldcf.log.
>>
>> Perhaps this log entry is caused by a mistake I did when I forgot to start
>> Solr before I configured the Solr connector. So the real problem may not be
>> logged at all.
>>
>> java.net.ConnectException: Connection refused
>>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>>         at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>         at
>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>         at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>>         at java.net.Socket.connect(Socket.java:529)
>>         at java.net.Socket.connect(Socket.java:478)
>>         at java.net.Socket.<init>(Socket.java:375)
>>         at java.net.Socket.<init>(Socket.java:189)
>>         at
>> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:733)
>>         at
>> org.apache.manifoldcf.agents.output.solr.HttpPoster$StatusThread.run(HttpPoster.java:1752)
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

You are seeing this on the job status screen, right?  That means that
the job is aborting because it can't talk to Solr.  This is exactly
the error I was expecting it to be, so no surprises.

If you think the behavior should be for ManifoldCF to retry if it
can't reach Solr, then we can change the Solr connector accordingly.
But based on the way it is coded now, that is why your job doesn't
last long.

Karl


On Fri, Feb 4, 2011 at 11:08 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
> On 04.02.11 16.58, Karl Wright wrote:
>>
>> Do you have a full stack trace with that message?  I would expect one
>> in manifoldcf.log.
>
> Perhaps this log entry is caused by a mistake I did when I forgot to start
> Solr before I configured the Solr connector. So the real problem may not be
> logged at all.
>
> java.net.ConnectException: Connection refused
>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>        at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>        at java.net.Socket.connect(Socket.java:529)
>        at java.net.Socket.connect(Socket.java:478)
>        at java.net.Socket.<init>(Socket.java:375)
>        at java.net.Socket.<init>(Socket.java:189)
>        at
> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:733)
>        at
> org.apache.manifoldcf.agents.output.solr.HttpPoster$StatusThread.run(HttpPoster.java:1752)
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

On 04.02.11 16.58, Karl Wright wrote:
> Do you have a full stack trace with that message?  I would expect one
> in manifoldcf.log.

Perhaps this log entry is caused by a mistake I did when I forgot to 
start Solr before I configured the Solr connector. So the real problem 
may not be logged at all.

java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
	at java.net.Socket.connect(Socket.java:529)
	at java.net.Socket.connect(Socket.java:478)
	at java.net.Socket.<init>(Socket.java:375)
	at java.net.Socket.<init>(Socket.java:189)
	at 
org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:733)
	at 
org.apache.manifoldcf.agents.output.solr.HttpPoster$StatusThread.run(HttpPoster.java:1752)


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

It looks like it's unable to connect to Solr.

This is currently treated as a "hard error" and it will abort the crawl.

Karl

On Fri, Feb 4, 2011 at 10:58 AM, Karl Wright <da...@gmail.com> wrote:
> Do you have a full stack trace with that message?  I would expect one
> in manifoldcf.log.
> Karl
>
>
> On Fri, Feb 4, 2011 at 10:55 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>>
>> OK, I will create two issues, but the patches will probably not be created
>> until I'm back in Norway. I'm going to Spain next week and will work with
>> completely different things the next two weeks.
>>
>> BTW, the Resin issue is quite funny. The CSS file is loaded if I use Safari.
>> It does not work on Firefox if ManifoldCF is deployed on Resin. It seems
>> that everything works, except I cannot start a new job, perhaps because I
>> have forgot to start a particular agent. I'm getting this error:
>> Error communicating with Ingestion API: Connection refused
>>
>> Erlend
>>
>>
>> On 04.02.11 16.30, Karl Wright wrote:
>>>
>>> Sure, feel free to create a ticket and try to come up with a patch.
>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>> communication, I would make those parameters local to the postgresql
>>> implementation.  So the parameters would be something like:
>>>
>>> org.apache.manifoldcf.postgresql.hostname
>>> org.apache.manifoldcf.postgresql.ssl
>>>
>>> or some such.  Then, you'd want to load the necessary parameters in
>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>> since that's not officially supported at this time.)
>>>
>>> The only problem I see is that the constructor architecture of
>>> Database.java may make it a challenge to load the appropriate
>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>> could make it work, and if not, we should probably fix this anyway to
>>> make it easier.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>>  wrote:
>>>>
>>>> You're right! I didn't read the command section in the documentation
>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>>> Resin
>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>>> other
>>>> errors have disappeared, which means that it should be possible to run
>>>> the
>>>> application on Resin.
>>>>
>>>> Do you think I should create a ticket about running an external
>>>> PostgreSQL
>>>> database as well? I think I know which classes to modify. I see there are
>>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>>> not PG specific.
>>>>
>>>> I was thinking about extending the property list like this:
>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>> localhost)
>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>
>>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>>> not sure when it comes to MySQL. Haven't touched that database system for
>>>> years.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>
>>>>> Ok, have you followed the instructions for a multi-process
>>>>> installation in how-to-build-and-deploy.html?
>>>>> You will need to create the database (using DBCreate), and install the
>>>>> schema (using Install), and register the crawler agent (using
>>>>> Register).  You will also need to create the properties.xml and
>>>>> logging.ini files by hand.
>>>>>
>>>>> Let me know if this helps.
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> I guess the problem is that the database does not exist. When I
>>>>>> installed
>>>>>> it
>>>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>>>> not
>>>>>> exist. I tried to build the database with the username and password
>>>>>> "test"
>>>>>> by the following command:
>>>>>> ./processes/script/executecommand.sh
>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>> test test
>>>>>>
>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>>> value="test"/>
>>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>>> value="test"/>
>>>>>>
>>>>>> I'm using the Derby database since I cannot see how I could contact an
>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>> database
>>>>>> systems the university supports. I guess there should be an option for
>>>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>>>> another ticket to create.
>>>>>>
>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>>> it
>>>>>> on
>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>>>> UI is trying to present an error and resin is handling one of the java
>>>>>>> standard tags in a non-standard way.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>> support
>>>>>>>> Tomcat.
>>>>>>>>
>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>>> message
>>>>>>>> comes
>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>> written.
>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>> ManifoldCF
>>>>>>>> on Resin.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>>>> basic stuff.
>>>>>>>>>
>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>>> nothing
>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>>  It
>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>>> and
>>>>>>>>>> set
>>>>>>>>>> the
>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>
>>>>>>>>>> <system-property
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>> />
>>>>>>>>>>
>>>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>>>> the
>>>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>>>> goes
>>>>>>>>>> to
>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>>> ugly
>>>>>>>>>> (I
>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>>> click
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>> headers
>>>>>>>>>>
>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>
>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>>> it
>>>>>>>>>> runs
>>>>>>>>>> stable on Resin.
>>>>>>>>>>
>>>>>>>>>> E
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>
>

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Do you have a full stack trace with that message?  I would expect one
in manifoldcf.log.
Karl


On Fri, Feb 4, 2011 at 10:55 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> OK, I will create two issues, but the patches will probably not be created
> until I'm back in Norway. I'm going to Spain next week and will work with
> completely different things the next two weeks.
>
> BTW, the Resin issue is quite funny. The CSS file is loaded if I use Safari.
> It does not work on Firefox if ManifoldCF is deployed on Resin. It seems
> that everything works, except I cannot start a new job, perhaps because I
> have forgot to start a particular agent. I'm getting this error:
> Error communicating with Ingestion API: Connection refused
>
> Erlend
>
>
> On 04.02.11 16.30, Karl Wright wrote:
>>
>> Sure, feel free to create a ticket and try to come up with a patch.
>> Since the DBInterfacePostgreSQL class is responsible for any remote
>> communication, I would make those parameters local to the postgresql
>> implementation.  So the parameters would be something like:
>>
>> org.apache.manifoldcf.postgresql.hostname
>> org.apache.manifoldcf.postgresql.ssl
>>
>> or some such.  Then, you'd want to load the necessary parameters in
>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>> since that's not officially supported at this time.)
>>
>> The only problem I see is that the constructor architecture of
>> Database.java may make it a challenge to load the appropriate
>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>> could make it work, and if not, we should probably fix this anyway to
>> make it easier.
>>
>> Thanks,
>> Karl
>>
>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> You're right! I didn't read the command section in the documentation
>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>> Resin
>>> is that the necessary resources, such as CSS, are not loaded. All the
>>> other
>>> errors have disappeared, which means that it should be possible to run
>>> the
>>> application on Resin.
>>>
>>> Do you think I should create a ticket about running an external
>>> PostgreSQL
>>> database as well? I think I know which classes to modify. I see there are
>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>> not PG specific.
>>>
>>> I was thinking about extending the property list like this:
>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>> localhost)
>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>
>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>> not sure when it comes to MySQL. Haven't touched that database system for
>>> years.
>>>
>>> Erlend
>>>
>>>
>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>
>>>> Ok, have you followed the instructions for a multi-process
>>>> installation in how-to-build-and-deploy.html?
>>>> You will need to create the database (using DBCreate), and install the
>>>> schema (using Install), and register the crawler agent (using
>>>> Register).  You will also need to create the properties.xml and
>>>> logging.ini files by hand.
>>>>
>>>> Let me know if this helps.
>>>> Karl
>>>>
>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> I guess the problem is that the database does not exist. When I
>>>>> installed
>>>>> it
>>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>>> not
>>>>> exist. I tried to build the database with the username and password
>>>>> "test"
>>>>> by the following command:
>>>>> ./processes/script/executecommand.sh
>>>>> org.apache.manifoldcf.core.DBCreate
>>>>> test test
>>>>>
>>>>> And I also configured these two lines in my properties.xml file:
>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>> value="test"/>
>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>> value="test"/>
>>>>>
>>>>> I'm using the Derby database since I cannot see how I could contact an
>>>>> external PostgreSQL server on another host. PG is one of several
>>>>> database
>>>>> systems the university supports. I guess there should be an option for
>>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>>> another ticket to create.
>>>>>
>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>> it
>>>>> on
>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>
>>>>> Erlend
>>>>>
>>>>>
>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>
>>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>>> UI is trying to present an error and resin is handling one of the java
>>>>>> standard tags in a non-standard way.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>  wrote:
>>>>>>>
>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>> support
>>>>>>> Tomcat.
>>>>>>>
>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>>>>>>> message
>>>>>>> comes
>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>> written.
>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>> ManifoldCF
>>>>>>> on Resin.
>>>>>>>
>>>>>>> Erlend
>>>>>>>
>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>
>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>>> basic stuff.
>>>>>>>>
>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>>>>>>> nothing
>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>>>>>>  It
>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> I have placed the three war files into Resin's webapps directory
>>>>>>>>> and
>>>>>>>>> set
>>>>>>>>> the
>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>
>>>>>>>>> <system-property
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>> />
>>>>>>>>>
>>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>>> the
>>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>>> goes
>>>>>>>>> to
>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>>>>>>>> ugly
>>>>>>>>> (I
>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>>>>>>>> click
>>>>>>>>> on
>>>>>>>>> a
>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>> headers
>>>>>>>>>
>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>
>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>>>>>>>>> it
>>>>>>>>> runs
>>>>>>>>> stable on Resin.
>>>>>>>>>
>>>>>>>>> E
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Erlend Garåsen
>>>>>>>>> Center for Information Technology Services
>>>>>>>>> University of Oslo
>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>>> VIP:
>>>>>>>>> 31050
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

OK, I will create two issues, but the patches will probably not be 
created until I'm back in Norway. I'm going to Spain next week and will 
work with completely different things the next two weeks.

BTW, the Resin issue is quite funny. The CSS file is loaded if I use 
Safari. It does not work on Firefox if ManifoldCF is deployed on Resin. 
It seems that everything works, except I cannot start a new job, perhaps 
because I have forgot to start a particular agent. I'm getting this error:
Error communicating with Ingestion API: Connection refused

Erlend


On 04.02.11 16.30, Karl Wright wrote:
> Sure, feel free to create a ticket and try to come up with a patch.
> Since the DBInterfacePostgreSQL class is responsible for any remote
> communication, I would make those parameters local to the postgresql
> implementation.  So the parameters would be something like:
>
> org.apache.manifoldcf.postgresql.hostname
> org.apache.manifoldcf.postgresql.ssl
>
> or some such.  Then, you'd want to load the necessary parameters in
> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
> since that's not officially supported at this time.)
>
> The only problem I see is that the constructor architecture of
> Database.java may make it a challenge to load the appropriate
> parameters in the DBInterfacePostgreSQL constructor.  But I think you
> could make it work, and if not, we should probably fix this anyway to
> make it easier.
>
> Thanks,
> Karl
>
> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> You're right! I didn't read the command section in the documentation
>> properly. ManifoldCF does also run on Resin now. The only problem with Resin
>> is that the necessary resources, such as CSS, are not loaded. All the other
>> errors have disappeared, which means that it should be possible to run the
>> application on Resin.
>>
>> Do you think I should create a ticket about running an external PostgreSQL
>> database as well? I think I know which classes to modify. I see there are
>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>> not PG specific.
>>
>> I was thinking about extending the property list like this:
>> org.apache.manifoldcf.database.hostname (not required, defaults to
>> localhost)
>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>
>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>> not sure when it comes to MySQL. Haven't touched that database system for
>> years.
>>
>> Erlend
>>
>>
>> On 04.02.11 15.42, Karl Wright wrote:
>>>
>>> Ok, have you followed the instructions for a multi-process
>>> installation in how-to-build-and-deploy.html?
>>> You will need to create the database (using DBCreate), and install the
>>> schema (using Install), and register the crawler agent (using
>>> Register).  You will also need to create the properties.xml and
>>> logging.ini files by hand.
>>>
>>> Let me know if this helps.
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>   wrote:
>>>>
>>>> I guess the problem is that the database does not exist. When I installed
>>>> it
>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>> not
>>>> exist. I tried to build the database with the username and password
>>>> "test"
>>>> by the following command:
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
>>>> test test
>>>>
>>>> And I also configured these two lines in my properties.xml file:
>>>>   <property name="org.apache.manifoldcf.database.username" value="test"/>
>>>>   <property name="org.apache.manifoldcf.database.password" value="test"/>
>>>>
>>>> I'm using the Derby database since I cannot see how I could contact an
>>>> external PostgreSQL server on another host. PG is one of several database
>>>> systems the university supports. I guess there should be an option for
>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>> another ticket to create.
>>>>
>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it
>>>> on
>>>> Resin and then try to configure an external PostgreSQL server.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>
>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>> the problems are.  I'd start with not being able to reach the
>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>> UI is trying to present an error and resin is handling one of the java
>>>>> standard tags in a non-standard way.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>   wrote:
>>>>>>
>>>>>> Resin is de facto application server at the university, but we also
>>>>>> support
>>>>>> Tomcat.
>>>>>>
>>>>>> I can try to deploy the war files on Tomcat instead. The error message
>>>>>> comes
>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>> written.
>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>> ManifoldCF
>>>>>> on Resin.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>
>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>> basic stuff.
>>>>>>>
>>>>>>> FWIW, when we first started the Jetty development, I had to do nothing
>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.  It
>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>   wrote:
>>>>>>>>
>>>>>>>> I have placed the three war files into Resin's webapps directory and
>>>>>>>> set
>>>>>>>> the
>>>>>>>> following system variable inside resin.xml:
>>>>>>>>
>>>>>>>> <system-property
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>> />
>>>>>>>>
>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>> the
>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>> goes
>>>>>>>> to
>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly
>>>>>>>> (I
>>>>>>>> guess the path to CSS files etc. are broken). And when I try to click
>>>>>>>> on
>>>>>>>> a
>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>> headers
>>>>>>>>
>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>
>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>>>>> runs
>>>>>>>> stable on Resin.
>>>>>>>>
>>>>>>>> E
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

RE: Oracle as backend DB

[Jeffrey Guo <je...@semtific.com>]

Karl, I will create the ticket.

Thanks,

Jeffrey

-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com] 
Sent: Monday, February 21, 2011 2:31 AM
To: connectors-user@incubator.apache.org
Subject: Re: Oracle as backend DB

Hi Jeffrey,

I was wondering how this was going, as I had not seen a jira ticket
get created.  Do you want me to create one?

Karl

On Fri, Feb 11, 2011 at 5:10 AM, Karl Wright <da...@gmail.com> wrote:
> I don't have any knowledge of anyone else that is working on an Oracle
> port at this time.  I presume someone will pipe up if they are.
>
> A procedural point of order...  You will want to create a ticket in
> jira to cover your work.  https://issues.apache.org/jira .  I will
> assign the ticket to myself.  Questions and answers should be attached
> as comments to that ticket, and when your code is ready, you will need
> to attach it to that ticket as well, and click the "grant license to
> ASF" button when you attach it.  Then, I can commit it to the code
> base.
>
> Thanks!
> Karl
>
> On Thu, Feb 10, 2011 at 10:49 PM, Jeffrey Guo <je...@semtific.com> wrote:
>> Karl, thanks!  I am interested in implementing the Oracle port, if it has
>> not been done or is not being worked on.  I have looked at the class
>> DBInterfacePostgreSQL (similar ones for Derby, HSQLDB, and MySQL), if
there
>> are any tips you could share with me, I'd greatly appreciate it, but
>> otherwise, I will post my questions as I move forward with the porting.
>>
>> On another note, does anyone on this list have experiences with Open
>> Pipeline (http://www.openpipeline.org/)?
>>
>> Jeffrey
>>
>> -----Original Message-----
>> From: Karl Wright [mailto:daddywri@gmail.com]
>> Sent: Tuesday, February 08, 2011 1:03 AM
>> To: connectors-user@incubator.apache.org
>> Subject: Re: Running ManifoldCF on Resin
>>
>> I take it you mean the Oracle database, not the Oracle app server?
>> Open-source software relies on the contributions of individuals, not
>> really on road-maps.  ManifoldCF's database architecture is reasonably
>> well-suited for adding support for new kinds of databases, although in
>> my experience adding new database support is much harder than adding a
>> connector, because of the unique operating characteristics of
>> individual databases, which leads to the possibility of failure.
>> Nevertheless, if you want to volunteer, I'd be happy to lend you
>> whatever assistance you might need.
>>
>> Karl
>>
>> On Tue, Feb 8, 2011 at 12:41 AM, Jeffrey Guo <je...@semtific.com> wrote:
>>> Is Oracle on the road map to be supported?
>>>
>>> -----Original Message-----
>>> From: Karl Wright [mailto:daddywri@gmail.com]
>>> Sent: Friday, February 04, 2011 7:31 AM
>>> To: connectors-user@incubator.apache.org
>>> Subject: Re: Running ManifoldCF on Resin
>>>
>>> Sure, feel free to create a ticket and try to come up with a patch.
>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>> communication, I would make those parameters local to the postgresql
>>> implementation.  So the parameters would be something like:
>>>
>>> org.apache.manifoldcf.postgresql.hostname
>>> org.apache.manifoldcf.postgresql.ssl
>>>
>>> or some such.  Then, you'd want to load the necessary parameters in
>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>> since that's not officially supported at this time.)
>>>
>>> The only problem I see is that the constructor architecture of
>>> Database.java may make it a challenge to load the appropriate
>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>> could make it work, and if not, we should probably fix this anyway to
>>> make it easier.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen
<e....@usit.uio.no>
>>> wrote:
>>>>
>>>> You're right! I didn't read the command section in the documentation
>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>> Resin
>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>> other
>>>> errors have disappeared, which means that it should be possible to run
>> the
>>>> application on Resin.
>>>>
>>>> Do you think I should create a ticket about running an external
>> PostgreSQL
>>>> database as well? I think I know which classes to modify. I see there
are
>>>> classes for MySQL as well, so perhaps the issue should be generally,
i.e.
>>>> not PG specific.
>>>>
>>>> I was thinking about extending the property list like this:
>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>> localhost)
>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>
>>>> For PostgreSQL I think it should be quite easy to implement this, but
I'm
>>>> not sure when it comes to MySQL. Haven't touched that database system
for
>>>> years.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>
>>>>> Ok, have you followed the instructions for a multi-process
>>>>> installation in how-to-build-and-deploy.html?
>>>>> You will need to create the database (using DBCreate), and install the
>>>>> schema (using Install), and register the crawler agent (using
>>>>> Register).  You will also need to create the properties.xml and
>>>>> logging.ini files by hand.
>>>>>
>>>>> Let me know if this helps.
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> I guess the problem is that the database does not exist. When I
>>> installed
>>>>>> it
>>>>>> on Tomcat, I got an error message telling me that the schema TEST
does
>>>>>> not
>>>>>> exist. I tried to build the database with the username and password
>>>>>> "test"
>>>>>> by the following command:
>>>>>> ./processes/script/executecommand.sh
>> org.apache.manifoldcf.core.DBCreate
>>>>>> test test
>>>>>>
>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>  <property name="org.apache.manifoldcf.database.username"
>> value="test"/>
>>>>>>  <property name="org.apache.manifoldcf.database.password"
>> value="test"/>
>>>>>>
>>>>>> I'm using the Derby database since I cannot see how I could contact
an
>>>>>> external PostgreSQL server on another host. PG is one of several
>>> database
>>>>>> systems the university supports. I guess there should be an option
for
>>>>>> configuring external hosts and optionally turn on SSL encryption.
Maybe
>>>>>> another ticket to create.
>>>>>>
>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>> it
>>>>>> on
>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, if you want, create a ticket, and see if you can figure out
what
>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>> stylesheet, actually.  The "redirection" error sound like the
crawler
>>>>>>> UI is trying to present an error and resin is handling one of the
java
>>>>>>> standard tags in a non-standard way.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>> support
>>>>>>>> Tomcat.
>>>>>>>>
>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>> message
>>>>>>>> comes
>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>> written.
>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>> ManifoldCF
>>>>>>>> on Resin.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.
 Clearly
>>>>>>>>> Resin as an application server is having difficulty doing some
very
>>>>>>>>> basic stuff.
>>>>>>>>>
>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>> nothing
>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>  It
>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>> and
>>>>>>>>>> set
>>>>>>>>>> the
>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>
>>>>>>>>>> <system-property
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>
>>
org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties
>>> .xml"
>>>>>>>>>> />
>>>>>>>>>>
>>>>>>>>>> I have also created my own home directory for ManifoldCF and
copied
>>>>>>>>>> the
>>>>>>>>>> necessary files from the example directory. When I start Resin
and
>>>>>>>>>> goes
>>>>>>>>>> to
>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>> ugly
>>>>>>>>>> (I
>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>> click
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>> headers
>>>>>>>>>>
>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>
>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>> it
>>>>>>>>>> runs
>>>>>>>>>> stable on Resin.
>>>>>>>>>>
>>>>>>>>>> E
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>>
>>>
>>>
>>
>>
>

Re: Oracle as backend DB

[Karl Wright <da...@gmail.com>]

Hi Jeffrey,

I was wondering how this was going, as I had not seen a jira ticket
get created.  Do you want me to create one?

Karl

On Fri, Feb 11, 2011 at 5:10 AM, Karl Wright <da...@gmail.com> wrote:
> I don't have any knowledge of anyone else that is working on an Oracle
> port at this time.  I presume someone will pipe up if they are.
>
> A procedural point of order...  You will want to create a ticket in
> jira to cover your work.  https://issues.apache.org/jira .  I will
> assign the ticket to myself.  Questions and answers should be attached
> as comments to that ticket, and when your code is ready, you will need
> to attach it to that ticket as well, and click the "grant license to
> ASF" button when you attach it.  Then, I can commit it to the code
> base.
>
> Thanks!
> Karl
>
> On Thu, Feb 10, 2011 at 10:49 PM, Jeffrey Guo <je...@semtific.com> wrote:
>> Karl, thanks!  I am interested in implementing the Oracle port, if it has
>> not been done or is not being worked on.  I have looked at the class
>> DBInterfacePostgreSQL (similar ones for Derby, HSQLDB, and MySQL), if there
>> are any tips you could share with me, I'd greatly appreciate it, but
>> otherwise, I will post my questions as I move forward with the porting.
>>
>> On another note, does anyone on this list have experiences with Open
>> Pipeline (http://www.openpipeline.org/)?
>>
>> Jeffrey
>>
>> -----Original Message-----
>> From: Karl Wright [mailto:daddywri@gmail.com]
>> Sent: Tuesday, February 08, 2011 1:03 AM
>> To: connectors-user@incubator.apache.org
>> Subject: Re: Running ManifoldCF on Resin
>>
>> I take it you mean the Oracle database, not the Oracle app server?
>> Open-source software relies on the contributions of individuals, not
>> really on road-maps.  ManifoldCF's database architecture is reasonably
>> well-suited for adding support for new kinds of databases, although in
>> my experience adding new database support is much harder than adding a
>> connector, because of the unique operating characteristics of
>> individual databases, which leads to the possibility of failure.
>> Nevertheless, if you want to volunteer, I'd be happy to lend you
>> whatever assistance you might need.
>>
>> Karl
>>
>> On Tue, Feb 8, 2011 at 12:41 AM, Jeffrey Guo <je...@semtific.com> wrote:
>>> Is Oracle on the road map to be supported?
>>>
>>> -----Original Message-----
>>> From: Karl Wright [mailto:daddywri@gmail.com]
>>> Sent: Friday, February 04, 2011 7:31 AM
>>> To: connectors-user@incubator.apache.org
>>> Subject: Re: Running ManifoldCF on Resin
>>>
>>> Sure, feel free to create a ticket and try to come up with a patch.
>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>> communication, I would make those parameters local to the postgresql
>>> implementation.  So the parameters would be something like:
>>>
>>> org.apache.manifoldcf.postgresql.hostname
>>> org.apache.manifoldcf.postgresql.ssl
>>>
>>> or some such.  Then, you'd want to load the necessary parameters in
>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>> since that's not officially supported at this time.)
>>>
>>> The only problem I see is that the constructor architecture of
>>> Database.java may make it a challenge to load the appropriate
>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>> could make it work, and if not, we should probably fix this anyway to
>>> make it easier.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen <e....@usit.uio.no>
>>> wrote:
>>>>
>>>> You're right! I didn't read the command section in the documentation
>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>> Resin
>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>> other
>>>> errors have disappeared, which means that it should be possible to run
>> the
>>>> application on Resin.
>>>>
>>>> Do you think I should create a ticket about running an external
>> PostgreSQL
>>>> database as well? I think I know which classes to modify. I see there are
>>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>>> not PG specific.
>>>>
>>>> I was thinking about extending the property list like this:
>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>> localhost)
>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>
>>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>>> not sure when it comes to MySQL. Haven't touched that database system for
>>>> years.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>
>>>>> Ok, have you followed the instructions for a multi-process
>>>>> installation in how-to-build-and-deploy.html?
>>>>> You will need to create the database (using DBCreate), and install the
>>>>> schema (using Install), and register the crawler agent (using
>>>>> Register).  You will also need to create the properties.xml and
>>>>> logging.ini files by hand.
>>>>>
>>>>> Let me know if this helps.
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> I guess the problem is that the database does not exist. When I
>>> installed
>>>>>> it
>>>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>>>> not
>>>>>> exist. I tried to build the database with the username and password
>>>>>> "test"
>>>>>> by the following command:
>>>>>> ./processes/script/executecommand.sh
>> org.apache.manifoldcf.core.DBCreate
>>>>>> test test
>>>>>>
>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>  <property name="org.apache.manifoldcf.database.username"
>> value="test"/>
>>>>>>  <property name="org.apache.manifoldcf.database.password"
>> value="test"/>
>>>>>>
>>>>>> I'm using the Derby database since I cannot see how I could contact an
>>>>>> external PostgreSQL server on another host. PG is one of several
>>> database
>>>>>> systems the university supports. I guess there should be an option for
>>>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>>>> another ticket to create.
>>>>>>
>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>> it
>>>>>> on
>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>>>> UI is trying to present an error and resin is handling one of the java
>>>>>>> standard tags in a non-standard way.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>>> support
>>>>>>>> Tomcat.
>>>>>>>>
>>>>>>>> I can try to deploy the war files on Tomcat instead. The error
>> message
>>>>>>>> comes
>>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>>> written.
>>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>>> ManifoldCF
>>>>>>>> on Resin.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>>>> basic stuff.
>>>>>>>>>
>>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>>> nothing
>>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>>  It
>>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> I have placed the three war files into Resin's webapps directory
>> and
>>>>>>>>>> set
>>>>>>>>>> the
>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>
>>>>>>>>>> <system-property
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>
>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties
>>> .xml"
>>>>>>>>>> />
>>>>>>>>>>
>>>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>>>> the
>>>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>>>> goes
>>>>>>>>>> to
>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>>> ugly
>>>>>>>>>> (I
>>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>>> click
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>>> headers
>>>>>>>>>>
>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>
>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
>> it
>>>>>>>>>> runs
>>>>>>>>>> stable on Resin.
>>>>>>>>>>
>>>>>>>>>> E
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>>
>>>
>>>
>>
>>
>

Re: Oracle as backend DB

[Karl Wright <da...@gmail.com>]

I don't have any knowledge of anyone else that is working on an Oracle
port at this time.  I presume someone will pipe up if they are.

A procedural point of order...  You will want to create a ticket in
jira to cover your work.  https://issues.apache.org/jira .  I will
assign the ticket to myself.  Questions and answers should be attached
as comments to that ticket, and when your code is ready, you will need
to attach it to that ticket as well, and click the "grant license to
ASF" button when you attach it.  Then, I can commit it to the code
base.

Thanks!
Karl

On Thu, Feb 10, 2011 at 10:49 PM, Jeffrey Guo <je...@semtific.com> wrote:
> Karl, thanks!  I am interested in implementing the Oracle port, if it has
> not been done or is not being worked on.  I have looked at the class
> DBInterfacePostgreSQL (similar ones for Derby, HSQLDB, and MySQL), if there
> are any tips you could share with me, I'd greatly appreciate it, but
> otherwise, I will post my questions as I move forward with the porting.
>
> On another note, does anyone on this list have experiences with Open
> Pipeline (http://www.openpipeline.org/)?
>
> Jeffrey
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: Tuesday, February 08, 2011 1:03 AM
> To: connectors-user@incubator.apache.org
> Subject: Re: Running ManifoldCF on Resin
>
> I take it you mean the Oracle database, not the Oracle app server?
> Open-source software relies on the contributions of individuals, not
> really on road-maps.  ManifoldCF's database architecture is reasonably
> well-suited for adding support for new kinds of databases, although in
> my experience adding new database support is much harder than adding a
> connector, because of the unique operating characteristics of
> individual databases, which leads to the possibility of failure.
> Nevertheless, if you want to volunteer, I'd be happy to lend you
> whatever assistance you might need.
>
> Karl
>
> On Tue, Feb 8, 2011 at 12:41 AM, Jeffrey Guo <je...@semtific.com> wrote:
>> Is Oracle on the road map to be supported?
>>
>> -----Original Message-----
>> From: Karl Wright [mailto:daddywri@gmail.com]
>> Sent: Friday, February 04, 2011 7:31 AM
>> To: connectors-user@incubator.apache.org
>> Subject: Re: Running ManifoldCF on Resin
>>
>> Sure, feel free to create a ticket and try to come up with a patch.
>> Since the DBInterfacePostgreSQL class is responsible for any remote
>> communication, I would make those parameters local to the postgresql
>> implementation.  So the parameters would be something like:
>>
>> org.apache.manifoldcf.postgresql.hostname
>> org.apache.manifoldcf.postgresql.ssl
>>
>> or some such.  Then, you'd want to load the necessary parameters in
>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>> since that's not officially supported at this time.)
>>
>> The only problem I see is that the constructor architecture of
>> Database.java may make it a challenge to load the appropriate
>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>> could make it work, and if not, we should probably fix this anyway to
>> make it easier.
>>
>> Thanks,
>> Karl
>>
>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen <e....@usit.uio.no>
>> wrote:
>>>
>>> You're right! I didn't read the command section in the documentation
>>> properly. ManifoldCF does also run on Resin now. The only problem with
>> Resin
>>> is that the necessary resources, such as CSS, are not loaded. All the
>> other
>>> errors have disappeared, which means that it should be possible to run
> the
>>> application on Resin.
>>>
>>> Do you think I should create a ticket about running an external
> PostgreSQL
>>> database as well? I think I know which classes to modify. I see there are
>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>> not PG specific.
>>>
>>> I was thinking about extending the property list like this:
>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>> localhost)
>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>
>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>> not sure when it comes to MySQL. Haven't touched that database system for
>>> years.
>>>
>>> Erlend
>>>
>>>
>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>
>>>> Ok, have you followed the instructions for a multi-process
>>>> installation in how-to-build-and-deploy.html?
>>>> You will need to create the database (using DBCreate), and install the
>>>> schema (using Install), and register the crawler agent (using
>>>> Register).  You will also need to create the properties.xml and
>>>> logging.ini files by hand.
>>>>
>>>> Let me know if this helps.
>>>> Karl
>>>>
>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> I guess the problem is that the database does not exist. When I
>> installed
>>>>> it
>>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>>> not
>>>>> exist. I tried to build the database with the username and password
>>>>> "test"
>>>>> by the following command:
>>>>> ./processes/script/executecommand.sh
> org.apache.manifoldcf.core.DBCreate
>>>>> test test
>>>>>
>>>>> And I also configured these two lines in my properties.xml file:
>>>>>  <property name="org.apache.manifoldcf.database.username"
> value="test"/>
>>>>>  <property name="org.apache.manifoldcf.database.password"
> value="test"/>
>>>>>
>>>>> I'm using the Derby database since I cannot see how I could contact an
>>>>> external PostgreSQL server on another host. PG is one of several
>> database
>>>>> systems the university supports. I guess there should be an option for
>>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>>> another ticket to create.
>>>>>
>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
> it
>>>>> on
>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>
>>>>> Erlend
>>>>>
>>>>>
>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>
>>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>>> UI is trying to present an error and resin is handling one of the java
>>>>>> standard tags in a non-standard way.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>> Garåsen<e....@usit.uio.no>
>>>>>>  wrote:
>>>>>>>
>>>>>>> Resin is de facto application server at the university, but we also
>>>>>>> support
>>>>>>> Tomcat.
>>>>>>>
>>>>>>> I can try to deploy the war files on Tomcat instead. The error
> message
>>>>>>> comes
>>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>>> written.
>>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>>> ManifoldCF
>>>>>>> on Resin.
>>>>>>>
>>>>>>> Erlend
>>>>>>>
>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>
>>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>>> basic stuff.
>>>>>>>>
>>>>>>>> FWIW, when we first started the Jetty development, I had to do
>> nothing
>>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>>  It
>>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> I have placed the three war files into Resin's webapps directory
> and
>>>>>>>>> set
>>>>>>>>> the
>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>
>>>>>>>>> <system-property
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>
> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties
>> .xml"
>>>>>>>>> />
>>>>>>>>>
>>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>>> the
>>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>>> goes
>>>>>>>>> to
>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
>> ugly
>>>>>>>>> (I
>>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
>> click
>>>>>>>>> on
>>>>>>>>> a
>>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>>> headers
>>>>>>>>>
>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>
>>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
> it
>>>>>>>>> runs
>>>>>>>>> stable on Resin.
>>>>>>>>>
>>>>>>>>> E
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Erlend Garåsen
>>>>>>>>> Center for Information Technology Services
>>>>>>>>> University of Oslo
>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>> VIP:
>>>>>>>>> 31050
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>> 31050
>>>
>>
>>
>
>

Oracle as backend DB

[Jeffrey Guo <je...@semtific.com>]

Karl, thanks!  I am interested in implementing the Oracle port, if it has
not been done or is not being worked on.  I have looked at the class
DBInterfacePostgreSQL (similar ones for Derby, HSQLDB, and MySQL), if there
are any tips you could share with me, I'd greatly appreciate it, but
otherwise, I will post my questions as I move forward with the porting.  

On another note, does anyone on this list have experiences with Open
Pipeline (http://www.openpipeline.org/)?

Jeffrey

-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com] 
Sent: Tuesday, February 08, 2011 1:03 AM
To: connectors-user@incubator.apache.org
Subject: Re: Running ManifoldCF on Resin

I take it you mean the Oracle database, not the Oracle app server?
Open-source software relies on the contributions of individuals, not
really on road-maps.  ManifoldCF's database architecture is reasonably
well-suited for adding support for new kinds of databases, although in
my experience adding new database support is much harder than adding a
connector, because of the unique operating characteristics of
individual databases, which leads to the possibility of failure.
Nevertheless, if you want to volunteer, I'd be happy to lend you
whatever assistance you might need.

Karl

On Tue, Feb 8, 2011 at 12:41 AM, Jeffrey Guo <je...@semtific.com> wrote:
> Is Oracle on the road map to be supported?
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: Friday, February 04, 2011 7:31 AM
> To: connectors-user@incubator.apache.org
> Subject: Re: Running ManifoldCF on Resin
>
> Sure, feel free to create a ticket and try to come up with a patch.
> Since the DBInterfacePostgreSQL class is responsible for any remote
> communication, I would make those parameters local to the postgresql
> implementation.  So the parameters would be something like:
>
> org.apache.manifoldcf.postgresql.hostname
> org.apache.manifoldcf.postgresql.ssl
>
> or some such.  Then, you'd want to load the necessary parameters in
> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
> since that's not officially supported at this time.)
>
> The only problem I see is that the constructor architecture of
> Database.java may make it a challenge to load the appropriate
> parameters in the DBInterfacePostgreSQL constructor.  But I think you
> could make it work, and if not, we should probably fix this anyway to
> make it easier.
>
> Thanks,
> Karl
>
> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen <e....@usit.uio.no>
> wrote:
>>
>> You're right! I didn't read the command section in the documentation
>> properly. ManifoldCF does also run on Resin now. The only problem with
> Resin
>> is that the necessary resources, such as CSS, are not loaded. All the
> other
>> errors have disappeared, which means that it should be possible to run
the
>> application on Resin.
>>
>> Do you think I should create a ticket about running an external
PostgreSQL
>> database as well? I think I know which classes to modify. I see there are
>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>> not PG specific.
>>
>> I was thinking about extending the property list like this:
>> org.apache.manifoldcf.database.hostname (not required, defaults to
>> localhost)
>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>
>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>> not sure when it comes to MySQL. Haven't touched that database system for
>> years.
>>
>> Erlend
>>
>>
>> On 04.02.11 15.42, Karl Wright wrote:
>>>
>>> Ok, have you followed the instructions for a multi-process
>>> installation in how-to-build-and-deploy.html?
>>> You will need to create the database (using DBCreate), and install the
>>> schema (using Install), and register the crawler agent (using
>>> Register).  You will also need to create the properties.xml and
>>> logging.ini files by hand.
>>>
>>> Let me know if this helps.
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>  wrote:
>>>>
>>>> I guess the problem is that the database does not exist. When I
> installed
>>>> it
>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>> not
>>>> exist. I tried to build the database with the username and password
>>>> "test"
>>>> by the following command:
>>>> ./processes/script/executecommand.sh
org.apache.manifoldcf.core.DBCreate
>>>> test test
>>>>
>>>> And I also configured these two lines in my properties.xml file:
>>>>  <property name="org.apache.manifoldcf.database.username"
value="test"/>
>>>>  <property name="org.apache.manifoldcf.database.password"
value="test"/>
>>>>
>>>> I'm using the Derby database since I cannot see how I could contact an
>>>> external PostgreSQL server on another host. PG is one of several
> database
>>>> systems the university supports. I guess there should be an option for
>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>> another ticket to create.
>>>>
>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
it
>>>> on
>>>> Resin and then try to configure an external PostgreSQL server.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>
>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>> the problems are.  I'd start with not being able to reach the
>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>> UI is trying to present an error and resin is handling one of the java
>>>>> standard tags in a non-standard way.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
> Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> Resin is de facto application server at the university, but we also
>>>>>> support
>>>>>> Tomcat.
>>>>>>
>>>>>> I can try to deploy the war files on Tomcat instead. The error
message
>>>>>> comes
>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>> written.
>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>> ManifoldCF
>>>>>> on Resin.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>
>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>> basic stuff.
>>>>>>>
>>>>>>> FWIW, when we first started the Jetty development, I had to do
> nothing
>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>  It
>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> I have placed the three war files into Resin's webapps directory
and
>>>>>>>> set
>>>>>>>> the
>>>>>>>> following system variable inside resin.xml:
>>>>>>>>
>>>>>>>> <system-property
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>
org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties
> .xml"
>>>>>>>> />
>>>>>>>>
>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>> the
>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>> goes
>>>>>>>> to
>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
> ugly
>>>>>>>> (I
>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
> click
>>>>>>>> on
>>>>>>>> a
>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>> headers
>>>>>>>>
>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>
>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until
it
>>>>>>>> runs
>>>>>>>> stable on Resin.
>>>>>>>>
>>>>>>>> E
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
> VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
> 31050
>>
>
>

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

I take it you mean the Oracle database, not the Oracle app server?
Open-source software relies on the contributions of individuals, not
really on road-maps.  ManifoldCF's database architecture is reasonably
well-suited for adding support for new kinds of databases, although in
my experience adding new database support is much harder than adding a
connector, because of the unique operating characteristics of
individual databases, which leads to the possibility of failure.
Nevertheless, if you want to volunteer, I'd be happy to lend you
whatever assistance you might need.

Karl

On Tue, Feb 8, 2011 at 12:41 AM, Jeffrey Guo <je...@semtific.com> wrote:
> Is Oracle on the road map to be supported?
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: Friday, February 04, 2011 7:31 AM
> To: connectors-user@incubator.apache.org
> Subject: Re: Running ManifoldCF on Resin
>
> Sure, feel free to create a ticket and try to come up with a patch.
> Since the DBInterfacePostgreSQL class is responsible for any remote
> communication, I would make those parameters local to the postgresql
> implementation.  So the parameters would be something like:
>
> org.apache.manifoldcf.postgresql.hostname
> org.apache.manifoldcf.postgresql.ssl
>
> or some such.  Then, you'd want to load the necessary parameters in
> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
> since that's not officially supported at this time.)
>
> The only problem I see is that the constructor architecture of
> Database.java may make it a challenge to load the appropriate
> parameters in the DBInterfacePostgreSQL constructor.  But I think you
> could make it work, and if not, we should probably fix this anyway to
> make it easier.
>
> Thanks,
> Karl
>
> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen <e....@usit.uio.no>
> wrote:
>>
>> You're right! I didn't read the command section in the documentation
>> properly. ManifoldCF does also run on Resin now. The only problem with
> Resin
>> is that the necessary resources, such as CSS, are not loaded. All the
> other
>> errors have disappeared, which means that it should be possible to run the
>> application on Resin.
>>
>> Do you think I should create a ticket about running an external PostgreSQL
>> database as well? I think I know which classes to modify. I see there are
>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>> not PG specific.
>>
>> I was thinking about extending the property list like this:
>> org.apache.manifoldcf.database.hostname (not required, defaults to
>> localhost)
>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>
>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>> not sure when it comes to MySQL. Haven't touched that database system for
>> years.
>>
>> Erlend
>>
>>
>> On 04.02.11 15.42, Karl Wright wrote:
>>>
>>> Ok, have you followed the instructions for a multi-process
>>> installation in how-to-build-and-deploy.html?
>>> You will need to create the database (using DBCreate), and install the
>>> schema (using Install), and register the crawler agent (using
>>> Register).  You will also need to create the properties.xml and
>>> logging.ini files by hand.
>>>
>>> Let me know if this helps.
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>>  wrote:
>>>>
>>>> I guess the problem is that the database does not exist. When I
> installed
>>>> it
>>>> on Tomcat, I got an error message telling me that the schema TEST does
>>>> not
>>>> exist. I tried to build the database with the username and password
>>>> "test"
>>>> by the following command:
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
>>>> test test
>>>>
>>>> And I also configured these two lines in my properties.xml file:
>>>>  <property name="org.apache.manifoldcf.database.username" value="test"/>
>>>>  <property name="org.apache.manifoldcf.database.password" value="test"/>
>>>>
>>>> I'm using the Derby database since I cannot see how I could contact an
>>>> external PostgreSQL server on another host. PG is one of several
> database
>>>> systems the university supports. I guess there should be an option for
>>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>>> another ticket to create.
>>>>
>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it
>>>> on
>>>> Resin and then try to configure an external PostgreSQL server.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>
>>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>>> the problems are.  I'd start with not being able to reach the
>>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>>> UI is trying to present an error and resin is handling one of the java
>>>>> standard tags in a non-standard way.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
> Garåsen<e....@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> Resin is de facto application server at the university, but we also
>>>>>> support
>>>>>> Tomcat.
>>>>>>
>>>>>> I can try to deploy the war files on Tomcat instead. The error message
>>>>>> comes
>>>>>> from Resin, probably caused by a redirect after the http header is
>>>>>> written.
>>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>>> ManifoldCF
>>>>>> on Resin.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>
>>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>>> Resin as an application server is having difficulty doing some very
>>>>>>> basic stuff.
>>>>>>>
>>>>>>> FWIW, when we first started the Jetty development, I had to do
> nothing
>>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
>  It
>>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> I have placed the three war files into Resin's webapps directory and
>>>>>>>> set
>>>>>>>> the
>>>>>>>> following system variable inside resin.xml:
>>>>>>>>
>>>>>>>> <system-property
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties
> .xml"
>>>>>>>> />
>>>>>>>>
>>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>>> the
>>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>>> goes
>>>>>>>> to
>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
> ugly
>>>>>>>> (I
>>>>>>>> guess the path to CSS files etc. are broken). And when I try to
> click
>>>>>>>> on
>>>>>>>> a
>>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>>> headers
>>>>>>>>
>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>
>>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>>>>> runs
>>>>>>>> stable on Resin.
>>>>>>>>
>>>>>>>> E
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
> VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
> 31050
>>
>
>

RE: Running ManifoldCF on Resin

[Jeffrey Guo <je...@semtific.com>]

Is Oracle on the road map to be supported?

-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com] 
Sent: Friday, February 04, 2011 7:31 AM
To: connectors-user@incubator.apache.org
Subject: Re: Running ManifoldCF on Resin

Sure, feel free to create a ticket and try to come up with a patch.
Since the DBInterfacePostgreSQL class is responsible for any remote
communication, I would make those parameters local to the postgresql
implementation.  So the parameters would be something like:

org.apache.manifoldcf.postgresql.hostname
org.apache.manifoldcf.postgresql.ssl

or some such.  Then, you'd want to load the necessary parameters in
DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
since that's not officially supported at this time.)

The only problem I see is that the constructor architecture of
Database.java may make it a challenge to load the appropriate
parameters in the DBInterfacePostgreSQL constructor.  But I think you
could make it work, and if not, we should probably fix this anyway to
make it easier.

Thanks,
Karl

On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen <e....@usit.uio.no>
wrote:
>
> You're right! I didn't read the command section in the documentation
> properly. ManifoldCF does also run on Resin now. The only problem with
Resin
> is that the necessary resources, such as CSS, are not loaded. All the
other
> errors have disappeared, which means that it should be possible to run the
> application on Resin.
>
> Do you think I should create a ticket about running an external PostgreSQL
> database as well? I think I know which classes to modify. I see there are
> classes for MySQL as well, so perhaps the issue should be generally, i.e.
> not PG specific.
>
> I was thinking about extending the property list like this:
> org.apache.manifoldcf.database.hostname (not required, defaults to
> localhost)
> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>
> For PostgreSQL I think it should be quite easy to implement this, but I'm
> not sure when it comes to MySQL. Haven't touched that database system for
> years.
>
> Erlend
>
>
> On 04.02.11 15.42, Karl Wright wrote:
>>
>> Ok, have you followed the instructions for a multi-process
>> installation in how-to-build-and-deploy.html?
>> You will need to create the database (using DBCreate), and install the
>> schema (using Install), and register the crawler agent (using
>> Register).  You will also need to create the properties.xml and
>> logging.ini files by hand.
>>
>> Let me know if this helps.
>> Karl
>>
>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> I guess the problem is that the database does not exist. When I
installed
>>> it
>>> on Tomcat, I got an error message telling me that the schema TEST does
>>> not
>>> exist. I tried to build the database with the username and password
>>> "test"
>>> by the following command:
>>> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
>>> test test
>>>
>>> And I also configured these two lines in my properties.xml file:
>>>  <property name="org.apache.manifoldcf.database.username" value="test"/>
>>>  <property name="org.apache.manifoldcf.database.password" value="test"/>
>>>
>>> I'm using the Derby database since I cannot see how I could contact an
>>> external PostgreSQL server on another host. PG is one of several
database
>>> systems the university supports. I guess there should be an option for
>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>> another ticket to create.
>>>
>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it
>>> on
>>> Resin and then try to configure an external PostgreSQL server.
>>>
>>> Erlend
>>>
>>>
>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>
>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>> the problems are.  I'd start with not being able to reach the
>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>> UI is trying to present an error and resin is handling one of the java
>>>> standard tags in a non-standard way.
>>>>
>>>> Karl
>>>>
>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> Resin is de facto application server at the university, but we also
>>>>> support
>>>>> Tomcat.
>>>>>
>>>>> I can try to deploy the war files on Tomcat instead. The error message
>>>>> comes
>>>>> from Resin, probably caused by a redirect after the http header is
>>>>> written.
>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>> ManifoldCF
>>>>> on Resin.
>>>>>
>>>>> Erlend
>>>>>
>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>
>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>> Resin as an application server is having difficulty doing some very
>>>>>> basic stuff.
>>>>>>
>>>>>> FWIW, when we first started the Jetty development, I had to do
nothing
>>>>>> whatsoever to the crawler UI or the war packaging to make it work.
 It
>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>> Resin is indeed working up to Java standards.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>  wrote:
>>>>>>>
>>>>>>> I have placed the three war files into Resin's webapps directory and
>>>>>>> set
>>>>>>> the
>>>>>>> following system variable inside resin.xml:
>>>>>>>
>>>>>>> <system-property
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties
.xml"
>>>>>>> />
>>>>>>>
>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>> the
>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>> goes
>>>>>>> to
>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks
ugly
>>>>>>> (I
>>>>>>> guess the path to CSS files etc. are broken). And when I try to
click
>>>>>>> on
>>>>>>> a
>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>> headers
>>>>>>>
>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>
>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>>>> runs
>>>>>>> stable on Resin.
>>>>>>>
>>>>>>> E
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
31050
>

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Sure, feel free to create a ticket and try to come up with a patch.
Since the DBInterfacePostgreSQL class is responsible for any remote
communication, I would make those parameters local to the postgresql
implementation.  So the parameters would be something like:

org.apache.manifoldcf.postgresql.hostname
org.apache.manifoldcf.postgresql.ssl

or some such.  Then, you'd want to load the necessary parameters in
DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
since that's not officially supported at this time.)

The only problem I see is that the constructor architecture of
Database.java may make it a challenge to load the appropriate
parameters in the DBInterfacePostgreSQL constructor.  But I think you
could make it work, and if not, we should probably fix this anyway to
make it easier.

Thanks,
Karl

On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> You're right! I didn't read the command section in the documentation
> properly. ManifoldCF does also run on Resin now. The only problem with Resin
> is that the necessary resources, such as CSS, are not loaded. All the other
> errors have disappeared, which means that it should be possible to run the
> application on Resin.
>
> Do you think I should create a ticket about running an external PostgreSQL
> database as well? I think I know which classes to modify. I see there are
> classes for MySQL as well, so perhaps the issue should be generally, i.e.
> not PG specific.
>
> I was thinking about extending the property list like this:
> org.apache.manifoldcf.database.hostname (not required, defaults to
> localhost)
> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>
> For PostgreSQL I think it should be quite easy to implement this, but I'm
> not sure when it comes to MySQL. Haven't touched that database system for
> years.
>
> Erlend
>
>
> On 04.02.11 15.42, Karl Wright wrote:
>>
>> Ok, have you followed the instructions for a multi-process
>> installation in how-to-build-and-deploy.html?
>> You will need to create the database (using DBCreate), and install the
>> schema (using Install), and register the crawler agent (using
>> Register).  You will also need to create the properties.xml and
>> logging.ini files by hand.
>>
>> Let me know if this helps.
>> Karl
>>
>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> I guess the problem is that the database does not exist. When I installed
>>> it
>>> on Tomcat, I got an error message telling me that the schema TEST does
>>> not
>>> exist. I tried to build the database with the username and password
>>> "test"
>>> by the following command:
>>> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
>>> test test
>>>
>>> And I also configured these two lines in my properties.xml file:
>>>  <property name="org.apache.manifoldcf.database.username" value="test"/>
>>>  <property name="org.apache.manifoldcf.database.password" value="test"/>
>>>
>>> I'm using the Derby database since I cannot see how I could contact an
>>> external PostgreSQL server on another host. PG is one of several database
>>> systems the university supports. I guess there should be an option for
>>> configuring external hosts and optionally turn on SSL encryption. Maybe
>>> another ticket to create.
>>>
>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it
>>> on
>>> Resin and then try to configure an external PostgreSQL server.
>>>
>>> Erlend
>>>
>>>
>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>
>>>> Sure, if you want, create a ticket, and see if you can figure out what
>>>> the problems are.  I'd start with not being able to reach the
>>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>>> UI is trying to present an error and resin is handling one of the java
>>>> standard tags in a non-standard way.
>>>>
>>>> Karl
>>>>
>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> Resin is de facto application server at the university, but we also
>>>>> support
>>>>> Tomcat.
>>>>>
>>>>> I can try to deploy the war files on Tomcat instead. The error message
>>>>> comes
>>>>> from Resin, probably caused by a redirect after the http header is
>>>>> written.
>>>>> I can notify you, eventually create a patch, if I manage to run
>>>>> ManifoldCF
>>>>> on Resin.
>>>>>
>>>>> Erlend
>>>>>
>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>
>>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>>> Resin as an application server is having difficulty doing some very
>>>>>> basic stuff.
>>>>>>
>>>>>> FWIW, when we first started the Jetty development, I had to do nothing
>>>>>> whatsoever to the crawler UI or the war packaging to make it work.  It
>>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>>> Resin is indeed working up to Java standards.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>> Garåsen<e....@usit.uio.no>
>>>>>>  wrote:
>>>>>>>
>>>>>>> I have placed the three war files into Resin's webapps directory and
>>>>>>> set
>>>>>>> the
>>>>>>> following system variable inside resin.xml:
>>>>>>>
>>>>>>> <system-property
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>> />
>>>>>>>
>>>>>>> I have also created my own home directory for ManifoldCF and copied
>>>>>>> the
>>>>>>> necessary files from the example directory. When I start Resin and
>>>>>>> goes
>>>>>>> to
>>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly
>>>>>>> (I
>>>>>>> guess the path to CSS files etc. are broken). And when I try to click
>>>>>>> on
>>>>>>> a
>>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>>> headers
>>>>>>>
>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>
>>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>>>> runs
>>>>>>> stable on Resin.
>>>>>>>
>>>>>>> E
>>>>>>>
>>>>>>> --
>>>>>>> Erlend Garåsen
>>>>>>> Center for Information Technology Services
>>>>>>> University of Oslo
>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>>> 31050
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

You're right! I didn't read the command section in the documentation 
properly. ManifoldCF does also run on Resin now. The only problem with 
Resin is that the necessary resources, such as CSS, are not loaded. All 
the other errors have disappeared, which means that it should be 
possible to run the application on Resin.

Do you think I should create a ticket about running an external 
PostgreSQL database as well? I think I know which classes to modify. I 
see there are classes for MySQL as well, so perhaps the issue should be 
generally, i.e. not PG specific.

I was thinking about extending the property list like this:
org.apache.manifoldcf.database.hostname (not required, defaults to 
localhost)
org.apache.manifoldcf.database.ssl (not required, defaults to false)

For PostgreSQL I think it should be quite easy to implement this, but 
I'm not sure when it comes to MySQL. Haven't touched that database 
system for years.

Erlend


On 04.02.11 15.42, Karl Wright wrote:
> Ok, have you followed the instructions for a multi-process
> installation in how-to-build-and-deploy.html?
> You will need to create the database (using DBCreate), and install the
> schema (using Install), and register the crawler agent (using
> Register).  You will also need to create the properties.xml and
> logging.ini files by hand.
>
> Let me know if this helps.
> Karl
>
> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> I guess the problem is that the database does not exist. When I installed it
>> on Tomcat, I got an error message telling me that the schema TEST does not
>> exist. I tried to build the database with the username and password "test"
>> by the following command:
>> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
>> test test
>>
>> And I also configured these two lines in my properties.xml file:
>>   <property name="org.apache.manifoldcf.database.username" value="test"/>
>>   <property name="org.apache.manifoldcf.database.password" value="test"/>
>>
>> I'm using the Derby database since I cannot see how I could contact an
>> external PostgreSQL server on another host. PG is one of several database
>> systems the university supports. I guess there should be an option for
>> configuring external hosts and optionally turn on SSL encryption. Maybe
>> another ticket to create.
>>
>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it on
>> Resin and then try to configure an external PostgreSQL server.
>>
>> Erlend
>>
>>
>> On 03.02.11 16.32, Karl Wright wrote:
>>>
>>> Sure, if you want, create a ticket, and see if you can figure out what
>>> the problems are.  I'd start with not being able to reach the
>>> stylesheet, actually.  The "redirection" error sound like the crawler
>>> UI is trying to present an error and resin is handling one of the java
>>> standard tags in a non-standard way.
>>>
>>> Karl
>>>
>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>>   wrote:
>>>>
>>>> Resin is de facto application server at the university, but we also
>>>> support
>>>> Tomcat.
>>>>
>>>> I can try to deploy the war files on Tomcat instead. The error message
>>>> comes
>>>> from Resin, probably caused by a redirect after the http header is
>>>> written.
>>>> I can notify you, eventually create a patch, if I manage to run
>>>> ManifoldCF
>>>> on Resin.
>>>>
>>>> Erlend
>>>>
>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>
>>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>>> Resin as an application server is having difficulty doing some very
>>>>> basic stuff.
>>>>>
>>>>> FWIW, when we first started the Jetty development, I had to do nothing
>>>>> whatsoever to the crawler UI or the war packaging to make it work.  It
>>>>> just worked.  Your experiences with Resin are making me wonder if
>>>>> Resin is indeed working up to Java standards.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>>   wrote:
>>>>>>
>>>>>> I have placed the three war files into Resin's webapps directory and
>>>>>> set
>>>>>> the
>>>>>> following system variable inside resin.xml:
>>>>>>
>>>>>> <system-property
>>>>>>
>>>>>>
>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>> />
>>>>>>
>>>>>> I have also created my own home directory for ManifoldCF and copied the
>>>>>> necessary files from the example directory. When I start Resin and goes
>>>>>> to
>>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly
>>>>>> (I
>>>>>> guess the path to CSS files etc. are broken). And when I try to click
>>>>>> on
>>>>>> a
>>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>>> headers
>>>>>>
>>>>>> I find nothing in manifoldcf.log.
>>>>>>
>>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>>> runs
>>>>>> stable on Resin.
>>>>>>
>>>>>> E
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Ok, have you followed the instructions for a multi-process
installation in how-to-build-and-deploy.html?
You will need to create the database (using DBCreate), and install the
schema (using Install), and register the crawler agent (using
Register).  You will also need to create the properties.xml and
logging.ini files by hand.

Let me know if this helps.
Karl

On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I guess the problem is that the database does not exist. When I installed it
> on Tomcat, I got an error message telling me that the schema TEST does not
> exist. I tried to build the database with the username and password "test"
> by the following command:
> ./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate
> test test
>
> And I also configured these two lines in my properties.xml file:
>  <property name="org.apache.manifoldcf.database.username" value="test"/>
>  <property name="org.apache.manifoldcf.database.password" value="test"/>
>
> I'm using the Derby database since I cannot see how I could contact an
> external PostgreSQL server on another host. PG is one of several database
> systems the university supports. I guess there should be an option for
> configuring external hosts and optionally turn on SSL encryption. Maybe
> another ticket to create.
>
> But first I want to run ManifoldCF on Tomcat using Derby, then deploy it on
> Resin and then try to configure an external PostgreSQL server.
>
> Erlend
>
>
> On 03.02.11 16.32, Karl Wright wrote:
>>
>> Sure, if you want, create a ticket, and see if you can figure out what
>> the problems are.  I'd start with not being able to reach the
>> stylesheet, actually.  The "redirection" error sound like the crawler
>> UI is trying to present an error and resin is handling one of the java
>> standard tags in a non-standard way.
>>
>> Karl
>>
>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> Resin is de facto application server at the university, but we also
>>> support
>>> Tomcat.
>>>
>>> I can try to deploy the war files on Tomcat instead. The error message
>>> comes
>>> from Resin, probably caused by a redirect after the http header is
>>> written.
>>> I can notify you, eventually create a patch, if I manage to run
>>> ManifoldCF
>>> on Resin.
>>>
>>> Erlend
>>>
>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>
>>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>>> Resin as an application server is having difficulty doing some very
>>>> basic stuff.
>>>>
>>>> FWIW, when we first started the Jetty development, I had to do nothing
>>>> whatsoever to the crawler UI or the war packaging to make it work.  It
>>>> just worked.  Your experiences with Resin are making me wonder if
>>>> Resin is indeed working up to Java standards.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend Garåsen<e....@usit.uio.no>
>>>>  wrote:
>>>>>
>>>>> I have placed the three war files into Resin's webapps directory and
>>>>> set
>>>>> the
>>>>> following system variable inside resin.xml:
>>>>>
>>>>> <system-property
>>>>>
>>>>>
>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>> />
>>>>>
>>>>> I have also created my own home directory for ManifoldCF and copied the
>>>>> necessary files from the example directory. When I start Resin and goes
>>>>> to
>>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly
>>>>> (I
>>>>> guess the path to CSS files etc. are broken). And when I try to click
>>>>> on
>>>>> a
>>>>> link, e.g. "List Output Connections", I get the following error:
>>>>> java.lang.IllegalStateException: can't forward after writing HTTP
>>>>> headers
>>>>>
>>>>> I find nothing in manifoldcf.log.
>>>>>
>>>>> BTW: PostgreSQL is not configured. I will start using Derby until it
>>>>> runs
>>>>> stable on Resin.
>>>>>
>>>>> E
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

I guess the problem is that the database does not exist. When I 
installed it on Tomcat, I got an error message telling me that the 
schema TEST does not exist. I tried to build the database with the 
username and password "test" by the following command:
./processes/script/executecommand.sh org.apache.manifoldcf.core.DBCreate 
test test

And I also configured these two lines in my properties.xml file:
  <property name="org.apache.manifoldcf.database.username" value="test"/>
  <property name="org.apache.manifoldcf.database.password" value="test"/>

I'm using the Derby database since I cannot see how I could contact an 
external PostgreSQL server on another host. PG is one of several 
database systems the university supports. I guess there should be an 
option for configuring external hosts and optionally turn on SSL 
encryption. Maybe another ticket to create.

But first I want to run ManifoldCF on Tomcat using Derby, then deploy it 
on Resin and then try to configure an external PostgreSQL server.

Erlend


On 03.02.11 16.32, Karl Wright wrote:
> Sure, if you want, create a ticket, and see if you can figure out what
> the problems are.  I'd start with not being able to reach the
> stylesheet, actually.  The "redirection" error sound like the crawler
> UI is trying to present an error and resin is handling one of the java
> standard tags in a non-standard way.
>
> Karl
>
> On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> Resin is de facto application server at the university, but we also support
>> Tomcat.
>>
>> I can try to deploy the war files on Tomcat instead. The error message comes
>> from Resin, probably caused by a redirect after the http header is written.
>> I can notify you, eventually create a patch, if I manage to run ManifoldCF
>> on Resin.
>>
>> Erlend
>>
>> On 03.02.11 16.16, Karl Wright wrote:
>>>
>>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>>> Resin as an application server is having difficulty doing some very
>>> basic stuff.
>>>
>>> FWIW, when we first started the Jetty development, I had to do nothing
>>> whatsoever to the crawler UI or the war packaging to make it work.  It
>>> just worked.  Your experiences with Resin are making me wonder if
>>> Resin is indeed working up to Java standards.
>>>
>>> Karl
>>>
>>>
>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend Garåsen<e....@usit.uio.no>
>>>   wrote:
>>>>
>>>> I have placed the three war files into Resin's webapps directory and set
>>>> the
>>>> following system variable inside resin.xml:
>>>>
>>>> <system-property
>>>>
>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>> />
>>>>
>>>> I have also created my own home directory for ManifoldCF and copied the
>>>> necessary files from the example directory. When I start Resin and goes
>>>> to
>>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly (I
>>>> guess the path to CSS files etc. are broken). And when I try to click on
>>>> a
>>>> link, e.g. "List Output Connections", I get the following error:
>>>> java.lang.IllegalStateException: can't forward after writing HTTP headers
>>>>
>>>> I find nothing in manifoldcf.log.
>>>>
>>>> BTW: PostgreSQL is not configured. I will start using Derby until it runs
>>>> stable on Resin.
>>>>
>>>> E
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

Sure, if you want, create a ticket, and see if you can figure out what
the problems are.  I'd start with not being able to reach the
stylesheet, actually.  The "redirection" error sound like the crawler
UI is trying to present an error and resin is handling one of the java
standard tags in a non-standard way.

Karl

On Thu, Feb 3, 2011 at 10:28 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> Resin is de facto application server at the university, but we also support
> Tomcat.
>
> I can try to deploy the war files on Tomcat instead. The error message comes
> from Resin, probably caused by a redirect after the http header is written.
> I can notify you, eventually create a patch, if I manage to run ManifoldCF
> on Resin.
>
> Erlend
>
> On 03.02.11 16.16, Karl Wright wrote:
>>
>> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
>> Resin as an application server is having difficulty doing some very
>> basic stuff.
>>
>> FWIW, when we first started the Jetty development, I had to do nothing
>> whatsoever to the crawler UI or the war packaging to make it work.  It
>> just worked.  Your experiences with Resin are making me wonder if
>> Resin is indeed working up to Java standards.
>>
>> Karl
>>
>>
>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend Garåsen<e....@usit.uio.no>
>>  wrote:
>>>
>>> I have placed the three war files into Resin's webapps directory and set
>>> the
>>> following system variable inside resin.xml:
>>>
>>> <system-property
>>>
>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>> />
>>>
>>> I have also created my own home directory for ManifoldCF and copied the
>>> necessary files from the example directory. When I start Resin and goes
>>> to
>>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly (I
>>> guess the path to CSS files etc. are broken). And when I try to click on
>>> a
>>> link, e.g. "List Output Connections", I get the following error:
>>> java.lang.IllegalStateException: can't forward after writing HTTP headers
>>>
>>> I find nothing in manifoldcf.log.
>>>
>>> BTW: PostgreSQL is not configured. I will start using Derby until it runs
>>> stable on Resin.
>>>
>>> E
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>> 31050
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Re: Running ManifoldCF on Resin

[Erlend Garåsen <e....@usit.uio.no>]

Resin is de facto application server at the university, but we also 
support Tomcat.

I can try to deploy the war files on Tomcat instead. The error message 
comes from Resin, probably caused by a redirect after the http header is 
written. I can notify you, eventually create a patch, if I manage to run 
ManifoldCF on Resin.

Erlend

On 03.02.11 16.16, Karl Wright wrote:
> I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
> Resin as an application server is having difficulty doing some very
> basic stuff.
>
> FWIW, when we first started the Jetty development, I had to do nothing
> whatsoever to the crawler UI or the war packaging to make it work.  It
> just worked.  Your experiences with Resin are making me wonder if
> Resin is indeed working up to Java standards.
>
> Karl
>
>
> On Thu, Feb 3, 2011 at 9:31 AM, Erlend Garåsen<e....@usit.uio.no>  wrote:
>>
>> I have placed the three war files into Resin's webapps directory and set the
>> following system variable inside resin.xml:
>>
>> <system-property
>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>> />
>>
>> I have also created my own home directory for ManifoldCF and copied the
>> necessary files from the example directory. When I start Resin and goes to
>> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly (I
>> guess the path to CSS files etc. are broken). And when I try to click on a
>> link, e.g. "List Output Connections", I get the following error:
>> java.lang.IllegalStateException: can't forward after writing HTTP headers
>>
>> I find nothing in manifoldcf.log.
>>
>> BTW: PostgreSQL is not configured. I will start using Derby until it runs
>> stable on Resin.
>>
>> E
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Re: Running ManifoldCF on Resin

[Karl Wright <da...@gmail.com>]

I have never tried this on Resin, just on Tomcat and Jetty.  Clearly
Resin as an application server is having difficulty doing some very
basic stuff.

FWIW, when we first started the Jetty development, I had to do nothing
whatsoever to the crawler UI or the war packaging to make it work.  It
just worked.  Your experiences with Resin are making me wonder if
Resin is indeed working up to Java standards.

Karl


On Thu, Feb 3, 2011 at 9:31 AM, Erlend Garåsen <e....@usit.uio.no> wrote:
>
> I have placed the three war files into Resin's webapps directory and set the
> following system variable inside resin.xml:
>
> <system-property
> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
> />
>
> I have also created my own home directory for ManifoldCF and copied the
> necessary files from the example directory. When I start Resin and goes to
> ... mcf-crawler-ui/, the start page shows up, but the page looks ugly (I
> guess the path to CSS files etc. are broken). And when I try to click on a
> link, e.g. "List Output Connections", I get the following error:
> java.lang.IllegalStateException: can't forward after writing HTTP headers
>
> I find nothing in manifoldcf.log.
>
> BTW: PostgreSQL is not configured. I will start using Derby until it runs
> stable on Resin.
>
> E
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>