You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Muhammed Olgun <mh...@gmail.com> on 2014/10/17 10:40:58 UTC
Logging
Hi,
I was thinking that we should add a feature which shows manifoldcf.log in
MCF's control panel. What do you think about it?
Muhammed
Re: Logging
Posted by Karl Wright <da...@gmail.com>.
Hi Richard,
The activities log contains both a "code" and a "description". The code is
supposedly something you can search on. But for each activity type one
expects that the set of codes may well differ, and there is nothing to
require or force consistency (even when the semantics may be similar).
I'll think about ways we can try to regularize the code values across
activity types to the extent possible.
Thanks,
Karl
On Fri, Oct 17, 2014 at 10:13 AM, Nichols, Richard <
Richard.Nichols@coriant.com> wrote:
> Just a thought.
> If the document rejection messages had a standard format, then matching a
> document to its rejection reason could be fairly simple, and possibly even
> displayed by the GUI (in the future).
>
> Rick
>
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: Friday, October 17, 2014 7:00 AM
> To: dev
> Subject: Re: Logging
>
> I've created a ticket for this work: CONNECTORS-1077. Some work has
> already been done (e.g. for the document filter transformer), but if you
> would like to help work on this also I would be happy to have a
> collaborator.
>
> Thanks,
> Karl
>
>
> On Fri, Oct 17, 2014 at 7:53 AM, Muhammed Olgun <mh...@gmail.com>
> wrote:
>
> > Hi Karl,
> >
> > You are right. Thank you.
> >
> > Muhammed
> >
> > On 17 Oct 2014, at 13:49, Karl Wright <da...@gmail.com> wrote:
> >
> > > We *have* a feature that is cross-cluster and is meant for precisely
> this
> > > purpose. This is the history table. If Tika exceptions are not logged
> > as
> > > the result of a transformation attempt in this table, they should be.
> > >
> > > We've in general been slow to add support for activity logging in our
> > > connectors; perhaps we should put our effort towards fixing that
> problem.
> > >
> > > Thanks,
> > > Karl
> > >
> > >
> > > On Fri, Oct 17, 2014 at 6:46 AM, Karl Wright <da...@gmail.com>
> wrote:
> > >
> > >> Hi Muhammed,
> > >>
> > >> In a cluster, where do you expect to look for the log file? The log
> > will
> > >> only apply to the cluster member running the crawler UI.
> > >>
> > >> Karl
> > >>
> > >>
> > >> On Fri, Oct 17, 2014 at 5:59 AM, Muhammed Olgun <mh...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Karl,
> > >>>
> > >>> I think that we can also read logging.ini file from the MCF and show,
> > for
> > >>> instance, last 50 lines of log file.
> > >>>
> > >>> We can see that is the document processed or not from the document
> > status
> > >>> report but we cannot see that why the document isn't processed. May
> be
> > >>> document isn't processed because there was a tika exception or
> > something
> > >>> different. We have to check the log file.
> > >>>
> > >>> If we can add such a feature, we will check the log file easily and
> we
> > >>> won’t have to access the server to check the log file.
> > >>>
> > >>> Muhammed
> > >>>
> > >>>
> > >>> On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
> > >>>
> > >>>> Hi Muhammed,
> > >>>>
> > >>>> The problems with that idea are several:
> > >>>>
> > >>>> (1) You don't know where the logs are, or if/when they rolled. All
> of
> > >>> that
> > >>>> is configured in the logging.ini file.
> > >>>> (2) The logs may be very very large.
> > >>>> (3) The logs are meant for a different sort of user than the typical
> > UI
> > >>>> user.
> > >>>>
> > >>>> The only programs I know who allow you to view logs capture them
> > >>> explicitly
> > >>>> themselves, for example Jenkins.
> > >>>>
> > >>>> It is probably far better to capture more kinds of activities in the
> > >>> simple
> > >>>> history than to try to provide a log viewer in the UI. Can you
> > describe
> > >>>> what you would want to look for?
> > >>>>
> > >>>> Karl
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh.olgun@gmail.com
> >
> > >>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> I was thinking that we should add a feature which shows
> > manifoldcf.log
> > >>> in
> > >>>>> MCF's control panel. What do you think about it?
> > >>>>>
> > >>>>> Muhammed
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>
> ============================================================
> The information contained in this message may be privileged
> and confidential and protected from disclosure. If the reader
> of this message is not the intended recipient, or an employee
> or agent responsible for delivering this message to the
> intended recipient, you are hereby notified that any reproduction,
> dissemination or distribution of this communication is strictly
> prohibited. If you have received this communication in error,
> please notify us immediately by replying to the message and
> deleting it from your computer. Thank you. Coriant-Tellabs
> ============================================================
>
RE: Logging
Posted by "Nichols, Richard" <Ri...@coriant.com>.
Just a thought.
If the document rejection messages had a standard format, then matching a document to its rejection reason could be fairly simple, and possibly even displayed by the GUI (in the future).
Rick
-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com]
Sent: Friday, October 17, 2014 7:00 AM
To: dev
Subject: Re: Logging
I've created a ticket for this work: CONNECTORS-1077. Some work has
already been done (e.g. for the document filter transformer), but if you
would like to help work on this also I would be happy to have a
collaborator.
Thanks,
Karl
On Fri, Oct 17, 2014 at 7:53 AM, Muhammed Olgun <mh...@gmail.com> wrote:
> Hi Karl,
>
> You are right. Thank you.
>
> Muhammed
>
> On 17 Oct 2014, at 13:49, Karl Wright <da...@gmail.com> wrote:
>
> > We *have* a feature that is cross-cluster and is meant for precisely this
> > purpose. This is the history table. If Tika exceptions are not logged
> as
> > the result of a transformation attempt in this table, they should be.
> >
> > We've in general been slow to add support for activity logging in our
> > connectors; perhaps we should put our effort towards fixing that problem.
> >
> > Thanks,
> > Karl
> >
> >
> > On Fri, Oct 17, 2014 at 6:46 AM, Karl Wright <da...@gmail.com> wrote:
> >
> >> Hi Muhammed,
> >>
> >> In a cluster, where do you expect to look for the log file? The log
> will
> >> only apply to the cluster member running the crawler UI.
> >>
> >> Karl
> >>
> >>
> >> On Fri, Oct 17, 2014 at 5:59 AM, Muhammed Olgun <mh...@gmail.com>
> >> wrote:
> >>
> >>> Hi Karl,
> >>>
> >>> I think that we can also read logging.ini file from the MCF and show,
> for
> >>> instance, last 50 lines of log file.
> >>>
> >>> We can see that is the document processed or not from the document
> status
> >>> report but we cannot see that why the document isn't processed. May be
> >>> document isn't processed because there was a tika exception or
> something
> >>> different. We have to check the log file.
> >>>
> >>> If we can add such a feature, we will check the log file easily and we
> >>> won’t have to access the server to check the log file.
> >>>
> >>> Muhammed
> >>>
> >>>
> >>> On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
> >>>
> >>>> Hi Muhammed,
> >>>>
> >>>> The problems with that idea are several:
> >>>>
> >>>> (1) You don't know where the logs are, or if/when they rolled. All of
> >>> that
> >>>> is configured in the logging.ini file.
> >>>> (2) The logs may be very very large.
> >>>> (3) The logs are meant for a different sort of user than the typical
> UI
> >>>> user.
> >>>>
> >>>> The only programs I know who allow you to view logs capture them
> >>> explicitly
> >>>> themselves, for example Jenkins.
> >>>>
> >>>> It is probably far better to capture more kinds of activities in the
> >>> simple
> >>>> history than to try to provide a log viewer in the UI. Can you
> describe
> >>>> what you would want to look for?
> >>>>
> >>>> Karl
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I was thinking that we should add a feature which shows
> manifoldcf.log
> >>> in
> >>>>> MCF's control panel. What do you think about it?
> >>>>>
> >>>>> Muhammed
> >>>>>
> >>>
> >>>
> >>
>
>
============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Coriant-Tellabs
============================================================
Re: Logging
Posted by Karl Wright <da...@gmail.com>.
I've created a ticket for this work: CONNECTORS-1077. Some work has
already been done (e.g. for the document filter transformer), but if you
would like to help work on this also I would be happy to have a
collaborator.
Thanks,
Karl
On Fri, Oct 17, 2014 at 7:53 AM, Muhammed Olgun <mh...@gmail.com> wrote:
> Hi Karl,
>
> You are right. Thank you.
>
> Muhammed
>
> On 17 Oct 2014, at 13:49, Karl Wright <da...@gmail.com> wrote:
>
> > We *have* a feature that is cross-cluster and is meant for precisely this
> > purpose. This is the history table. If Tika exceptions are not logged
> as
> > the result of a transformation attempt in this table, they should be.
> >
> > We've in general been slow to add support for activity logging in our
> > connectors; perhaps we should put our effort towards fixing that problem.
> >
> > Thanks,
> > Karl
> >
> >
> > On Fri, Oct 17, 2014 at 6:46 AM, Karl Wright <da...@gmail.com> wrote:
> >
> >> Hi Muhammed,
> >>
> >> In a cluster, where do you expect to look for the log file? The log
> will
> >> only apply to the cluster member running the crawler UI.
> >>
> >> Karl
> >>
> >>
> >> On Fri, Oct 17, 2014 at 5:59 AM, Muhammed Olgun <mh...@gmail.com>
> >> wrote:
> >>
> >>> Hi Karl,
> >>>
> >>> I think that we can also read logging.ini file from the MCF and show,
> for
> >>> instance, last 50 lines of log file.
> >>>
> >>> We can see that is the document processed or not from the document
> status
> >>> report but we cannot see that why the document isn't processed. May be
> >>> document isn't processed because there was a tika exception or
> something
> >>> different. We have to check the log file.
> >>>
> >>> If we can add such a feature, we will check the log file easily and we
> >>> won’t have to access the server to check the log file.
> >>>
> >>> Muhammed
> >>>
> >>>
> >>> On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
> >>>
> >>>> Hi Muhammed,
> >>>>
> >>>> The problems with that idea are several:
> >>>>
> >>>> (1) You don't know where the logs are, or if/when they rolled. All of
> >>> that
> >>>> is configured in the logging.ini file.
> >>>> (2) The logs may be very very large.
> >>>> (3) The logs are meant for a different sort of user than the typical
> UI
> >>>> user.
> >>>>
> >>>> The only programs I know who allow you to view logs capture them
> >>> explicitly
> >>>> themselves, for example Jenkins.
> >>>>
> >>>> It is probably far better to capture more kinds of activities in the
> >>> simple
> >>>> history than to try to provide a log viewer in the UI. Can you
> describe
> >>>> what you would want to look for?
> >>>>
> >>>> Karl
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I was thinking that we should add a feature which shows
> manifoldcf.log
> >>> in
> >>>>> MCF's control panel. What do you think about it?
> >>>>>
> >>>>> Muhammed
> >>>>>
> >>>
> >>>
> >>
>
>
Re: Logging
Posted by Muhammed Olgun <mh...@gmail.com>.
Hi Karl,
You are right. Thank you.
Muhammed
On 17 Oct 2014, at 13:49, Karl Wright <da...@gmail.com> wrote:
> We *have* a feature that is cross-cluster and is meant for precisely this
> purpose. This is the history table. If Tika exceptions are not logged as
> the result of a transformation attempt in this table, they should be.
>
> We've in general been slow to add support for activity logging in our
> connectors; perhaps we should put our effort towards fixing that problem.
>
> Thanks,
> Karl
>
>
> On Fri, Oct 17, 2014 at 6:46 AM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Muhammed,
>>
>> In a cluster, where do you expect to look for the log file? The log will
>> only apply to the cluster member running the crawler UI.
>>
>> Karl
>>
>>
>> On Fri, Oct 17, 2014 at 5:59 AM, Muhammed Olgun <mh...@gmail.com>
>> wrote:
>>
>>> Hi Karl,
>>>
>>> I think that we can also read logging.ini file from the MCF and show, for
>>> instance, last 50 lines of log file.
>>>
>>> We can see that is the document processed or not from the document status
>>> report but we cannot see that why the document isn't processed. May be
>>> document isn't processed because there was a tika exception or something
>>> different. We have to check the log file.
>>>
>>> If we can add such a feature, we will check the log file easily and we
>>> won’t have to access the server to check the log file.
>>>
>>> Muhammed
>>>
>>>
>>> On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi Muhammed,
>>>>
>>>> The problems with that idea are several:
>>>>
>>>> (1) You don't know where the logs are, or if/when they rolled. All of
>>> that
>>>> is configured in the logging.ini file.
>>>> (2) The logs may be very very large.
>>>> (3) The logs are meant for a different sort of user than the typical UI
>>>> user.
>>>>
>>>> The only programs I know who allow you to view logs capture them
>>> explicitly
>>>> themselves, for example Jenkins.
>>>>
>>>> It is probably far better to capture more kinds of activities in the
>>> simple
>>>> history than to try to provide a log viewer in the UI. Can you describe
>>>> what you would want to look for?
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>> On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com>
>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I was thinking that we should add a feature which shows manifoldcf.log
>>> in
>>>>> MCF's control panel. What do you think about it?
>>>>>
>>>>> Muhammed
>>>>>
>>>
>>>
>>
Re: Logging
Posted by Karl Wright <da...@gmail.com>.
We *have* a feature that is cross-cluster and is meant for precisely this
purpose. This is the history table. If Tika exceptions are not logged as
the result of a transformation attempt in this table, they should be.
We've in general been slow to add support for activity logging in our
connectors; perhaps we should put our effort towards fixing that problem.
Thanks,
Karl
On Fri, Oct 17, 2014 at 6:46 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Muhammed,
>
> In a cluster, where do you expect to look for the log file? The log will
> only apply to the cluster member running the crawler UI.
>
> Karl
>
>
> On Fri, Oct 17, 2014 at 5:59 AM, Muhammed Olgun <mh...@gmail.com>
> wrote:
>
>> Hi Karl,
>>
>> I think that we can also read logging.ini file from the MCF and show, for
>> instance, last 50 lines of log file.
>>
>> We can see that is the document processed or not from the document status
>> report but we cannot see that why the document isn't processed. May be
>> document isn't processed because there was a tika exception or something
>> different. We have to check the log file.
>>
>> If we can add such a feature, we will check the log file easily and we
>> won’t have to access the server to check the log file.
>>
>> Muhammed
>>
>>
>> On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
>>
>> > Hi Muhammed,
>> >
>> > The problems with that idea are several:
>> >
>> > (1) You don't know where the logs are, or if/when they rolled. All of
>> that
>> > is configured in the logging.ini file.
>> > (2) The logs may be very very large.
>> > (3) The logs are meant for a different sort of user than the typical UI
>> > user.
>> >
>> > The only programs I know who allow you to view logs capture them
>> explicitly
>> > themselves, for example Jenkins.
>> >
>> > It is probably far better to capture more kinds of activities in the
>> simple
>> > history than to try to provide a log viewer in the UI. Can you describe
>> > what you would want to look for?
>> >
>> > Karl
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I was thinking that we should add a feature which shows manifoldcf.log
>> in
>> >> MCF's control panel. What do you think about it?
>> >>
>> >> Muhammed
>> >>
>>
>>
>
Re: Logging
Posted by Karl Wright <da...@gmail.com>.
Hi Muhammed,
In a cluster, where do you expect to look for the log file? The log will
only apply to the cluster member running the crawler UI.
Karl
On Fri, Oct 17, 2014 at 5:59 AM, Muhammed Olgun <mh...@gmail.com> wrote:
> Hi Karl,
>
> I think that we can also read logging.ini file from the MCF and show, for
> instance, last 50 lines of log file.
>
> We can see that is the document processed or not from the document status
> report but we cannot see that why the document isn't processed. May be
> document isn't processed because there was a tika exception or something
> different. We have to check the log file.
>
> If we can add such a feature, we will check the log file easily and we
> won’t have to access the server to check the log file.
>
> Muhammed
>
>
> On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
>
> > Hi Muhammed,
> >
> > The problems with that idea are several:
> >
> > (1) You don't know where the logs are, or if/when they rolled. All of
> that
> > is configured in the logging.ini file.
> > (2) The logs may be very very large.
> > (3) The logs are meant for a different sort of user than the typical UI
> > user.
> >
> > The only programs I know who allow you to view logs capture them
> explicitly
> > themselves, for example Jenkins.
> >
> > It is probably far better to capture more kinds of activities in the
> simple
> > history than to try to provide a log viewer in the UI. Can you describe
> > what you would want to look for?
> >
> > Karl
> >
> >
> >
> > On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> I was thinking that we should add a feature which shows manifoldcf.log
> in
> >> MCF's control panel. What do you think about it?
> >>
> >> Muhammed
> >>
>
>
Re: Logging
Posted by Muhammed Olgun <mh...@gmail.com>.
Hi Karl,
I think that we can also read logging.ini file from the MCF and show, for instance, last 50 lines of log file.
We can see that is the document processed or not from the document status report but we cannot see that why the document isn't processed. May be document isn't processed because there was a tika exception or something different. We have to check the log file.
If we can add such a feature, we will check the log file easily and we won’t have to access the server to check the log file.
Muhammed
On 17 Oct 2014, at 11:53, Karl Wright <da...@gmail.com> wrote:
> Hi Muhammed,
>
> The problems with that idea are several:
>
> (1) You don't know where the logs are, or if/when they rolled. All of that
> is configured in the logging.ini file.
> (2) The logs may be very very large.
> (3) The logs are meant for a different sort of user than the typical UI
> user.
>
> The only programs I know who allow you to view logs capture them explicitly
> themselves, for example Jenkins.
>
> It is probably far better to capture more kinds of activities in the simple
> history than to try to provide a log viewer in the UI. Can you describe
> what you would want to look for?
>
> Karl
>
>
>
> On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com> wrote:
>
>> Hi,
>>
>> I was thinking that we should add a feature which shows manifoldcf.log in
>> MCF's control panel. What do you think about it?
>>
>> Muhammed
>>
Re: Logging
Posted by Karl Wright <da...@gmail.com>.
Hi Muhammed,
The problems with that idea are several:
(1) You don't know where the logs are, or if/when they rolled. All of that
is configured in the logging.ini file.
(2) The logs may be very very large.
(3) The logs are meant for a different sort of user than the typical UI
user.
The only programs I know who allow you to view logs capture them explicitly
themselves, for example Jenkins.
It is probably far better to capture more kinds of activities in the simple
history than to try to provide a log viewer in the UI. Can you describe
what you would want to look for?
Karl
On Fri, Oct 17, 2014 at 4:40 AM, Muhammed Olgun <mh...@gmail.com> wrote:
> Hi,
>
> I was thinking that we should add a feature which shows manifoldcf.log in
> MCF's control panel. What do you think about it?
>
> Muhammed
>