You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Vova Vysotskyi <vv...@gmail.com> on 2017/08/31 11:06:54 UTC
Disable Lilith SOCKET connection by default.
Hi all,
I am having troubles when running Drill unit tests on the node where
Hiveserver2 service is running. In this case, run hangs in the middle. It
is because Lilith and Hiveserver2 use the same port 10000.
I tried to change the Lilith port in the logback.xml, but we cannot change
this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
I have no other Ideas, how to avoid this issue without disabling Lilith or
stopping HS2.
Do you mind if I disable Lilith SOCKET connection by default?
--
Kind regards,
Volodymyr Vysotskyi
Re: multiple logback.xml in the project
Posted by Jinfeng Ni <jn...@apache.org>.
Makes sense to simply logback.xml to 2 : one for distribution, another one
for unit test.
I did not run into conflicts resulting from multiple logback.xml in the
past (possibly because I used InteliJ). When I run test under under
java-exec, I will modify logbak.xml under that package, and the change
seems to have effect when I run the unit test from IDE.
On Wed, Sep 6, 2017 at 2:06 PM, Paul Rogers <pr...@mapr.com> wrote:
> Great idea.
>
> Some background for others: logback.xml is meant to be a single, global
> resource.
>
> The challenge is that each Maven project has its own tests, so each need a
> logback.xml (or, more properly, logback-test.xml).
>
> When run under Maven, only one test/resources directory will be in the
> class path, so only one logback-test.xml file will be found.
>
> But, Eclipse seems to put all test/resources directories on the class
> path, resulting in conflicts. (I see this today for the many logback.xml
> files.)
>
> Ideally, the logback-test.xml files would:
>
> * Have a default level of ERROR
> * Omit the SOCKET appender
>
> This allows individual tests to use code to enable logging for the
> specific bits of code under tests. (See LogFixture and ExampleTest.)
>
> When we’ve tried to do this in the past, we found that developers who use
> Lilith want the SOCKET appender set at DEBUG or finer. Hence, we get tons
> of output to the console when tests run. This is because, by default,
> Logback does not provide a way to set separate levels for console and
> socket appenders. (There seem to be plugins required, and that is where I
> ran out of time…)
>
> So, in general, this would be a great improvement. We just need to find a
> path through the details mentioned above (plus any other issues that folks
> can identify.)
>
> Thanks,
>
> - Paul
>
>
> > On Sep 5, 2017, at 11:15 AM, Vlad Rozov <vr...@apache.org> wrote:
> >
> > I'd suggest to unify multiple logback.xml into a single or max 2 (one
> for unit test and one for release) file. Any objections/cons to the
> proposal?
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 9/4/17 05:00, Vova Vysotskyi wrote:
> >> Vlad Rozov, thanks for the idea to conditionally enable Lilith appender.
> >>
> >> I agree with you that the order of logback.xml files is undefined. I
> >> suppose that these files are used for debugging separate tests when
> >> logback config file is set manually (-Dlogback.configurationFile).
> >>
> >> 2017-09-01 22:49 GMT+00:00 Vlad Rozov <vr...@apache.org>:
> >>
> >>> I'd suggest to conditionally enable lilith appender in logback.xml
> based
> >>> on a property (for example drill.lilith.level). By default it can be
> set to
> >>> OFF and lilith appender not being added to loggers specified in the
> >>> logback.xml. An individual unit test can be executed with
> >>> -Ddrill.lilith.level=DEBUG, in which case, the appender will be in
> effect
> >>> if necessary.
> >>>
> >>> I have a related question - there are several logback.xml in the
> >>> drill/exec/java-exec: ./test/sh/logback.xml,
> ./test/resources/logback.xml
> >>> and ./main/resources/logback.xml. on top of that, java-exec depends on
> >>> drill-common-test that has it's own logback.xml and they all end up
> being
> >>> on the classpath during unit test. Which one is used during unit test
> >>> depends on their order in the classpath and my experience with maven
> >>> dependency says that the order is undefined. Do I miss something here
> and
> >>> there is an additional mechanism that resolves which logback
> configuration
> >>> is in use?
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>>
> >>>
> >>> On 9/1/17 10:34, Jinfeng Ni wrote:
> >>>
> >>>> If you run the whole unit test suite, you do not have to open Lilith
> at
> >>>> the
> >>>> same time. My experience is Lilith will significantly slow down unit
> test
> >>>> suite running, and even fail testcases, as Drill's code put many
> DEBUG log
> >>>> which chocked Lilith.
> >>>>
> >>>> Lilith is useful only if you are running individual testcases to debug
> >>>> certain issues.
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Sep 1, 2017 at 1:36 AM, Arina Yelchiyeva <
> >>>> arina.yelchiyeva@gmail.com
> >>>>
> >>>>> wrote:
> >>>>> I don't think that Lilith is critical for running unit tests. Since
> it is
> >>>>> used to view the logs, I guess you can disable it by default but
> please
> >>>>> leave the comment indicating the reason of the change.
> >>>>>
> >>>>> Kind regards
> >>>>> Arina
> >>>>>
> >>>>> On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> Hi all,
> >>>>>> I am having troubles when running Drill unit tests on the node where
> >>>>>> Hiveserver2 service is running. In this case, run hangs in the
> middle.
> >>>>>> It
> >>>>>> is because Lilith and Hiveserver2 use the same port 10000.
> >>>>>> I tried to change the Lilith port in the logback.xml, but we cannot
> >>>>>>
> >>>>> change
> >>>>>
> >>>>>> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
> >>>>>> I have no other Ideas, how to avoid this issue without disabling
> Lilith
> >>>>>>
> >>>>> or
> >>>>>
> >>>>>> stopping HS2.
> >>>>>>
> >>>>>> Do you mind if I disable Lilith SOCKET connection by default?
> >>>>>>
> >>>>>> --
> >>>>>> Kind regards,
> >>>>>> Volodymyr Vysotskyi
> >>>>>>
> >>>>>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>
> >>
> >
> >
> > Thank you,
> >
> > Vlad
>
>
Re: multiple logback.xml in the project
Posted by Paul Rogers <pr...@mapr.com>.
Great idea.
Some background for others: logback.xml is meant to be a single, global resource.
The challenge is that each Maven project has its own tests, so each need a logback.xml (or, more properly, logback-test.xml).
When run under Maven, only one test/resources directory will be in the class path, so only one logback-test.xml file will be found.
But, Eclipse seems to put all test/resources directories on the class path, resulting in conflicts. (I see this today for the many logback.xml files.)
Ideally, the logback-test.xml files would:
* Have a default level of ERROR
* Omit the SOCKET appender
This allows individual tests to use code to enable logging for the specific bits of code under tests. (See LogFixture and ExampleTest.)
When we’ve tried to do this in the past, we found that developers who use Lilith want the SOCKET appender set at DEBUG or finer. Hence, we get tons of output to the console when tests run. This is because, by default, Logback does not provide a way to set separate levels for console and socket appenders. (There seem to be plugins required, and that is where I ran out of time…)
So, in general, this would be a great improvement. We just need to find a path through the details mentioned above (plus any other issues that folks can identify.)
Thanks,
- Paul
> On Sep 5, 2017, at 11:15 AM, Vlad Rozov <vr...@apache.org> wrote:
>
> I'd suggest to unify multiple logback.xml into a single or max 2 (one for unit test and one for release) file. Any objections/cons to the proposal?
>
> Thank you,
>
> Vlad
>
>
> On 9/4/17 05:00, Vova Vysotskyi wrote:
>> Vlad Rozov, thanks for the idea to conditionally enable Lilith appender.
>>
>> I agree with you that the order of logback.xml files is undefined. I
>> suppose that these files are used for debugging separate tests when
>> logback config file is set manually (-Dlogback.configurationFile).
>>
>> 2017-09-01 22:49 GMT+00:00 Vlad Rozov <vr...@apache.org>:
>>
>>> I'd suggest to conditionally enable lilith appender in logback.xml based
>>> on a property (for example drill.lilith.level). By default it can be set to
>>> OFF and lilith appender not being added to loggers specified in the
>>> logback.xml. An individual unit test can be executed with
>>> -Ddrill.lilith.level=DEBUG, in which case, the appender will be in effect
>>> if necessary.
>>>
>>> I have a related question - there are several logback.xml in the
>>> drill/exec/java-exec: ./test/sh/logback.xml, ./test/resources/logback.xml
>>> and ./main/resources/logback.xml. on top of that, java-exec depends on
>>> drill-common-test that has it's own logback.xml and they all end up being
>>> on the classpath during unit test. Which one is used during unit test
>>> depends on their order in the classpath and my experience with maven
>>> dependency says that the order is undefined. Do I miss something here and
>>> there is an additional mechanism that resolves which logback configuration
>>> is in use?
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>>
>>> On 9/1/17 10:34, Jinfeng Ni wrote:
>>>
>>>> If you run the whole unit test suite, you do not have to open Lilith at
>>>> the
>>>> same time. My experience is Lilith will significantly slow down unit test
>>>> suite running, and even fail testcases, as Drill's code put many DEBUG log
>>>> which chocked Lilith.
>>>>
>>>> Lilith is useful only if you are running individual testcases to debug
>>>> certain issues.
>>>>
>>>>
>>>>
>>>> On Fri, Sep 1, 2017 at 1:36 AM, Arina Yelchiyeva <
>>>> arina.yelchiyeva@gmail.com
>>>>
>>>>> wrote:
>>>>> I don't think that Lilith is critical for running unit tests. Since it is
>>>>> used to view the logs, I guess you can disable it by default but please
>>>>> leave the comment indicating the reason of the change.
>>>>>
>>>>> Kind regards
>>>>> Arina
>>>>>
>>>>> On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>> I am having troubles when running Drill unit tests on the node where
>>>>>> Hiveserver2 service is running. In this case, run hangs in the middle.
>>>>>> It
>>>>>> is because Lilith and Hiveserver2 use the same port 10000.
>>>>>> I tried to change the Lilith port in the logback.xml, but we cannot
>>>>>>
>>>>> change
>>>>>
>>>>>> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
>>>>>> I have no other Ideas, how to avoid this issue without disabling Lilith
>>>>>>
>>>>> or
>>>>>
>>>>>> stopping HS2.
>>>>>>
>>>>>> Do you mind if I disable Lilith SOCKET connection by default?
>>>>>>
>>>>>> --
>>>>>> Kind regards,
>>>>>> Volodymyr Vysotskyi
>>>>>>
>>>>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>
>>
>
>
> Thank you,
>
> Vlad
multiple logback.xml in the project
Posted by Vlad Rozov <vr...@apache.org>.
I'd suggest to unify multiple logback.xml into a single or max 2 (one
for unit test and one for release) file. Any objections/cons to the
proposal?
Thank you,
Vlad
On 9/4/17 05:00, Vova Vysotskyi wrote:
> Vlad Rozov, thanks for the idea to conditionally enable Lilith appender.
>
> I agree with you that the order of logback.xml files is undefined. I
> suppose that these files are used for debugging separate tests when
> logback config file is set manually (-Dlogback.configurationFile).
>
> 2017-09-01 22:49 GMT+00:00 Vlad Rozov <vr...@apache.org>:
>
>> I'd suggest to conditionally enable lilith appender in logback.xml based
>> on a property (for example drill.lilith.level). By default it can be set to
>> OFF and lilith appender not being added to loggers specified in the
>> logback.xml. An individual unit test can be executed with
>> -Ddrill.lilith.level=DEBUG, in which case, the appender will be in effect
>> if necessary.
>>
>> I have a related question - there are several logback.xml in the
>> drill/exec/java-exec: ./test/sh/logback.xml, ./test/resources/logback.xml
>> and ./main/resources/logback.xml. on top of that, java-exec depends on
>> drill-common-test that has it's own logback.xml and they all end up being
>> on the classpath during unit test. Which one is used during unit test
>> depends on their order in the classpath and my experience with maven
>> dependency says that the order is undefined. Do I miss something here and
>> there is an additional mechanism that resolves which logback configuration
>> is in use?
>>
>> Thank you,
>>
>> Vlad
>>
>>
>>
>> On 9/1/17 10:34, Jinfeng Ni wrote:
>>
>>> If you run the whole unit test suite, you do not have to open Lilith at
>>> the
>>> same time. My experience is Lilith will significantly slow down unit test
>>> suite running, and even fail testcases, as Drill's code put many DEBUG log
>>> which chocked Lilith.
>>>
>>> Lilith is useful only if you are running individual testcases to debug
>>> certain issues.
>>>
>>>
>>>
>>> On Fri, Sep 1, 2017 at 1:36 AM, Arina Yelchiyeva <
>>> arina.yelchiyeva@gmail.com
>>>
>>>> wrote:
>>>> I don't think that Lilith is critical for running unit tests. Since it is
>>>> used to view the logs, I guess you can disable it by default but please
>>>> leave the comment indicating the reason of the change.
>>>>
>>>> Kind regards
>>>> Arina
>>>>
>>>> On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi all,
>>>>> I am having troubles when running Drill unit tests on the node where
>>>>> Hiveserver2 service is running. In this case, run hangs in the middle.
>>>>> It
>>>>> is because Lilith and Hiveserver2 use the same port 10000.
>>>>> I tried to change the Lilith port in the logback.xml, but we cannot
>>>>>
>>>> change
>>>>
>>>>> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
>>>>> I have no other Ideas, how to avoid this issue without disabling Lilith
>>>>>
>>>> or
>>>>
>>>>> stopping HS2.
>>>>>
>>>>> Do you mind if I disable Lilith SOCKET connection by default?
>>>>>
>>>>> --
>>>>> Kind regards,
>>>>> Volodymyr Vysotskyi
>>>>>
>>>>>
>> Thank you,
>>
>> Vlad
>>
>
>
Thank you,
Vlad
Re: Disable Lilith SOCKET connection by default.
Posted by Vova Vysotskyi <vv...@gmail.com>.
Vlad Rozov, thanks for the idea to conditionally enable Lilith appender.
I agree with you that the order of logback.xml files is undefined. I
suppose that these files are used for debugging separate tests when
logback config file is set manually (-Dlogback.configurationFile).
2017-09-01 22:49 GMT+00:00 Vlad Rozov <vr...@apache.org>:
> I'd suggest to conditionally enable lilith appender in logback.xml based
> on a property (for example drill.lilith.level). By default it can be set to
> OFF and lilith appender not being added to loggers specified in the
> logback.xml. An individual unit test can be executed with
> -Ddrill.lilith.level=DEBUG, in which case, the appender will be in effect
> if necessary.
>
> I have a related question - there are several logback.xml in the
> drill/exec/java-exec: ./test/sh/logback.xml, ./test/resources/logback.xml
> and ./main/resources/logback.xml. on top of that, java-exec depends on
> drill-common-test that has it's own logback.xml and they all end up being
> on the classpath during unit test. Which one is used during unit test
> depends on their order in the classpath and my experience with maven
> dependency says that the order is undefined. Do I miss something here and
> there is an additional mechanism that resolves which logback configuration
> is in use?
>
> Thank you,
>
> Vlad
>
>
>
> On 9/1/17 10:34, Jinfeng Ni wrote:
>
>> If you run the whole unit test suite, you do not have to open Lilith at
>> the
>> same time. My experience is Lilith will significantly slow down unit test
>> suite running, and even fail testcases, as Drill's code put many DEBUG log
>> which chocked Lilith.
>>
>> Lilith is useful only if you are running individual testcases to debug
>> certain issues.
>>
>>
>>
>> On Fri, Sep 1, 2017 at 1:36 AM, Arina Yelchiyeva <
>> arina.yelchiyeva@gmail.com
>>
>>> wrote:
>>> I don't think that Lilith is critical for running unit tests. Since it is
>>> used to view the logs, I guess you can disable it by default but please
>>> leave the comment indicating the reason of the change.
>>>
>>> Kind regards
>>> Arina
>>>
>>> On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com>
>>> wrote:
>>>
>>> Hi all,
>>>>
>>>> I am having troubles when running Drill unit tests on the node where
>>>> Hiveserver2 service is running. In this case, run hangs in the middle.
>>>> It
>>>> is because Lilith and Hiveserver2 use the same port 10000.
>>>> I tried to change the Lilith port in the logback.xml, but we cannot
>>>>
>>> change
>>>
>>>> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
>>>> I have no other Ideas, how to avoid this issue without disabling Lilith
>>>>
>>> or
>>>
>>>> stopping HS2.
>>>>
>>>> Do you mind if I disable Lilith SOCKET connection by default?
>>>>
>>>> --
>>>> Kind regards,
>>>> Volodymyr Vysotskyi
>>>>
>>>>
>
> Thank you,
>
> Vlad
>
--
Kind regards,
Volodymyr Vysotskyi
Re: Disable Lilith SOCKET connection by default.
Posted by Vlad Rozov <vr...@apache.org>.
I'd suggest to conditionally enable lilith appender in logback.xml based
on a property (for example drill.lilith.level). By default it can be set
to OFF and lilith appender not being added to loggers specified in the
logback.xml. An individual unit test can be executed with
-Ddrill.lilith.level=DEBUG, in which case, the appender will be in
effect if necessary.
I have a related question - there are several logback.xml in the
drill/exec/java-exec: ./test/sh/logback.xml,
./test/resources/logback.xml and ./main/resources/logback.xml. on top of
that, java-exec depends on drill-common-test that has it's own
logback.xml and they all end up being on the classpath during unit test.
Which one is used during unit test depends on their order in the
classpath and my experience with maven dependency says that the order is
undefined. Do I miss something here and there is an additional mechanism
that resolves which logback configuration is in use?
Thank you,
Vlad
On 9/1/17 10:34, Jinfeng Ni wrote:
> If you run the whole unit test suite, you do not have to open Lilith at the
> same time. My experience is Lilith will significantly slow down unit test
> suite running, and even fail testcases, as Drill's code put many DEBUG log
> which chocked Lilith.
>
> Lilith is useful only if you are running individual testcases to debug
> certain issues.
>
>
>
> On Fri, Sep 1, 2017 at 1:36 AM, Arina Yelchiyeva <arina.yelchiyeva@gmail.com
>> wrote:
>> I don't think that Lilith is critical for running unit tests. Since it is
>> used to view the logs, I guess you can disable it by default but please
>> leave the comment indicating the reason of the change.
>>
>> Kind regards
>> Arina
>>
>> On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I am having troubles when running Drill unit tests on the node where
>>> Hiveserver2 service is running. In this case, run hangs in the middle. It
>>> is because Lilith and Hiveserver2 use the same port 10000.
>>> I tried to change the Lilith port in the logback.xml, but we cannot
>> change
>>> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
>>> I have no other Ideas, how to avoid this issue without disabling Lilith
>> or
>>> stopping HS2.
>>>
>>> Do you mind if I disable Lilith SOCKET connection by default?
>>>
>>> --
>>> Kind regards,
>>> Volodymyr Vysotskyi
>>>
Thank you,
Vlad
Re: Disable Lilith SOCKET connection by default.
Posted by Jinfeng Ni <jn...@apache.org>.
If you run the whole unit test suite, you do not have to open Lilith at the
same time. My experience is Lilith will significantly slow down unit test
suite running, and even fail testcases, as Drill's code put many DEBUG log
which chocked Lilith.
Lilith is useful only if you are running individual testcases to debug
certain issues.
On Fri, Sep 1, 2017 at 1:36 AM, Arina Yelchiyeva <arina.yelchiyeva@gmail.com
> wrote:
> I don't think that Lilith is critical for running unit tests. Since it is
> used to view the logs, I guess you can disable it by default but please
> leave the comment indicating the reason of the change.
>
> Kind regards
> Arina
>
> On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com> wrote:
>
> > Hi all,
> >
> > I am having troubles when running Drill unit tests on the node where
> > Hiveserver2 service is running. In this case, run hangs in the middle. It
> > is because Lilith and Hiveserver2 use the same port 10000.
> > I tried to change the Lilith port in the logback.xml, but we cannot
> change
> > this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
> > I have no other Ideas, how to avoid this issue without disabling Lilith
> or
> > stopping HS2.
> >
> > Do you mind if I disable Lilith SOCKET connection by default?
> >
> > --
> > Kind regards,
> > Volodymyr Vysotskyi
> >
>
Re: Disable Lilith SOCKET connection by default.
Posted by Arina Yelchiyeva <ar...@gmail.com>.
I don't think that Lilith is critical for running unit tests. Since it is
used to view the logs, I guess you can disable it by default but please
leave the comment indicating the reason of the change.
Kind regards
Arina
On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi <vv...@gmail.com> wrote:
> Hi all,
>
> I am having troubles when running Drill unit tests on the node where
> Hiveserver2 service is running. In this case, run hangs in the middle. It
> is because Lilith and Hiveserver2 use the same port 10000.
> I tried to change the Lilith port in the logback.xml, but we cannot change
> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
> I have no other Ideas, how to avoid this issue without disabling Lilith or
> stopping HS2.
>
> Do you mind if I disable Lilith SOCKET connection by default?
>
> --
> Kind regards,
> Volodymyr Vysotskyi
>