You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by James Turton <dz...@apache.org> on 2023/08/21 14:36:37 UTC
Re: drill tests not passing
Hi Mike
I took a look at the build log that you shared with me recently and what
happened in that run was OOM (first report on line 9348).
| 9355 2198813 ERROR [UserServer-1]
[org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
communication. 9355 Connection: /127.0.0.1:31046 <-->
/127.0.0.1:39558 (user server). Closing connection.
9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
|
My command to run Drill tests set memory limits as follows.
|alias mvn-test-drill="mvn test \
-Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
-Duser.region=US\" \
-DmemoryMb=2560 -DdirectMemoryMb=2560 \
-DforkCount=2"
|
Hope this helps...
James
On 2023/07/25 00:00, Mike Beckerle wrote:
> Hi drill devs,
>
> I'm still stuck on this problem. Can anyone suggest a way past this?
>
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair |www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> Owl Cyber Defense |www.owlcyberdefense.com
>
>
>
> On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle<mb...@apache.org> wrote:
>
>> Looks like I attached the wrong file. Doing too many things at once.
>>
>> The correct file is attached here.
>>
>>
>>
>> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle<mb...@apache.org>
>> wrote:
>>
>>> Update: I did a clean and install -DskipTests=true.
>>>
>>> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>>>
>>> But alas, it still fails, this time the failure unique and is only in
>>> "Java Execution Engine"
>>>
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
>>> (unpack-vector-types) on project drill-java-exec: Artifact has not been
>>> packaged yet. When used on reactor artifact, unpack should be executed
>>> after packaging: see MDEP-98. -> [Help 1]
>>>
>>> The command and complete trace output are below.
>>>
>>> I need assistance on how to proceed.
>>>
>>> Complete trace from the mvn test is attached.
>>>
>>>
>>> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle<mb...@apache.org>
>>> wrote:
>>>
>>>> To answer questions:
>>>>
>>>> 1. Paul: This is a 100% stock build. All I have done is clone the repo
>>>> (master branch). Make a new git branch (in case I make future changes). Try
>>>> to build (success) and test (failed so far).
>>>>
>>>> 2. James: The /opt/drill directory I created is owned by my userid and
>>>> has full read/write access for all the development activities. I just put
>>>> it there so it would have a shorter path to fix the first Hive-related
>>>> glitch I encountered with the Linux 255 limit on file pathname length.
>>>>
>>>> I will try the suggested maven command line for non-UTC and see if
>>>> things improve.
>>>>
>>>> The challenge for me as a newby is how do I know if I have everything
>>>> properly configured?
>>>>
>>>> Can I just turn off building and testing of the Hive-related stuff in
>>>> some supported/well-known way?
>>>>
>>>> If so, I would suggest I'd like to turn off not just Hive, but *as much
>>>> as possible*. I really just need the embedded drill to work.
>>>>
>>>> I would agree with @Charles Givre<cg...@gmail.com> that a contrib
>>>> package addition is the ideal approach and that's what I'll be attempting.
>>>>
>>>> -mikeb
>>>>
>>>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre<cg...@gmail.com> wrote:
>>>>
>>>>> I'll add some heresy here... IMHO, for the purposes of developing a
>>>>> DFDL extension, you probably don't need all the Drill tests to run. For
>>>>> your project, my suggestion would be to add a module to the contrib package
>>>>> and that way your changes are relatively self contained.
>>>>> Best,
>>>>> -- C
>>>>>
>>>>>
>>>>>
>>>>>> On Jul 13, 2023, at 10:27 AM, James Turton<dz...@apache.org> wrote:
>>>>>>
>>>>>> Hi Mike
>>>>>>
>>>>>> Here's the command line I use to run tests on a machine that's not in
>>>>> the UTC time zone (plus some unrelated memory size arguments).
>>>>>> mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>>>>> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>>>>>> I have one other question to add to Paul's comments - does the OS
>>>>> user that you're running Maven under have write access to all of the source
>>>>> tree that you put at /opt/drill?
>>>>>> On 2023/07/11 22:12, Paul Rogers wrote:
>>>>>>> Hi Mike,
>>>>>>>
>>>>>>> A quick glance at the log suggests a failure in the tests for the
>>>>> JSON
>>>>>>> reader, in the Mongo extended types. Drill's date/time support has
>>>>>>> historically been fragile. Some tests only work if your machine is
>>>>> set to
>>>>>>> use the UTC time zone (or Java is told to pretend that the time is
>>>>> UTC.)
>>>>>>> The Mongo types test failure seems to be around a date/time test so
>>>>> maybe
>>>>>>> this is the issue?
>>>>>>>
>>>>>>> There are also failures indicating that the Drillbit (Drill server)
>>>>> died.
>>>>>>> Not sure how this can happen, as tests run Drill embedded (or used
>>>>> to.)
>>>>>>> Looking earlier in the logs, it seems that the Drillbit didn't start
>>>>> due to
>>>>>>> UDF (user-defined function) failures:
>>>>>>>
>>>>>>> Found duplicated function in drill-custom-lower.jar:
>>>>>>> custom_lower(VARCHAR-REQUIRED)
>>>>>>> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>>>>>>>
>>>>>>> Not sure how this could occur: it should have failed in all builds.
>>>>>>>
>>>>>>> Also:
>>>>>>>
>>>>>>> File
>>>>>>>
>>>>> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
>>>>>>> does not exist on file systemfile:///
>>>>>>>
>>>>>>> This is complaining that Drill needs the source code (not just class
>>>>> file)
>>>>>>> for its built-in functions. Again, this should not fail in a standard
>>>>>>> build, because if it did, it would fail in all builds.
>>>>>>>
>>>>>>> There are other odd errors as well.
>>>>>>>
>>>>>>> Perhaps we should ask: is this a "stock" build? Check out Drill and
>>>>> run
>>>>>>> tests? Or, have you already started making changes for your project?
>>>>>>>
>>>>>>> - Paul
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle<mb...@apache.org>
>>>>> wrote:
>>>>>>>> I have drill building and running its tests. Some tests fail:
>>>>> [ERROR]
>>>>>>>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
>>>>>>>>
>>>>>>>> I am wondering if there is perhaps some setup step that I missed in
>>>>> the
>>>>>>>> instructions.
>>>>>>>>
>>>>>>>> I have attached the output from the 'mvn clean install
>>>>> -DskipTests=false'
>>>>>>>> execution. (zipped)
>>>>>>>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
>>>>>>>>
>>>>>>>> I'm hoping someone can skim it and spot the issue(s).
>>>>>>>>
>>>>>>>> Thanks for any help
>>>>>>>>
>>>>>>>> Mike Beckerle
>>>>>>>> Apache Daffodil PMC | daffodil.apache.org
>>>>>>>> OGF DFDL Workgroup Co-Chair |
>>>>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>>>>>>>> Owl Cyber Defense |www.owlcyberdefense.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
Re: drill tests not passing
Posted by Mike Beckerle <mb...@apache.org>.
Thank you for your help.
To review, this command works and succeeds:
mvn clean install -DskipTests=true
mvn test ... with args per your prior email...
fails with the "drill-java-exec: Artifact has not been packaged yet..."
error.
On Fri, Aug 25, 2023 at 3:54 AM James Turton <dz...@apache.org> wrote:
> Okay, that attempt couldn't even build java-exec. What does the
> following do when run from the root of the source tree?"
>
> mvn package -f exec/vector/pom.xml
>
The above builds successfully.
The below builds, but when executing tests, huge number of errors. File
attached.
> mvn package -f exec/java-exec/pom.xml
>
> On 2023/08/24 17:09, Mike Beckerle wrote:
> >
> > Still no luck.
> >
> > Output from your mvn-test-drill command is attached.
> >
> > drill-java-exec: Artifact has not been packaged yet. ....
> >
> >
> > On Mon, Aug 21, 2023 at 10:36 AM James Turton <dzamo@apache.org
> > <ma...@apache.org>> wrote:
> >
> > __
> > Hi Mike
> >
> > I took a look at the build log that you shared with me recently and
> > what happened in that run was OOM (first report on line 9348).
> >
> > | 9355 2198813 ERROR [UserServer-1]
> > [org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
> > communication. 9355 Connection: /127.0.0.1:31046
> > <http://127.0.0.1:31046> <--> /127.0.0.1:39558
> > <http://127.0.0.1:39558> (user server). Closing connection.
> > 9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
> > |
> > My command to run Drill tests set memory limits as follows.
> >
> > |alias mvn-test-drill="mvn test \
> > -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
> > -Duser.region=US\" \
> > -DmemoryMb=2560 -DdirectMemoryMb=2560 \
> > -DforkCount=2"
> > |
> > Hope this helps...
> > James
> >
> > On 2023/07/25 00:00, Mike Beckerle wrote:
> >> Hi drill devs,
> >>
> >> I'm still stuck on this problem. Can anyone suggest a way past this?
> >>
> >> Mike Beckerle
> >> Apache Daffodil PMC |daffodil.apache.org <
> http://daffodil.apache.org>
> >> OGF DFDL Workgroup Co-Chair |
> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl <
> http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl>
> >> Owl Cyber Defense |www.owlcyberdefense.com <
> http://www.owlcyberdefense.com>
> >>
> >>
> >>
> >> On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle<mb...@apache.org>
> <ma...@apache.org> wrote:
> >>
> >>> Looks like I attached the wrong file. Doing too many things at
> once.
> >>>
> >>> The correct file is attached here.
> >>>
> >>>
> >>>
> >>> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle<mb...@apache.org>
> <ma...@apache.org>
> >>> wrote:
> >>>
> >>>> Update: I did a clean and install -DskipTests=true.
> >>>>
> >>>> Then I tried the mvn test using the non-UTC timezone stuff, as
> suggested.
> >>>>
> >>>> But alas, it still fails, this time the failure unique and is
> only in
> >>>> "Java Execution Engine"
> >>>>
> >>>> [ERROR] Failed to execute goal
> >>>> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
> >>>> (unpack-vector-types) on project drill-java-exec: Artifact has
> not been
> >>>> packaged yet. When used on reactor artifact, unpack should be
> executed
> >>>> after packaging: see MDEP-98. -> [Help 1]
> >>>>
> >>>> The command and complete trace output are below.
> >>>>
> >>>> I need assistance on how to proceed.
> >>>>
> >>>> Complete trace from the mvn test is attached.
> >>>>
> >>>>
> >>>> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle<
> mbeckerle@apache.org> <ma...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> To answer questions:
> >>>>>
> >>>>> 1. Paul: This is a 100% stock build. All I have done is clone
> the repo
> >>>>> (master branch). Make a new git branch (in case I make future
> changes). Try
> >>>>> to build (success) and test (failed so far).
> >>>>>
> >>>>> 2. James: The /opt/drill directory I created is owned by my
> userid and
> >>>>> has full read/write access for all the development activities. I
> just put
> >>>>> it there so it would have a shorter path to fix the first
> Hive-related
> >>>>> glitch I encountered with the Linux 255 limit on file pathname
> length.
> >>>>>
> >>>>> I will try the suggested maven command line for non-UTC and see
> if
> >>>>> things improve.
> >>>>>
> >>>>> The challenge for me as a newby is how do I know if I have
> everything
> >>>>> properly configured?
> >>>>>
> >>>>> Can I just turn off building and testing of the Hive-related
> stuff in
> >>>>> some supported/well-known way?
> >>>>>
> >>>>> If so, I would suggest I'd like to turn off not just Hive, but
> *as much
> >>>>> as possible*. I really just need the embedded drill to work.
> >>>>>
> >>>>> I would agree with @Charles Givre<cg...@gmail.com> <mailto:
> cgivre@gmail.com> that a contrib
> >>>>> package addition is the ideal approach and that's what I'll be
> attempting.
> >>>>>
> >>>>> -mikeb
> >>>>>
> >>>>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre<cg...@gmail.com>
> <ma...@gmail.com> wrote:
> >>>>>
> >>>>>> I'll add some heresy here... IMHO, for the purposes of
> developing a
> >>>>>> DFDL extension, you probably don't need all the Drill tests to
> run. For
> >>>>>> your project, my suggestion would be to add a module to the
> contrib package
> >>>>>> and that way your changes are relatively self contained.
> >>>>>> Best,
> >>>>>> -- C
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Jul 13, 2023, at 10:27 AM, James Turton<dz...@apache.org>
> <ma...@apache.org> wrote:
> >>>>>>>
> >>>>>>> Hi Mike
> >>>>>>>
> >>>>>>> Here's the command line I use to run tests on a machine that's
> not in
> >>>>>> the UTC time zone (plus some unrelated memory size arguments).
> >>>>>>> mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
> >>>>>> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
> >>>>>>> I have one other question to add to Paul's comments - does the
> OS
> >>>>>> user that you're running Maven under have write access to all
> of the source
> >>>>>> tree that you put at /opt/drill?
> >>>>>>> On 2023/07/11 22:12, Paul Rogers wrote:
> >>>>>>>> Hi Mike,
> >>>>>>>>
> >>>>>>>> A quick glance at the log suggests a failure in the tests for
> the
> >>>>>> JSON
> >>>>>>>> reader, in the Mongo extended types. Drill's date/time
> support has
> >>>>>>>> historically been fragile. Some tests only work if your
> machine is
> >>>>>> set to
> >>>>>>>> use the UTC time zone (or Java is told to pretend that the
> time is
> >>>>>> UTC.)
> >>>>>>>> The Mongo types test failure seems to be around a date/time
> test so
> >>>>>> maybe
> >>>>>>>> this is the issue?
> >>>>>>>>
> >>>>>>>> There are also failures indicating that the Drillbit (Drill
> server)
> >>>>>> died.
> >>>>>>>> Not sure how this can happen, as tests run Drill embedded (or
> used
> >>>>>> to.)
> >>>>>>>> Looking earlier in the logs, it seems that the Drillbit
> didn't start
> >>>>>> due to
> >>>>>>>> UDF (user-defined function) failures:
> >>>>>>>>
> >>>>>>>> Found duplicated function in drill-custom-lower.jar:
> >>>>>>>> custom_lower(VARCHAR-REQUIRED)
> >>>>>>>> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
> >>>>>>>>
> >>>>>>>> Not sure how this could occur: it should have failed in all
> builds.
> >>>>>>>>
> >>>>>>>> Also:
> >>>>>>>>
> >>>>>>>> File
> >>>>>>>>
> >>>>>>
> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
> >>>>>>>> does not exist on file systemfile:///
> >>>>>>>>
> >>>>>>>> This is complaining that Drill needs the source code (not
> just class
> >>>>>> file)
> >>>>>>>> for its built-in functions. Again, this should not fail in a
> standard
> >>>>>>>> build, because if it did, it would fail in all builds.
> >>>>>>>>
> >>>>>>>> There are other odd errors as well.
> >>>>>>>>
> >>>>>>>> Perhaps we should ask: is this a "stock" build? Check out
> Drill and
> >>>>>> run
> >>>>>>>> tests? Or, have you already started making changes for your
> project?
> >>>>>>>>
> >>>>>>>> - Paul
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle<
> mbeckerle@apache.org> <ma...@apache.org>
> >>>>>> wrote:
> >>>>>>>>> I have drill building and running its tests. Some tests fail:
> >>>>>> [ERROR]
> >>>>>>>>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
> >>>>>>>>>
> >>>>>>>>> I am wondering if there is perhaps some setup step that I
> missed in
> >>>>>> the
> >>>>>>>>> instructions.
> >>>>>>>>>
> >>>>>>>>> I have attached the output from the 'mvn clean install
> >>>>>> -DskipTests=false'
> >>>>>>>>> execution. (zipped)
> >>>>>>>>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
> >>>>>>>>>
> >>>>>>>>> I'm hoping someone can skim it and spot the issue(s).
> >>>>>>>>>
> >>>>>>>>> Thanks for any help
> >>>>>>>>>
> >>>>>>>>> Mike Beckerle
> >>>>>>>>> Apache Daffodil PMC |daffodil.apache.org <
> http://daffodil.apache.org>
> >>>>>>>>> OGF DFDL Workgroup Co-Chair |
> >>>>>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl <
> http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl>
> >>>>>>>>> Owl Cyber Defense |www.owlcyberdefense.com <
> http://www.owlcyberdefense.com>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >
>
Re: drill tests not passing
Posted by James Turton <dz...@apache.org>.
Okay, that attempt couldn't even build java-exec. What does the
following do when run from the root of the source tree?"
mvn package -f exec/vector/pom.xml
mvn package -f exec/java-exec/pom.xml
On 2023/08/24 17:09, Mike Beckerle wrote:
>
> Still no luck.
>
> Output from your mvn-test-drill command is attached.
>
> drill-java-exec: Artifact has not been packaged yet. ....
>
>
> On Mon, Aug 21, 2023 at 10:36 AM James Turton <dzamo@apache.org
> <ma...@apache.org>> wrote:
>
> __
> Hi Mike
>
> I took a look at the build log that you shared with me recently and
> what happened in that run was OOM (first report on line 9348).
>
> | 9355 2198813 ERROR [UserServer-1]
> [org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
> communication. 9355 Connection: /127.0.0.1:31046
> <http://127.0.0.1:31046> <--> /127.0.0.1:39558
> <http://127.0.0.1:39558> (user server). Closing connection.
> 9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
> |
> My command to run Drill tests set memory limits as follows.
>
> |alias mvn-test-drill="mvn test \
> -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
> -Duser.region=US\" \
> -DmemoryMb=2560 -DdirectMemoryMb=2560 \
> -DforkCount=2"
> |
> Hope this helps...
> James
>
> On 2023/07/25 00:00, Mike Beckerle wrote:
>> Hi drill devs,
>>
>> I'm still stuck on this problem. Can anyone suggest a way past this?
>>
>> Mike Beckerle
>> Apache Daffodil PMC |daffodil.apache.org <http://daffodil.apache.org>
>> OGF DFDL Workgroup Co-Chair |www.ogf.org/ogf/doku.php/standards/dfdl/dfdl <http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl>
>> Owl Cyber Defense |www.owlcyberdefense.com <http://www.owlcyberdefense.com>
>>
>>
>>
>> On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle<mb...@apache.org> <ma...@apache.org> wrote:
>>
>>> Looks like I attached the wrong file. Doing too many things at once.
>>>
>>> The correct file is attached here.
>>>
>>>
>>>
>>> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle<mb...@apache.org> <ma...@apache.org>
>>> wrote:
>>>
>>>> Update: I did a clean and install -DskipTests=true.
>>>>
>>>> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>>>>
>>>> But alas, it still fails, this time the failure unique and is only in
>>>> "Java Execution Engine"
>>>>
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
>>>> (unpack-vector-types) on project drill-java-exec: Artifact has not been
>>>> packaged yet. When used on reactor artifact, unpack should be executed
>>>> after packaging: see MDEP-98. -> [Help 1]
>>>>
>>>> The command and complete trace output are below.
>>>>
>>>> I need assistance on how to proceed.
>>>>
>>>> Complete trace from the mvn test is attached.
>>>>
>>>>
>>>> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle<mb...@apache.org> <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> To answer questions:
>>>>>
>>>>> 1. Paul: This is a 100% stock build. All I have done is clone the repo
>>>>> (master branch). Make a new git branch (in case I make future changes). Try
>>>>> to build (success) and test (failed so far).
>>>>>
>>>>> 2. James: The /opt/drill directory I created is owned by my userid and
>>>>> has full read/write access for all the development activities. I just put
>>>>> it there so it would have a shorter path to fix the first Hive-related
>>>>> glitch I encountered with the Linux 255 limit on file pathname length.
>>>>>
>>>>> I will try the suggested maven command line for non-UTC and see if
>>>>> things improve.
>>>>>
>>>>> The challenge for me as a newby is how do I know if I have everything
>>>>> properly configured?
>>>>>
>>>>> Can I just turn off building and testing of the Hive-related stuff in
>>>>> some supported/well-known way?
>>>>>
>>>>> If so, I would suggest I'd like to turn off not just Hive, but *as much
>>>>> as possible*. I really just need the embedded drill to work.
>>>>>
>>>>> I would agree with @Charles Givre<cg...@gmail.com> <ma...@gmail.com> that a contrib
>>>>> package addition is the ideal approach and that's what I'll be attempting.
>>>>>
>>>>> -mikeb
>>>>>
>>>>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre<cg...@gmail.com> <ma...@gmail.com> wrote:
>>>>>
>>>>>> I'll add some heresy here... IMHO, for the purposes of developing a
>>>>>> DFDL extension, you probably don't need all the Drill tests to run. For
>>>>>> your project, my suggestion would be to add a module to the contrib package
>>>>>> and that way your changes are relatively self contained.
>>>>>> Best,
>>>>>> -- C
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Jul 13, 2023, at 10:27 AM, James Turton<dz...@apache.org> <ma...@apache.org> wrote:
>>>>>>>
>>>>>>> Hi Mike
>>>>>>>
>>>>>>> Here's the command line I use to run tests on a machine that's not in
>>>>>> the UTC time zone (plus some unrelated memory size arguments).
>>>>>>> mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>>>>>> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>>>>>>> I have one other question to add to Paul's comments - does the OS
>>>>>> user that you're running Maven under have write access to all of the source
>>>>>> tree that you put at /opt/drill?
>>>>>>> On 2023/07/11 22:12, Paul Rogers wrote:
>>>>>>>> Hi Mike,
>>>>>>>>
>>>>>>>> A quick glance at the log suggests a failure in the tests for the
>>>>>> JSON
>>>>>>>> reader, in the Mongo extended types. Drill's date/time support has
>>>>>>>> historically been fragile. Some tests only work if your machine is
>>>>>> set to
>>>>>>>> use the UTC time zone (or Java is told to pretend that the time is
>>>>>> UTC.)
>>>>>>>> The Mongo types test failure seems to be around a date/time test so
>>>>>> maybe
>>>>>>>> this is the issue?
>>>>>>>>
>>>>>>>> There are also failures indicating that the Drillbit (Drill server)
>>>>>> died.
>>>>>>>> Not sure how this can happen, as tests run Drill embedded (or used
>>>>>> to.)
>>>>>>>> Looking earlier in the logs, it seems that the Drillbit didn't start
>>>>>> due to
>>>>>>>> UDF (user-defined function) failures:
>>>>>>>>
>>>>>>>> Found duplicated function in drill-custom-lower.jar:
>>>>>>>> custom_lower(VARCHAR-REQUIRED)
>>>>>>>> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>>>>>>>>
>>>>>>>> Not sure how this could occur: it should have failed in all builds.
>>>>>>>>
>>>>>>>> Also:
>>>>>>>>
>>>>>>>> File
>>>>>>>>
>>>>>> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
>>>>>>>> does not exist on file systemfile:///
>>>>>>>>
>>>>>>>> This is complaining that Drill needs the source code (not just class
>>>>>> file)
>>>>>>>> for its built-in functions. Again, this should not fail in a standard
>>>>>>>> build, because if it did, it would fail in all builds.
>>>>>>>>
>>>>>>>> There are other odd errors as well.
>>>>>>>>
>>>>>>>> Perhaps we should ask: is this a "stock" build? Check out Drill and
>>>>>> run
>>>>>>>> tests? Or, have you already started making changes for your project?
>>>>>>>>
>>>>>>>> - Paul
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle<mb...@apache.org> <ma...@apache.org>
>>>>>> wrote:
>>>>>>>>> I have drill building and running its tests. Some tests fail:
>>>>>> [ERROR]
>>>>>>>>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
>>>>>>>>>
>>>>>>>>> I am wondering if there is perhaps some setup step that I missed in
>>>>>> the
>>>>>>>>> instructions.
>>>>>>>>>
>>>>>>>>> I have attached the output from the 'mvn clean install
>>>>>> -DskipTests=false'
>>>>>>>>> execution. (zipped)
>>>>>>>>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
>>>>>>>>>
>>>>>>>>> I'm hoping someone can skim it and spot the issue(s).
>>>>>>>>>
>>>>>>>>> Thanks for any help
>>>>>>>>>
>>>>>>>>> Mike Beckerle
>>>>>>>>> Apache Daffodil PMC |daffodil.apache.org <http://daffodil.apache.org>
>>>>>>>>> OGF DFDL Workgroup Co-Chair |
>>>>>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl <http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl>
>>>>>>>>> Owl Cyber Defense |www.owlcyberdefense.com <http://www.owlcyberdefense.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>
Re: drill tests not passing
Posted by Mike Beckerle <mb...@apache.org>.
Still no luck.
Output from your mvn-test-drill command is attached.
drill-java-exec: Artifact has not been packaged yet. ....
On Mon, Aug 21, 2023 at 10:36 AM James Turton <dz...@apache.org> wrote:
> Hi Mike
>
> I took a look at the build log that you shared with me recently and what
> happened in that run was OOM (first report on line 9348).
>
> 9355 2198813 ERROR [UserServer-1]
> [org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
> communication. 9355 Connection: /127.0.0.1:31046 <--> /127.0.0.1:39558
> (user server). Closing connection.
> 9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> My command to run Drill tests set memory limits as follows.
>
> alias mvn-test-drill="mvn test \
> -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
> -Duser.region=US\" \
> -DmemoryMb=2560 -DdirectMemoryMb=2560 \
> -DforkCount=2"
>
> Hope this helps...
> James
>
> On 2023/07/25 00:00, Mike Beckerle wrote:
>
> Hi drill devs,
>
> I'm still stuck on this problem. Can anyone suggest a way past this?
>
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> Owl Cyber Defense | www.owlcyberdefense.com
>
>
>
> On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle <mb...@apache.org> <mb...@apache.org> wrote:
>
>
> Looks like I attached the wrong file. Doing too many things at once.
>
> The correct file is attached here.
>
>
>
> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle <mb...@apache.org> <mb...@apache.org>
> wrote:
>
>
> Update: I did a clean and install -DskipTests=true.
>
> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>
> But alas, it still fails, this time the failure unique and is only in
> "Java Execution Engine"
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
> (unpack-vector-types) on project drill-java-exec: Artifact has not been
> packaged yet. When used on reactor artifact, unpack should be executed
> after packaging: see MDEP-98. -> [Help 1]
>
> The command and complete trace output are below.
>
> I need assistance on how to proceed.
>
> Complete trace from the mvn test is attached.
>
>
> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle <mb...@apache.org> <mb...@apache.org>
> wrote:
>
>
> To answer questions:
>
> 1. Paul: This is a 100% stock build. All I have done is clone the repo
> (master branch). Make a new git branch (in case I make future changes). Try
> to build (success) and test (failed so far).
>
> 2. James: The /opt/drill directory I created is owned by my userid and
> has full read/write access for all the development activities. I just put
> it there so it would have a shorter path to fix the first Hive-related
> glitch I encountered with the Linux 255 limit on file pathname length.
>
> I will try the suggested maven command line for non-UTC and see if
> things improve.
>
> The challenge for me as a newby is how do I know if I have everything
> properly configured?
>
> Can I just turn off building and testing of the Hive-related stuff in
> some supported/well-known way?
>
> If so, I would suggest I'd like to turn off not just Hive, but *as much
> as possible*. I really just need the embedded drill to work.
>
> I would agree with @Charles Givre <cg...@gmail.com> <cg...@gmail.com> that a contrib
> package addition is the ideal approach and that's what I'll be attempting.
>
> -mikeb
>
> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre <cg...@gmail.com> <cg...@gmail.com> wrote:
>
>
> I'll add some heresy here... IMHO, for the purposes of developing a
> DFDL extension, you probably don't need all the Drill tests to run. For
> your project, my suggestion would be to add a module to the contrib package
> and that way your changes are relatively self contained.
> Best,
> -- C
>
>
>
>
> On Jul 13, 2023, at 10:27 AM, James Turton <dz...@apache.org> <dz...@apache.org> wrote:
>
> Hi Mike
>
> Here's the command line I use to run tests on a machine that's not in
>
> the UTC time zone (plus some unrelated memory size arguments).
>
> mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>
> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>
> I have one other question to add to Paul's comments - does the OS
>
> user that you're running Maven under have write access to all of the source
> tree that you put at /opt/drill?
>
> On 2023/07/11 22:12, Paul Rogers wrote:
>
> Hi Mike,
>
> A quick glance at the log suggests a failure in the tests for the
>
> JSON
>
> reader, in the Mongo extended types. Drill's date/time support has
> historically been fragile. Some tests only work if your machine is
>
> set to
>
> use the UTC time zone (or Java is told to pretend that the time is
>
> UTC.)
>
> The Mongo types test failure seems to be around a date/time test so
>
> maybe
>
> this is the issue?
>
> There are also failures indicating that the Drillbit (Drill server)
>
> died.
>
> Not sure how this can happen, as tests run Drill embedded (or used
>
> to.)
>
> Looking earlier in the logs, it seems that the Drillbit didn't start
>
> due to
>
> UDF (user-defined function) failures:
>
> Found duplicated function in drill-custom-lower.jar:
> custom_lower(VARCHAR-REQUIRED)
> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>
> Not sure how this could occur: it should have failed in all builds.
>
> Also:
>
> File
>
>
> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
>
> does not exist on file system file:///
>
> This is complaining that Drill needs the source code (not just class
>
> file)
>
> for its built-in functions. Again, this should not fail in a standard
> build, because if it did, it would fail in all builds.
>
> There are other odd errors as well.
>
> Perhaps we should ask: is this a "stock" build? Check out Drill and
>
> run
>
> tests? Or, have you already started making changes for your project?
>
> - Paul
>
>
> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle <mb...@apache.org> <mb...@apache.org>
>
> wrote:
>
> I have drill building and running its tests. Some tests fail:
>
> [ERROR]
>
> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
>
> I am wondering if there is perhaps some setup step that I missed in
>
> the
>
> instructions.
>
> I have attached the output from the 'mvn clean install
>
> -DskipTests=false'
>
> execution. (zipped)
> I am running on Ubuntu 20.04, definitely have Java 8 setup.
>
> I'm hoping someone can skim it and spot the issue(s).
>
> Thanks for any help
>
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair |
>
> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>
> Owl Cyber Defense | www.owlcyberdefense.com
>
>
>
>