You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by bobtins <bo...@gmail.com> on 2021/03/12 02:38:09 UTC
[JAVA] issues encountered during build
My mail client took out all the linefeeds, so let me reformat; sorry about that!
In the process of slogging through the build, I've bumped into various issues. I'm happy to document them in java/README.md or make any other changes that might be helpful to others.
I'm pretty experienced with Java and Maven, so I think these are not beginner's mistakes, but let me know if I'm missing something.
A lot of these may be Windows-specific. I normally prefer Linux but just got a new laptop and haven't set it up, but this experience is giving me a lot of incentive to run screaming back to Linux ;-)
Environment details:
* Windows 10
* Java 8
here's the output of java -version:
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
* Cygwin environment
* Maven 3.6.2
Issues encountered thus far:
* Build does require Java 8, not "8 or later" as stated in java/README.md
There's a reference to sun.misc.Unsafe in memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java which of course went away in JDK 9.
* The build won't work on Windows because the java/format POM downloads a binary flatc executable, but there's no artifact for Windows, just Linux and OSX. I wound up downloading Visual Studio and building the flatbuffers project.
* I bumped into what looks like a spurious checkstyle error: it reports memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no linefeed at the end when it definitely does. I set up Git not to do Windows conversions, and I checked with various editors and binary dump utilities. One source says that this because I'm running on Windows, checkstyle actually expects a CR-LF and throws an error if it doesn't find it! I've worked around this by disabling the check.
* The one thing that I'm stuck on now is failures on the jdbc module:
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209 expected:<45935000> but was:<74735000>
[ERROR] JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213 expected:<1518439535000> but was:<1518468335000>
[ERROR] JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205 expected:<-365> but was:<-364>
[ERROR] JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209 expected:<17574> but was:<17573>
[ERROR] JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206 expected:<17574> but was:<17573>
[ERROR] JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277 expected:<17574> but was:<17573>
[ERROR] JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277 expected:<17574> but was:<17573>
[INFO]
[ERROR] Tests run: 93, Failures: 7, Errors: 0, Skipped: 0
I attached the full build output.
Looking more closely at these errors, they seem to be due to the timezone difference; for example, the difference between 74735000 (actual value) and 45935000 (expected) is 2880000, or 8 hours in milliseconds, which is the PST timezone offset.
I see in the pom.xml that user.timezone is set to UTC. I have seen these types of errors in tests before; I know there are ways to insulate the test from the user's current timezone but maybe someone knows what's going on.
Thanks for any input!
On 2021/03/11 23:49:37, Bob Tinsman <bo...@pacbell.net> wrote:
> I've been mostly lurking for awhile, but I would like to start picking off some bugs in the Java implementation.In the process of slogging through the build, I've bumped into various issues. I'm happy to document them in java/README.md or make any other changes that might be helpful to others. I'm pretty experienced with Java and Maven, so I think these are not super-obvious, but let me know if I'm missing something.A lot of these may be Windows-specific. I normally prefer Linux but just got a new laptop and haven't set it up, but this experience is giving me a lot of incentive to run screaming back to Linux ;-)
> Environment details:- Windows 10- Java 8:openjdk version "1.8.0_282"OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)- Cygwin environment- Maven 3.6.2
> Issues encountered thus far:- Build does require Java 8, not "8 or later" as stated in java/README.md There's a reference to sun.misc.Unsafe in memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java which of course went away in JDK 9. I can update the build instructions.- The build won't work on Windows because the java/format POM downloads a binary flatc executables; when I looked, there was no version for Windows, just Linux and OSX. I wound up downloading Visual Studio and building the flatbuffers project.- I bumped into what looks like a spurious checkstyle error: it reports memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no linefeed at the end when it definitely does. I set up Git not to do Windows conversions, and I checked with various editors and binary dump utilities. One source says that this because I'm running on Windows, checkstyle actually expects a CR-LF and throws an error if it doesn't find it! I've worked
around this by disabling the check.- The one thing that I'm stuck on now is failures on jdbc:
> [INFO] Results:[INFO][ERROR] Failures:[ERROR] JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209 expected:<45935000> but was:<74735000>[ERROR] JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213 expected:<1518439535000> but was:<1518468335000>[ERROR] JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205 expected:<-365> but was:<-364>[ERROR] JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209 expected:<17574> but was:<17573>[ERROR] JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206 expected:<17574> but was:<17573>[ERROR] JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277 expected:<17574> but was:<17573>[ERROR] JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277 expected:<17574> but was:<17573>[INFO][ERROR] Tests run: 93, Failures: 7, Errors: 0, Skipped: 0
> I attached the full build output.Looking more closely at these errors, they seem to be due to the timezone difference; for example, the difference between 74735000 (actual value) and 45935000 (expected) is 2880000, or 8 hours in milliseconds, which is the PST timezone offset. I see in the pom.xml that user.timezone is set to UTC. I have seen these types of errors in tests before; I know there are ways to insulate the test from the user's current timezone but maybe someone knows what's going on.
> Thanks for any input!
>
Re: [JAVA] issues encountered during build
Posted by Fan Liya <li...@gmail.com>.
Hi Bob,
Thanks a lot for your follow-up.
Maybe you need to send a separate email to the dev to apply for the
contributor permission.
Best,
Liya Fan
On Thu, Mar 18, 2021 at 3:37 AM bobtins <bo...@gmail.com> wrote:
>
>
> On 2021/03/12 06:36:24, Fan Liya <li...@gmail.com> wrote:
> > Hi Bob,
> >
> > Thanks for reporting the issues.
> > I remember encountering the same problems with the JDBC tests (over one
> > year ago).
> >
> > Maybe it is not just related to the time zone, it is also related to the
> > machine locale.
> > I think we can open an issue to track it.
> >
> OK, opened an issue: https://issues.apache.org/jira/browse/ARROW-11957
> I'll create the pull request today, probably.
> I noticed that I can't add watchers or even assign to myself; saw on the
> doc that someone needs to make me a Contributor.
> Here's the bug for various Windows build nits:
> https://issues.apache.org/jira/browse/ARROW-12006
>
>
> @Liya, I don't think it has anything to do with locale; it's the offset
> associated with the time zone which is showing up.
>
> @Micah, I guess I'm used to doing local builds and having the JVM pick up
> my timezone; now that we're on daylight savings, everything will be off by
> 7 hours ;-)
>
> >
> >
> > On Fri, Mar 12, 2021 at 12:09 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > Hi Bob,
> > > Thanks for some feedback, I don't think a lot of people are developing
> on
> > > windows. Some answers in line:
> > >
> > > * Build does require Java 8, not "8 or later" as stated in
> java/README.md
> > > > There's a reference to sun.misc.Unsafe
> > > > in
> > >
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > > which of course went away in JDK 9.
> > >
> > > Is this the case even using
> "-Dio.netty.tryReflectionSetAccessible=true" as
> > > mentioned in the README?
> > >
> > >
> > > * The build won't work on Windows because the java/format POM
> downloads a
> > > > binary flatc executable, but there's no artifact for Windows, just
> Linux
> > > > and OSX. I wound up downloading Visual Studio and building the
> > > flatbuffers
> > > > project.
> > >
> > > This unfortunately sounds familiar, I think this indicate the
> popularity on
> > > windows. I also think the hosting of the mac and linux are not exactly
> > > official (they are hosted by a former contributor). Updating the
> Readme
> > > might be a good first step with instructions on how to do this.
> > >
> > > I see in the pom.xml that user.timezone is set to UTC. I have seen
> these
> > > > types of errors in tests before; I know there are ways to insulate
> the
> > > test
> > > > from the user's current timezone but maybe someone knows what's
> going on
> > >
> > > This is somewhat surprising. I would thought we had the user.timezone
> set
> > > for exactly this reason. There might have been a regression, this
> might
> > > make a good second contribution if you wanted to look into fixing it.
> > >
> > > * I bumped into what looks like a spurious checkstyle error: it reports
> > > > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
> having no
> > > > linefeed at the end when it definitely does. I set up Git not to do
> > > Windows
> > > > conversions, and I checked with various editors and binary dump
> > > utilities.
> > > > One source says that this because I'm running on Windows, checkstyle
> > > > actually expects a CR-LF and throws an error if it doesn't find it!
> I've
> > > > worked around this by disabling the check.
> > >
> > > It looks like we can force the checker to assume linux line feeds:
> > >
> > >
> https://stackoverflow.com/questions/997021/how-to-get-rid-of-checkstyle-message-file-does-not-end-with-a-newline
> > > (second answer). This would also make a good contribution for someone
> new.
> > >
> > > Cheers,
> > > Micah
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Mar 11, 2021 at 7:14 PM bobtins <bo...@gmail.com> wrote:
> > >
> > > > My mail client took out all the linefeeds, so let me reformat; sorry
> > > about
> > > > that!
> > > >
> > > >
> > > > In the process of slogging through the build, I've bumped into
> various
> > > > issues. I'm happy to document them in java/README.md or make any
> other
> > > > changes that might be helpful to others.
> > > >
> > > > I'm pretty experienced with Java and Maven, so I think these are not
> > > > beginner's mistakes, but let me know if I'm missing something.
> > > >
> > > > A lot of these may be Windows-specific. I normally prefer Linux but
> just
> > > > got a new laptop and haven't set it up, but this experience is
> giving me
> > > a
> > > > lot of incentive to run screaming back to Linux ;-)
> > > >
> > > > Environment details:
> > > > * Windows 10
> > > > * Java 8
> > > > here's the output of java -version:
> > > > openjdk version "1.8.0_282"
> > > > OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
> > > > OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
> > > >
> > > > * Cygwin environment
> > > > * Maven 3.6.2
> > > >
> > > >
> > > > Issues encountered thus far:
> > > >
> > > > * Build does require Java 8, not "8 or later" as stated in
> java/README.md
> > > > There's a reference to sun.misc.Unsafe
> > > > in
> > >
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > > which of course went away in JDK 9.
> > > > * The build won't work on Windows because the java/format POM
> downloads a
> > > > binary flatc executable, but there's no artifact for Windows, just
> Linux
> > > > and OSX. I wound up downloading Visual Studio and building the
> > > flatbuffers
> > > > project.
> > > >
> > > > * I bumped into what looks like a spurious checkstyle error: it
> reports
> > > > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
> having no
> > > > linefeed at the end when it definitely does. I set up Git not to do
> > > Windows
> > > > conversions, and I checked with various editors and binary dump
> > > utilities.
> > > > One source says that this because I'm running on Windows, checkstyle
> > > > actually expects a CR-LF and throws an error if it doesn't find it!
> I've
> > > > worked around this by disabling the check.
> > > >
> > > > * The one thing that I'm stuck on now is failures on the jdbc module:
> > > > [INFO]
> > > > [INFO] Results:
> > > > [INFO]
> > > > [ERROR] Failures:
> > > > [ERROR]
> > > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> > > > expected:<45935000> but was:<74735000>
> > > > [ERROR]
> > > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> > > > expected:<1518439535000> but was:<1518468335000>
> > > > [ERROR]
> > > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> > > > expected:<-365> but was:<-364>
> > > > [ERROR]
> > > >
> > >
> JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> > > > expected:<17574> but was:<17573>
> > > > [ERROR] JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> > > > expected:<17574> but was:<17573>
> > > > [ERROR]
> > > >
> > >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> > > > expected:<17574> but was:<17573>
> > > > [ERROR]
> > > >
> > >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> > > > expected:<17574> but was:<17573>
> > > > [INFO]
> > > > [ERROR] Tests run: 93, Failures: 7, Errors: 0, Skipped: 0
> > > >
> > > > I attached the full build output.
> > > > Looking more closely at these errors, they seem to be due to the
> timezone
> > > > difference; for example, the difference between 74735000 (actual
> value)
> > > and
> > > > 45935000 (expected) is 2880000, or 8 hours in milliseconds, which is
> the
> > > > PST timezone offset.
> > > >
> > > > I see in the pom.xml that user.timezone is set to UTC. I have seen
> these
> > > > types of errors in tests before; I know there are ways to insulate
> the
> > > test
> > > > from the user's current timezone but maybe someone knows what's
> going on.
> > > >
> > > > Thanks for any input!
> > > >
> > > >
> > > > On 2021/03/11 23:49:37, Bob Tinsman <bo...@pacbell.net> wrote:
> > > > > I've been mostly lurking for awhile, but I would like to start
> picking
> > > > off some bugs in the Java implementation.In the process of slogging
> > > through
> > > > the build, I've bumped into various issues. I'm happy to document
> them in
> > > > java/README.md or make any other changes that might be helpful to
> others.
> > > > I'm pretty experienced with Java and Maven, so I think these are not
> > > > super-obvious, but let me know if I'm missing something.A lot of
> these
> > > may
> > > > be Windows-specific. I normally prefer Linux but just got a new
> laptop
> > > and
> > > > haven't set it up, but this experience is giving me a lot of
> incentive to
> > > > run screaming back to Linux ;-)
> > > > > Environment details:- Windows 10- Java 8:openjdk version
> > > > "1.8.0_282"OpenJDK Runtime Environment (AdoptOpenJDK)(build
> > > > 1.8.0_282-b08)OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build
> 25.282-b08,
> > > > mixed mode)- Cygwin environment- Maven 3.6.2
> > > > > Issues encountered thus far:- Build does require Java 8, not "8 or
> > > > later" as stated in java/README.md There's a reference to
> > > > sun.misc.Unsafe
> > > > in
> > >
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > > which of course went away in JDK 9. I can update the build
> > > > instructions.- The build won't work on Windows because the
> java/format
> > > POM
> > > > downloads a binary flatc executables; when I looked, there was no
> version
> > > > for Windows, just Linux and OSX. I wound up downloading Visual
> Studio and
> > > > building the flatbuffers project.- I bumped into what looks like a
> > > spurious
> > > > checkstyle error: it
> > > > reports
> memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
> > > > having no linefeed at the end when it definitely does. I set up Git
> not
> > > to
> > > > do Windows conversions, and I checked with various editors and binary
> > > dump
> > > > utilities. One source says that this because I'm running on Windows,
> > > > checkstyle actually expects a CR-LF and throws an error if it doesn't
> > > find
> > > > it! I've worked
> > > > around this by disabling the check.- The one thing that I'm stuck
> on
> > > now
> > > > is failures on jdbc:
> > > > > [INFO] Results:[INFO][ERROR] Failures:[ERROR]
> > > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> > > > expected:<45935000> but was:<74735000>[ERROR]
> > > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> > > > expected:<1518439535000> but was:<1518468335000>[ERROR]
> > > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> > > > expected:<-365> but was:<-364>[ERROR]
> > > >
> > >
> JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> > > > expected:<17574> but was:<17573>[ERROR]
> > > > JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> > > > expected:<17574> but was:<17573>[ERROR]
> > > >
> > >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> > > > expected:<17574> but was:<17573>[ERROR]
> > > >
> > >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> > > > expected:<17574> but was:<17573>[INFO][ERROR] Tests run: 93,
> Failures: 7,
> > > > Errors: 0, Skipped: 0
> > > > > I attached the full build output.Looking more closely at these
> errors,
> > > > they seem to be due to the timezone difference; for example, the
> > > difference
> > > > between 74735000 (actual value) and 45935000 (expected) is 2880000,
> or 8
> > > > hours in milliseconds, which is the PST timezone offset. I see in the
> > > > pom.xml that user.timezone is set to UTC. I have seen these types of
> > > errors
> > > > in tests before; I know there are ways to insulate the test from the
> > > user's
> > > > current timezone but maybe someone knows what's going on.
> > > > > Thanks for any input!
> > > > >
> > > >
> > >
> >
>
Re: [JAVA] issues encountered during build
Posted by bobtins <bo...@gmail.com>.
On 2021/03/12 06:36:24, Fan Liya <li...@gmail.com> wrote:
> Hi Bob,
>
> Thanks for reporting the issues.
> I remember encountering the same problems with the JDBC tests (over one
> year ago).
>
> Maybe it is not just related to the time zone, it is also related to the
> machine locale.
> I think we can open an issue to track it.
>
OK, opened an issue: https://issues.apache.org/jira/browse/ARROW-11957
I'll create the pull request today, probably.
I noticed that I can't add watchers or even assign to myself; saw on the doc that someone needs to make me a Contributor.
Here's the bug for various Windows build nits: https://issues.apache.org/jira/browse/ARROW-12006
@Liya, I don't think it has anything to do with locale; it's the offset associated with the time zone which is showing up.
@Micah, I guess I'm used to doing local builds and having the JVM pick up my timezone; now that we're on daylight savings, everything will be off by 7 hours ;-)
>
>
> On Fri, Mar 12, 2021 at 12:09 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Hi Bob,
> > Thanks for some feedback, I don't think a lot of people are developing on
> > windows. Some answers in line:
> >
> > * Build does require Java 8, not "8 or later" as stated in java/README.md
> > > There's a reference to sun.misc.Unsafe
> > > in
> > memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > which of course went away in JDK 9.
> >
> > Is this the case even using "-Dio.netty.tryReflectionSetAccessible=true" as
> > mentioned in the README?
> >
> >
> > * The build won't work on Windows because the java/format POM downloads a
> > > binary flatc executable, but there's no artifact for Windows, just Linux
> > > and OSX. I wound up downloading Visual Studio and building the
> > flatbuffers
> > > project.
> >
> > This unfortunately sounds familiar, I think this indicate the popularity on
> > windows. I also think the hosting of the mac and linux are not exactly
> > official (they are hosted by a former contributor). Updating the Readme
> > might be a good first step with instructions on how to do this.
> >
> > I see in the pom.xml that user.timezone is set to UTC. I have seen these
> > > types of errors in tests before; I know there are ways to insulate the
> > test
> > > from the user's current timezone but maybe someone knows what's going on
> >
> > This is somewhat surprising. I would thought we had the user.timezone set
> > for exactly this reason. There might have been a regression, this might
> > make a good second contribution if you wanted to look into fixing it.
> >
> > * I bumped into what looks like a spurious checkstyle error: it reports
> > > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> > > linefeed at the end when it definitely does. I set up Git not to do
> > Windows
> > > conversions, and I checked with various editors and binary dump
> > utilities.
> > > One source says that this because I'm running on Windows, checkstyle
> > > actually expects a CR-LF and throws an error if it doesn't find it! I've
> > > worked around this by disabling the check.
> >
> > It looks like we can force the checker to assume linux line feeds:
> >
> > https://stackoverflow.com/questions/997021/how-to-get-rid-of-checkstyle-message-file-does-not-end-with-a-newline
> > (second answer). This would also make a good contribution for someone new.
> >
> > Cheers,
> > Micah
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 11, 2021 at 7:14 PM bobtins <bo...@gmail.com> wrote:
> >
> > > My mail client took out all the linefeeds, so let me reformat; sorry
> > about
> > > that!
> > >
> > >
> > > In the process of slogging through the build, I've bumped into various
> > > issues. I'm happy to document them in java/README.md or make any other
> > > changes that might be helpful to others.
> > >
> > > I'm pretty experienced with Java and Maven, so I think these are not
> > > beginner's mistakes, but let me know if I'm missing something.
> > >
> > > A lot of these may be Windows-specific. I normally prefer Linux but just
> > > got a new laptop and haven't set it up, but this experience is giving me
> > a
> > > lot of incentive to run screaming back to Linux ;-)
> > >
> > > Environment details:
> > > * Windows 10
> > > * Java 8
> > > here's the output of java -version:
> > > openjdk version "1.8.0_282"
> > > OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
> > > OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
> > >
> > > * Cygwin environment
> > > * Maven 3.6.2
> > >
> > >
> > > Issues encountered thus far:
> > >
> > > * Build does require Java 8, not "8 or later" as stated in java/README.md
> > > There's a reference to sun.misc.Unsafe
> > > in
> > memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > which of course went away in JDK 9.
> > > * The build won't work on Windows because the java/format POM downloads a
> > > binary flatc executable, but there's no artifact for Windows, just Linux
> > > and OSX. I wound up downloading Visual Studio and building the
> > flatbuffers
> > > project.
> > >
> > > * I bumped into what looks like a spurious checkstyle error: it reports
> > > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> > > linefeed at the end when it definitely does. I set up Git not to do
> > Windows
> > > conversions, and I checked with various editors and binary dump
> > utilities.
> > > One source says that this because I'm running on Windows, checkstyle
> > > actually expects a CR-LF and throws an error if it doesn't find it! I've
> > > worked around this by disabling the check.
> > >
> > > * The one thing that I'm stuck on now is failures on the jdbc module:
> > > [INFO]
> > > [INFO] Results:
> > > [INFO]
> > > [ERROR] Failures:
> > > [ERROR]
> > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> > > expected:<45935000> but was:<74735000>
> > > [ERROR]
> > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> > > expected:<1518439535000> but was:<1518468335000>
> > > [ERROR]
> > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> > > expected:<-365> but was:<-364>
> > > [ERROR]
> > >
> > JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> > > expected:<17574> but was:<17573>
> > > [ERROR] JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> > > expected:<17574> but was:<17573>
> > > [ERROR]
> > >
> > JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> > > expected:<17574> but was:<17573>
> > > [ERROR]
> > >
> > JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> > > expected:<17574> but was:<17573>
> > > [INFO]
> > > [ERROR] Tests run: 93, Failures: 7, Errors: 0, Skipped: 0
> > >
> > > I attached the full build output.
> > > Looking more closely at these errors, they seem to be due to the timezone
> > > difference; for example, the difference between 74735000 (actual value)
> > and
> > > 45935000 (expected) is 2880000, or 8 hours in milliseconds, which is the
> > > PST timezone offset.
> > >
> > > I see in the pom.xml that user.timezone is set to UTC. I have seen these
> > > types of errors in tests before; I know there are ways to insulate the
> > test
> > > from the user's current timezone but maybe someone knows what's going on.
> > >
> > > Thanks for any input!
> > >
> > >
> > > On 2021/03/11 23:49:37, Bob Tinsman <bo...@pacbell.net> wrote:
> > > > I've been mostly lurking for awhile, but I would like to start picking
> > > off some bugs in the Java implementation.In the process of slogging
> > through
> > > the build, I've bumped into various issues. I'm happy to document them in
> > > java/README.md or make any other changes that might be helpful to others.
> > > I'm pretty experienced with Java and Maven, so I think these are not
> > > super-obvious, but let me know if I'm missing something.A lot of these
> > may
> > > be Windows-specific. I normally prefer Linux but just got a new laptop
> > and
> > > haven't set it up, but this experience is giving me a lot of incentive to
> > > run screaming back to Linux ;-)
> > > > Environment details:- Windows 10- Java 8:openjdk version
> > > "1.8.0_282"OpenJDK Runtime Environment (AdoptOpenJDK)(build
> > > 1.8.0_282-b08)OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08,
> > > mixed mode)- Cygwin environment- Maven 3.6.2
> > > > Issues encountered thus far:- Build does require Java 8, not "8 or
> > > later" as stated in java/README.md There's a reference to
> > > sun.misc.Unsafe
> > > in
> > memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > which of course went away in JDK 9. I can update the build
> > > instructions.- The build won't work on Windows because the java/format
> > POM
> > > downloads a binary flatc executables; when I looked, there was no version
> > > for Windows, just Linux and OSX. I wound up downloading Visual Studio and
> > > building the flatbuffers project.- I bumped into what looks like a
> > spurious
> > > checkstyle error: it
> > > reports memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
> > > having no linefeed at the end when it definitely does. I set up Git not
> > to
> > > do Windows conversions, and I checked with various editors and binary
> > dump
> > > utilities. One source says that this because I'm running on Windows,
> > > checkstyle actually expects a CR-LF and throws an error if it doesn't
> > find
> > > it! I've worked
> > > around this by disabling the check.- The one thing that I'm stuck on
> > now
> > > is failures on jdbc:
> > > > [INFO] Results:[INFO][ERROR] Failures:[ERROR]
> > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> > > expected:<45935000> but was:<74735000>[ERROR]
> > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> > > expected:<1518439535000> but was:<1518468335000>[ERROR]
> > > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> > > expected:<-365> but was:<-364>[ERROR]
> > >
> > JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> > > expected:<17574> but was:<17573>[ERROR]
> > > JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> > > expected:<17574> but was:<17573>[ERROR]
> > >
> > JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> > > expected:<17574> but was:<17573>[ERROR]
> > >
> > JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> > > expected:<17574> but was:<17573>[INFO][ERROR] Tests run: 93, Failures: 7,
> > > Errors: 0, Skipped: 0
> > > > I attached the full build output.Looking more closely at these errors,
> > > they seem to be due to the timezone difference; for example, the
> > difference
> > > between 74735000 (actual value) and 45935000 (expected) is 2880000, or 8
> > > hours in milliseconds, which is the PST timezone offset. I see in the
> > > pom.xml that user.timezone is set to UTC. I have seen these types of
> > errors
> > > in tests before; I know there are ways to insulate the test from the
> > user's
> > > current timezone but maybe someone knows what's going on.
> > > > Thanks for any input!
> > > >
> > >
> >
>
Re: [JAVA] issues encountered during build
Posted by Fan Liya <li...@gmail.com>.
Hi Bob,
Thanks for reporting the issues.
I remember encountering the same problems with the JDBC tests (over one
year ago).
Maybe it is not just related to the time zone, it is also related to the
machine locale.
I think we can open an issue to track it.
Best,
Liya Fan
On Fri, Mar 12, 2021 at 12:09 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Bob,
> Thanks for some feedback, I don't think a lot of people are developing on
> windows. Some answers in line:
>
> * Build does require Java 8, not "8 or later" as stated in java/README.md
> > There's a reference to sun.misc.Unsafe
> > in
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > which of course went away in JDK 9.
>
> Is this the case even using "-Dio.netty.tryReflectionSetAccessible=true" as
> mentioned in the README?
>
>
> * The build won't work on Windows because the java/format POM downloads a
> > binary flatc executable, but there's no artifact for Windows, just Linux
> > and OSX. I wound up downloading Visual Studio and building the
> flatbuffers
> > project.
>
> This unfortunately sounds familiar, I think this indicate the popularity on
> windows. I also think the hosting of the mac and linux are not exactly
> official (they are hosted by a former contributor). Updating the Readme
> might be a good first step with instructions on how to do this.
>
> I see in the pom.xml that user.timezone is set to UTC. I have seen these
> > types of errors in tests before; I know there are ways to insulate the
> test
> > from the user's current timezone but maybe someone knows what's going on
>
> This is somewhat surprising. I would thought we had the user.timezone set
> for exactly this reason. There might have been a regression, this might
> make a good second contribution if you wanted to look into fixing it.
>
> * I bumped into what looks like a spurious checkstyle error: it reports
> > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> > linefeed at the end when it definitely does. I set up Git not to do
> Windows
> > conversions, and I checked with various editors and binary dump
> utilities.
> > One source says that this because I'm running on Windows, checkstyle
> > actually expects a CR-LF and throws an error if it doesn't find it! I've
> > worked around this by disabling the check.
>
> It looks like we can force the checker to assume linux line feeds:
>
> https://stackoverflow.com/questions/997021/how-to-get-rid-of-checkstyle-message-file-does-not-end-with-a-newline
> (second answer). This would also make a good contribution for someone new.
>
> Cheers,
> Micah
>
>
>
>
>
>
>
>
>
>
> On Thu, Mar 11, 2021 at 7:14 PM bobtins <bo...@gmail.com> wrote:
>
> > My mail client took out all the linefeeds, so let me reformat; sorry
> about
> > that!
> >
> >
> > In the process of slogging through the build, I've bumped into various
> > issues. I'm happy to document them in java/README.md or make any other
> > changes that might be helpful to others.
> >
> > I'm pretty experienced with Java and Maven, so I think these are not
> > beginner's mistakes, but let me know if I'm missing something.
> >
> > A lot of these may be Windows-specific. I normally prefer Linux but just
> > got a new laptop and haven't set it up, but this experience is giving me
> a
> > lot of incentive to run screaming back to Linux ;-)
> >
> > Environment details:
> > * Windows 10
> > * Java 8
> > here's the output of java -version:
> > openjdk version "1.8.0_282"
> > OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
> > OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
> >
> > * Cygwin environment
> > * Maven 3.6.2
> >
> >
> > Issues encountered thus far:
> >
> > * Build does require Java 8, not "8 or later" as stated in java/README.md
> > There's a reference to sun.misc.Unsafe
> > in
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > which of course went away in JDK 9.
> > * The build won't work on Windows because the java/format POM downloads a
> > binary flatc executable, but there's no artifact for Windows, just Linux
> > and OSX. I wound up downloading Visual Studio and building the
> flatbuffers
> > project.
> >
> > * I bumped into what looks like a spurious checkstyle error: it reports
> > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> > linefeed at the end when it definitely does. I set up Git not to do
> Windows
> > conversions, and I checked with various editors and binary dump
> utilities.
> > One source says that this because I'm running on Windows, checkstyle
> > actually expects a CR-LF and throws an error if it doesn't find it! I've
> > worked around this by disabling the check.
> >
> > * The one thing that I'm stuck on now is failures on the jdbc module:
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Failures:
> > [ERROR]
> > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> > expected:<45935000> but was:<74735000>
> > [ERROR]
> > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> > expected:<1518439535000> but was:<1518468335000>
> > [ERROR]
> > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> > expected:<-365> but was:<-364>
> > [ERROR]
> >
> JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> > expected:<17574> but was:<17573>
> > [ERROR] JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> > expected:<17574> but was:<17573>
> > [ERROR]
> >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> > expected:<17574> but was:<17573>
> > [ERROR]
> >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> > expected:<17574> but was:<17573>
> > [INFO]
> > [ERROR] Tests run: 93, Failures: 7, Errors: 0, Skipped: 0
> >
> > I attached the full build output.
> > Looking more closely at these errors, they seem to be due to the timezone
> > difference; for example, the difference between 74735000 (actual value)
> and
> > 45935000 (expected) is 2880000, or 8 hours in milliseconds, which is the
> > PST timezone offset.
> >
> > I see in the pom.xml that user.timezone is set to UTC. I have seen these
> > types of errors in tests before; I know there are ways to insulate the
> test
> > from the user's current timezone but maybe someone knows what's going on.
> >
> > Thanks for any input!
> >
> >
> > On 2021/03/11 23:49:37, Bob Tinsman <bo...@pacbell.net> wrote:
> > > I've been mostly lurking for awhile, but I would like to start picking
> > off some bugs in the Java implementation.In the process of slogging
> through
> > the build, I've bumped into various issues. I'm happy to document them in
> > java/README.md or make any other changes that might be helpful to others.
> > I'm pretty experienced with Java and Maven, so I think these are not
> > super-obvious, but let me know if I'm missing something.A lot of these
> may
> > be Windows-specific. I normally prefer Linux but just got a new laptop
> and
> > haven't set it up, but this experience is giving me a lot of incentive to
> > run screaming back to Linux ;-)
> > > Environment details:- Windows 10- Java 8:openjdk version
> > "1.8.0_282"OpenJDK Runtime Environment (AdoptOpenJDK)(build
> > 1.8.0_282-b08)OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08,
> > mixed mode)- Cygwin environment- Maven 3.6.2
> > > Issues encountered thus far:- Build does require Java 8, not "8 or
> > later" as stated in java/README.md There's a reference to
> > sun.misc.Unsafe
> > in
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > which of course went away in JDK 9. I can update the build
> > instructions.- The build won't work on Windows because the java/format
> POM
> > downloads a binary flatc executables; when I looked, there was no version
> > for Windows, just Linux and OSX. I wound up downloading Visual Studio and
> > building the flatbuffers project.- I bumped into what looks like a
> spurious
> > checkstyle error: it
> > reports memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
> > having no linefeed at the end when it definitely does. I set up Git not
> to
> > do Windows conversions, and I checked with various editors and binary
> dump
> > utilities. One source says that this because I'm running on Windows,
> > checkstyle actually expects a CR-LF and throws an error if it doesn't
> find
> > it! I've worked
> > around this by disabling the check.- The one thing that I'm stuck on
> now
> > is failures on jdbc:
> > > [INFO] Results:[INFO][ERROR] Failures:[ERROR]
> > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> > expected:<45935000> but was:<74735000>[ERROR]
> > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> > expected:<1518439535000> but was:<1518468335000>[ERROR]
> > JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> > expected:<-365> but was:<-364>[ERROR]
> >
> JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> > expected:<17574> but was:<17573>[ERROR]
> > JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> > expected:<17574> but was:<17573>[ERROR]
> >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> > expected:<17574> but was:<17573>[ERROR]
> >
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> > expected:<17574> but was:<17573>[INFO][ERROR] Tests run: 93, Failures: 7,
> > Errors: 0, Skipped: 0
> > > I attached the full build output.Looking more closely at these errors,
> > they seem to be due to the timezone difference; for example, the
> difference
> > between 74735000 (actual value) and 45935000 (expected) is 2880000, or 8
> > hours in milliseconds, which is the PST timezone offset. I see in the
> > pom.xml that user.timezone is set to UTC. I have seen these types of
> errors
> > in tests before; I know there are ways to insulate the test from the
> user's
> > current timezone but maybe someone knows what's going on.
> > > Thanks for any input!
> > >
> >
>
Re: [JAVA] issues encountered during build
Posted by Micah Kornfield <em...@gmail.com>.
>
> I would think that this would show up in nightly builds. I guess I could
> try older versions, or I'll keep tracking it down to the cause.
I think by default most machines have a default timezone set to UTC, so it
might not.
On Fri, Mar 12, 2021 at 7:53 AM bobtins <bo...@gmail.com> wrote:
>
>
> On 2021/03/12 04:09:21, Micah Kornfield <em...@gmail.com> wrote:
> >
> > * Build does require Java 8, not "8 or later" as stated in java/README.md
> > > There's a reference to sun.misc.Unsafe
> > > in
> memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > > which of course went away in JDK 9.
> >
> > Is this the case even using "-Dio.netty.tryReflectionSetAccessible=true"
> as
> > mentioned in the README?
> >
> OK, now I can't duplicate the compile-time issue; now if I point it at JDK
> 11 it gets a warning, but previously it was actually unable to resolve
> "sun.misc.Unsafe" and got a compile time error; I'm not sure why it
> happened before and doesn't now, so I'll move on.
> >
> > * The build won't work on Windows because the java/format POM downloads a
> > > binary flatc executable, but there's no artifact for Windows, just
> Linux
> > > and OSX. I wound up downloading Visual Studio and building the
> flatbuffers
> > > project.
> >
> > This unfortunately sounds familiar, I think this indicate the popularity
> on
> > windows. I also think the hosting of the mac and linux are not exactly
> > official (they are hosted by a former contributor). Updating the Readme
> > might be a good first step with instructions on how to do this.
> >
> As far as providing instructions on building the flatbuffers project on
> Windows, I'm not an expert at all with the Microsoft ecosystem, but I could
> provide a summary.
>
> > I see in the pom.xml that user.timezone is set to UTC. I have seen these
> > > types of errors in tests before; I know there are ways to insulate the
> test
> > > from the user's current timezone but maybe someone knows what's going
> on
> >
> > This is somewhat surprising. I would thought we had the user.timezone
> set
> > for exactly this reason. There might have been a regression, this might
> > make a good second contribution if you wanted to look into fixing it.
> > ;
> I would think that this would show up in nightly builds. I guess I could
> try older versions, or I'll keep tracking it down to the cause.
>
> > * I bumped into what looks like a spurious checkstyle error: it reports
> > > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having
> no
> > > linefeed at the end when it definitely does. I set up Git not to do
> Windows
> > > conversions, and I checked with various editors and binary dump
> utilities.
> > > One source says that this because I'm running on Windows, checkstyle
> > > actually expects a CR-LF and throws an error if it doesn't find it!
> I've
> > > worked around this by disabling the check.
> >
> > It looks like we can force the checker to assume linux line feeds:
> >
> https://stackoverflow.com/questions/997021/how-to-get-rid-of-checkstyle-message-file-does-not-end-with-a-newline
> > (second answer). This would also make a good contribution for someone
> new.
> >
> Sure!
> >
> >
>
>
Re: [JAVA] issues encountered during build
Posted by bobtins <bo...@gmail.com>.
On 2021/03/12 04:09:21, Micah Kornfield <em...@gmail.com> wrote:
>
> * Build does require Java 8, not "8 or later" as stated in java/README.md
> > There's a reference to sun.misc.Unsafe
> > in memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> > which of course went away in JDK 9.
>
> Is this the case even using "-Dio.netty.tryReflectionSetAccessible=true" as
> mentioned in the README?
>
OK, now I can't duplicate the compile-time issue; now if I point it at JDK 11 it gets a warning, but previously it was actually unable to resolve "sun.misc.Unsafe" and got a compile time error; I'm not sure why it happened before and doesn't now, so I'll move on.
>
> * The build won't work on Windows because the java/format POM downloads a
> > binary flatc executable, but there's no artifact for Windows, just Linux
> > and OSX. I wound up downloading Visual Studio and building the flatbuffers
> > project.
>
> This unfortunately sounds familiar, I think this indicate the popularity on
> windows. I also think the hosting of the mac and linux are not exactly
> official (they are hosted by a former contributor). Updating the Readme
> might be a good first step with instructions on how to do this.
>
As far as providing instructions on building the flatbuffers project on Windows, I'm not an expert at all with the Microsoft ecosystem, but I could provide a summary.
> I see in the pom.xml that user.timezone is set to UTC. I have seen these
> > types of errors in tests before; I know there are ways to insulate the test
> > from the user's current timezone but maybe someone knows what's going on
>
> This is somewhat surprising. I would thought we had the user.timezone set
> for exactly this reason. There might have been a regression, this might
> make a good second contribution if you wanted to look into fixing it.
> ;
I would think that this would show up in nightly builds. I guess I could try older versions, or I'll keep tracking it down to the cause.
> * I bumped into what looks like a spurious checkstyle error: it reports
> > memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> > linefeed at the end when it definitely does. I set up Git not to do Windows
> > conversions, and I checked with various editors and binary dump utilities.
> > One source says that this because I'm running on Windows, checkstyle
> > actually expects a CR-LF and throws an error if it doesn't find it! I've
> > worked around this by disabling the check.
>
> It looks like we can force the checker to assume linux line feeds:
> https://stackoverflow.com/questions/997021/how-to-get-rid-of-checkstyle-message-file-does-not-end-with-a-newline
> (second answer). This would also make a good contribution for someone new.
>
Sure!
>
>
Re: [JAVA] issues encountered during build
Posted by Micah Kornfield <em...@gmail.com>.
Hi Bob,
Thanks for some feedback, I don't think a lot of people are developing on
windows. Some answers in line:
* Build does require Java 8, not "8 or later" as stated in java/README.md
> There's a reference to sun.misc.Unsafe
> in memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> which of course went away in JDK 9.
Is this the case even using "-Dio.netty.tryReflectionSetAccessible=true" as
mentioned in the README?
* The build won't work on Windows because the java/format POM downloads a
> binary flatc executable, but there's no artifact for Windows, just Linux
> and OSX. I wound up downloading Visual Studio and building the flatbuffers
> project.
This unfortunately sounds familiar, I think this indicate the popularity on
windows. I also think the hosting of the mac and linux are not exactly
official (they are hosted by a former contributor). Updating the Readme
might be a good first step with instructions on how to do this.
I see in the pom.xml that user.timezone is set to UTC. I have seen these
> types of errors in tests before; I know there are ways to insulate the test
> from the user's current timezone but maybe someone knows what's going on
This is somewhat surprising. I would thought we had the user.timezone set
for exactly this reason. There might have been a regression, this might
make a good second contribution if you wanted to look into fixing it.
* I bumped into what looks like a spurious checkstyle error: it reports
> memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> linefeed at the end when it definitely does. I set up Git not to do Windows
> conversions, and I checked with various editors and binary dump utilities.
> One source says that this because I'm running on Windows, checkstyle
> actually expects a CR-LF and throws an error if it doesn't find it! I've
> worked around this by disabling the check.
It looks like we can force the checker to assume linux line feeds:
https://stackoverflow.com/questions/997021/how-to-get-rid-of-checkstyle-message-file-does-not-end-with-a-newline
(second answer). This would also make a good contribution for someone new.
Cheers,
Micah
On Thu, Mar 11, 2021 at 7:14 PM bobtins <bo...@gmail.com> wrote:
> My mail client took out all the linefeeds, so let me reformat; sorry about
> that!
>
>
> In the process of slogging through the build, I've bumped into various
> issues. I'm happy to document them in java/README.md or make any other
> changes that might be helpful to others.
>
> I'm pretty experienced with Java and Maven, so I think these are not
> beginner's mistakes, but let me know if I'm missing something.
>
> A lot of these may be Windows-specific. I normally prefer Linux but just
> got a new laptop and haven't set it up, but this experience is giving me a
> lot of incentive to run screaming back to Linux ;-)
>
> Environment details:
> * Windows 10
> * Java 8
> here's the output of java -version:
> openjdk version "1.8.0_282"
> OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
> OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
>
> * Cygwin environment
> * Maven 3.6.2
>
>
> Issues encountered thus far:
>
> * Build does require Java 8, not "8 or later" as stated in java/README.md
> There's a reference to sun.misc.Unsafe
> in memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> which of course went away in JDK 9.
> * The build won't work on Windows because the java/format POM downloads a
> binary flatc executable, but there's no artifact for Windows, just Linux
> and OSX. I wound up downloading Visual Studio and building the flatbuffers
> project.
>
> * I bumped into what looks like a spurious checkstyle error: it reports
> memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java having no
> linefeed at the end when it definitely does. I set up Git not to do Windows
> conversions, and I checked with various editors and binary dump utilities.
> One source says that this because I'm running on Windows, checkstyle
> actually expects a CR-LF and throws an error if it doesn't find it! I've
> worked around this by disabling the check.
>
> * The one thing that I'm stuck on now is failures on the jdbc module:
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]
> JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> expected:<45935000> but was:<74735000>
> [ERROR]
> JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> expected:<1518439535000> but was:<1518468335000>
> [ERROR]
> JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> expected:<-365> but was:<-364>
> [ERROR]
> JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> expected:<17574> but was:<17573>
> [ERROR] JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> expected:<17574> but was:<17573>
> [ERROR]
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> expected:<17574> but was:<17573>
> [ERROR]
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> expected:<17574> but was:<17573>
> [INFO]
> [ERROR] Tests run: 93, Failures: 7, Errors: 0, Skipped: 0
>
> I attached the full build output.
> Looking more closely at these errors, they seem to be due to the timezone
> difference; for example, the difference between 74735000 (actual value) and
> 45935000 (expected) is 2880000, or 8 hours in milliseconds, which is the
> PST timezone offset.
>
> I see in the pom.xml that user.timezone is set to UTC. I have seen these
> types of errors in tests before; I know there are ways to insulate the test
> from the user's current timezone but maybe someone knows what's going on.
>
> Thanks for any input!
>
>
> On 2021/03/11 23:49:37, Bob Tinsman <bo...@pacbell.net> wrote:
> > I've been mostly lurking for awhile, but I would like to start picking
> off some bugs in the Java implementation.In the process of slogging through
> the build, I've bumped into various issues. I'm happy to document them in
> java/README.md or make any other changes that might be helpful to others.
> I'm pretty experienced with Java and Maven, so I think these are not
> super-obvious, but let me know if I'm missing something.A lot of these may
> be Windows-specific. I normally prefer Linux but just got a new laptop and
> haven't set it up, but this experience is giving me a lot of incentive to
> run screaming back to Linux ;-)
> > Environment details:- Windows 10- Java 8:openjdk version
> "1.8.0_282"OpenJDK Runtime Environment (AdoptOpenJDK)(build
> 1.8.0_282-b08)OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08,
> mixed mode)- Cygwin environment- Maven 3.6.2
> > Issues encountered thus far:- Build does require Java 8, not "8 or
> later" as stated in java/README.md There's a reference to
> sun.misc.Unsafe
> in memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
> which of course went away in JDK 9. I can update the build
> instructions.- The build won't work on Windows because the java/format POM
> downloads a binary flatc executables; when I looked, there was no version
> for Windows, just Linux and OSX. I wound up downloading Visual Studio and
> building the flatbuffers project.- I bumped into what looks like a spurious
> checkstyle error: it
> reports memory/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
> having no linefeed at the end when it definitely does. I set up Git not to
> do Windows conversions, and I checked with various editors and binary dump
> utilities. One source says that this because I'm running on Windows,
> checkstyle actually expects a CR-LF and throws an error if it doesn't find
> it! I've worked
> around this by disabling the check.- The one thing that I'm stuck on now
> is failures on jdbc:
> > [INFO] Results:[INFO][ERROR] Failures:[ERROR]
> JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:209
> expected:<45935000> but was:<74735000>[ERROR]
> JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:213
> expected:<1518439535000> but was:<1518468335000>[ERROR]
> JdbcToArrowDataTypesTest.testJdbcToArrowValues:146->testDataSets:205
> expected:<-365> but was:<-364>[ERROR]
> JdbcToArrowNullTest.testJdbcToArrowValues:123->testDataSets:165->testAllVectorValues:209
> expected:<17574> but was:<17573>[ERROR]
> JdbcToArrowTest.testJdbcToArrowValues:138->testDataSets:206
> expected:<17574> but was:<17573>[ERROR]
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValuesNoLimit:107->validate:199->assertDateDayVectorValues:277
> expected:<17574> but was:<17573>[ERROR]
> JdbcToArrowVectorIteratorTest.testJdbcToArrowValues:95->validate:199->assertDateDayVectorValues:277
> expected:<17574> but was:<17573>[INFO][ERROR] Tests run: 93, Failures: 7,
> Errors: 0, Skipped: 0
> > I attached the full build output.Looking more closely at these errors,
> they seem to be due to the timezone difference; for example, the difference
> between 74735000 (actual value) and 45935000 (expected) is 2880000, or 8
> hours in milliseconds, which is the PST timezone offset. I see in the
> pom.xml that user.timezone is set to UTC. I have seen these types of errors
> in tests before; I know there are ways to insulate the test from the user's
> current timezone but maybe someone knows what's going on.
> > Thanks for any input!
> >
>