You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "llama90 (via GitHub)" <gi...@apache.org> on 2024/06/08 16:25:56 UTC

[PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

llama90 opened a new pull request, #42038:
URL: https://github.com/apache/arrow/pull/42038

   <!--
   Thanks for opening a pull request!
   If this is your first pull request you can find detailed information on how 
   to contribute here:
     * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
     * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
   
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   In the case of PARQUET issues on JIRA the title also supports:
   
       PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   
   Update package from JUnit 4(`org.junit`) to JUnit 5(`org.junit.jupiter`).
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ### What changes are included in this PR?
   
   - `avro` and `jdbc` module
     - [x] Replacing `org.junit` with `org.junit.jupiter.api`.
     - [x] Updating `Assertions.assertXXX` to `assertXXX` using static imports.
     - [x] Updating annotations such as `@Before`, `@After`.
       - `@Before` -> `@BeforeEach`
       - `@After` -> `@AfterEach`
       - `@Test` -> `@Test` with `org.junit.jupiter`
       - `@ClassRule` -> `@TempDir` and `@BeforeAll`
     - [x] Updating `Parameterized` test
     - [ ] Exploring a more effective structure for `ParameterizedTest` in JDBC tests. 
     - [ ] Doing self review
   - `orc` module
     - [ ] Reviewing the build method
     - [ ] ...
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ### Are these changes tested?
   
   Yes, existing tests have passed.
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ### Are there any user-facing changes?
   
   No.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
   -->
   <!-- **This PR includes breaking changes to public APIs.** -->
   
   <!--
   Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
   -->
   <!-- **This PR contains a "Critical Fix".** -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637564428


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   @llama90 CI seems to be still failing? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1632430705


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/h2/JdbcToArrowCharSetTest.java:
##########
@@ -60,24 +57,9 @@ public class JdbcToArrowCharSetTest extends AbstractJdbcToArrowTest {
     "h2/test1_charset_kr_h2.yml"
   };
 
-  /**
-   * Constructor which populates the table object for each test iteration.
-   *
-   * @param table Table oject
-   */
-  public JdbcToArrowCharSetTest(Table table) {
+  public void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   should we `@Override` here? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1632430530


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/AbstractJdbcToArrowTest.java:
##########
@@ -91,15 +89,15 @@ protected static Table getTable(String ymlFilePath, @SuppressWarnings("rawtypes"
             clss.getClassLoader().getResourceAsStream(ymlFilePath), Table.class);
   }
 
-
   /**
    * This method creates Connection object and DB table and also populate data into table for test.
    *
    * @throws SQLException on error
    * @throws ClassNotFoundException on error
    */
-  @Before
-  public void setUp() throws SQLException, ClassNotFoundException {
+  protected void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   I guess what you are concerned about is making the database connection for each test case with `@BeforeEach` setup method? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2165382368

   > @llama90 the CI failures are due to spotless issue. Could you please try `mvn spotless:apply` before building? We will integrate `pre-commit` sooner, so that this won't be an issue.
   
   Yes, I tried to check (`mvn spotless:apply`) before PR, but sometimes I missed it 🥲
   
   I can't test the environment for Windows locally, I am repeatedly testing it through CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637875726


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   Yes, let's try that out. I am a little less responsive today. Sorry about that. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2156094304

   :warning: GitHub issue #42030 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1638386051


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroToArrowIteratorTest.java:
##########


Review Comment:
   I was getting an error in Windows saying "failed to delete temp directory" because "The process cannot access the file because it is being used by another process."
   
   - [Error Link](https://github.com/apache/arrow/actions/runs/9498747854/job/26178122374?pr=42038#step:5:2537)
   
   To fix this, I found out that using `try-with-resources` to properly close resources would help.
   
   But for the `convert` method, the `FileInputStream` (fis) needs to stay open for the `iterator` after the `convert` method is called. This caused the error because the `fis` wasn't properly closed. So, I made a `writeDataToFile` method for the `FileOutputStream` and used `try-with-resources` for `fis` in the same method where the `iterator` is used.
   
   Now, the tests run fine on Windows too.
   
   Here are the different approaches I tried:
   
   **Attempt 1:**
   
   convert method:
   - FileOutputStream with `try-with-resources`
   - FileInputStream with `try-with-resources`
   
   Error: `Stream Closed` - because the `fis` was closed while the `iterator` still needed it.
   
   **Attempt 2:**
   
   convert method:
   - FileOutputStream with `try-with-resources`
   - FileInputStream
   
   Error: `failed to delete temp directory` - because the `fis` wasn't closed.
   
   **Attempt 3:**
   
   convert method:
   - FileOutputStream with `try-with-resources`
   - Added `FileInputStream` as a parameter to the convert method and explicitly closed it in the test method.
   
   Didn't work as expected.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1635852451


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/AbstractJdbcToArrowTest.java:
##########
@@ -91,15 +89,15 @@ protected static Table getTable(String ymlFilePath, @SuppressWarnings("rawtypes"
             clss.getClassLoader().getResourceAsStream(ymlFilePath), Table.class);
   }
 
-
   /**
    * This method creates Connection object and DB table and also populate data into table for test.
    *
    * @throws SQLException on error
    * @throws ClassNotFoundException on error
    */
-  @Before
-  public void setUp() throws SQLException, ClassNotFoundException {
+  protected void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   I think this is fine



##########
java/bom/pom.xml:
##########
@@ -177,6 +185,10 @@
         <artifactId>spotless-maven-plugin</artifactId>
         <configuration>
           <pom>
+            <licenseHeader>
+              <file>${maven.multiModuleProjectDirectory}/dev/license/asf-xml.license</file>
+              <delimiter>(&lt;configuration|&lt;project)</delimiter>
+            </licenseHeader>

Review Comment:
   Where did this come from?



##########
java/bom/pom.xml:
##########
@@ -177,6 +185,10 @@
         <artifactId>spotless-maven-plugin</artifactId>
         <configuration>
           <pom>
+            <licenseHeader>
+              <file>${maven.multiModuleProjectDirectory}/dev/license/asf-xml.license</file>
+              <delimiter>(&lt;configuration|&lt;project)</delimiter>
+            </licenseHeader>

Review Comment:
   Oh, maybe this is a merge artifact...it seems to already be on main?



##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -45,18 +45,17 @@
 import org.apache.avro.io.DatumWriter;
 import org.apache.avro.io.DecoderFactory;
 import org.apache.avro.io.EncoderFactory;
-import org.junit.Before;
-import org.junit.ClassRule;
-import org.junit.rules.TemporaryFolder;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.io.TempDir;
 
 public class AvroTestBase {
 
-  @ClassRule
-  public static final TemporaryFolder TMP = new TemporaryFolder();
+  @TempDir
+  public File TMP;

Review Comment:
   The new JUnit API is different.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637392005


##########
java/bom/pom.xml:
##########
@@ -177,6 +185,10 @@
         <artifactId>spotless-maven-plugin</artifactId>
         <configuration>
           <pom>
+            <licenseHeader>
+              <file>${maven.multiModuleProjectDirectory}/dev/license/asf-xml.license</file>
+              <delimiter>(&lt;configuration|&lt;project)</delimiter>
+            </licenseHeader>

Review Comment:
   I think we need to fix it here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1632428811


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -45,18 +45,17 @@
 import org.apache.avro.io.DatumWriter;
 import org.apache.avro.io.DecoderFactory;
 import org.apache.avro.io.EncoderFactory;
-import org.junit.Before;
-import org.junit.ClassRule;
-import org.junit.rules.TemporaryFolder;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.io.TempDir;
 
 public class AvroTestBase {
 
-  @ClassRule
-  public static final TemporaryFolder TMP = new TemporaryFolder();
+  @TempDir
+  public File TMP;

Review Comment:
   Any specific reason for this change? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637893463


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   No problem! Please take a look when you have free time.
   
   Fortunately, we are making progress, and we just need to handle the `AvroToArrowIteratorTest` class properly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637893463


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   No problem! Please take a look when you have free time.
   
   Fortunately, we are making progress, and we just need to handle the `AvroToArrowIteratorTest` class properly.
   
   UPDATE: Also `TestWriteReadAvroRecord`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1636461952


##########
java/bom/pom.xml:
##########
@@ -177,6 +185,10 @@
         <artifactId>spotless-maven-plugin</artifactId>
         <configuration>
           <pom>
+            <licenseHeader>
+              <file>${maven.multiModuleProjectDirectory}/dev/license/asf-xml.license</file>
+              <delimiter>(&lt;configuration|&lt;project)</delimiter>
+            </licenseHeader>

Review Comment:
   I did a rebase and saw that changes from another PR were left out.
   
   What do you think about issues with `@TempDir` when using JDK 11 on Windows?
   
   - https://github.com/apache/arrow/pull/42038#issuecomment-2156643841
   
   If possible, can I fix the problem with a new PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2169021127

   > Thanks for figuring it out! So basically, we weren't properly closing the file in all cases and that stopped Windows from deleting the temp files?
   
   File deletion operations differ across operating systems. Specifically, Windows only marks a file for deletion on close. Therefore, it's important to use explicit resource management, such as try-with-resources, to ensure proper release of resources.
   
   - reference: https://github.com/junit-team/junit5/issues/2811#issuecomment-1703476137
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637388923


##########
java/bom/pom.xml:
##########
@@ -177,6 +185,10 @@
         <artifactId>spotless-maven-plugin</artifactId>
         <configuration>
           <pom>
+            <licenseHeader>
+              <file>${maven.multiModuleProjectDirectory}/dev/license/asf-xml.license</file>
+              <delimiter>(&lt;configuration|&lt;project)</delimiter>
+            </licenseHeader>

Review Comment:
   @lidavidm we seems to have a problem here with Windows: https://github.com/apache/arrow/actions/runs/9490384583/job/26153755020?pr=42038#step:5:10625



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637526807


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   not sure if this is would do any affect here? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637396857


##########
java/bom/pom.xml:
##########
@@ -177,6 +185,10 @@
         <artifactId>spotless-maven-plugin</artifactId>
         <configuration>
           <pom>
+            <licenseHeader>
+              <file>${maven.multiModuleProjectDirectory}/dev/license/asf-xml.license</file>
+              <delimiter>(&lt;configuration|&lt;project)</delimiter>
+            </licenseHeader>

Review Comment:
   Could this be helpful?
   
   https://github.com/junit-team/junit5/issues/3236
   https://github.com/junit-team/junit5/pull/3249
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1638386051


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroToArrowIteratorTest.java:
##########


Review Comment:
   I was getting an error in **Windows** saying "failed to delete temp directory" because "The process cannot access the file because it is being used by another process."
   
   - [Error Link](https://github.com/apache/arrow/actions/runs/9498747854/job/26178122374?pr=42038#step:5:2537)
   
   To fix this, I found out that using `try-with-resources` to properly close resources would help.
   
   But for the `convert` method, the `FileInputStream` (fis) needs to stay open for the `iterator` after the `convert` method is called. This caused the error because the `fis` wasn't properly closed. So, I made a `writeDataToFile` method for the `FileOutputStream` and used `try-with-resources` for `fis` in the same method where the `iterator` is used.
   
   Now, the tests run fine on Windows too.
   
   Here are the different approaches I tried:
   
   **Attempt 1:**
   
   convert method:
   - FileOutputStream with `try-with-resources`
   - FileInputStream with `try-with-resources`
   
   Error: `Stream Closed` - because the `fis` was closed while the `iterator` still needed it.
   
   **Attempt 2:**
   
   convert method:
   - FileOutputStream with `try-with-resources`
   - FileInputStream
   
   Error: `failed to delete temp directory` - because the `fis` wasn't closed.
   
   **Attempt 3:**
   
   convert method:
   - FileOutputStream with `try-with-resources`
   - Added `FileInputStream` as a parameter to the convert method and explicitly closed it in the test method.
   
   Didn't work as expected.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1632268067


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/AbstractJdbcToArrowTest.java:
##########
@@ -91,15 +89,15 @@ protected static Table getTable(String ymlFilePath, @SuppressWarnings("rawtypes"
             clss.getClassLoader().getResourceAsStream(ymlFilePath), Table.class);
   }
 
-
   /**
    * This method creates Connection object and DB table and also populate data into table for test.
    *
    * @throws SQLException on error
    * @throws ClassNotFoundException on error
    */
-  @Before
-  public void setUp() throws SQLException, ClassNotFoundException {
+  protected void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   @llama90 I will review this tomorrow. Thanks. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #42038:
URL: https://github.com/apache/arrow/pull/42038


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637546739


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   Actually, we can't know for sure what effect it will have.
   
   Referring to the issue you shared, the problem on Windows + JDK 11 still seems to be open and unresolved. I saw a temporary workaround in the comments, so I tested it.
   
   - https://github.com/junit-team/junit5/issues/2811#issuecomment-2084917705
   
   At the moment, this seems to be the worst approach and the last resort.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637134996


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/h2/JdbcToArrowTimeZoneTest.java:
##########
@@ -129,6 +121,25 @@ public void testJdbcToArrowValues() throws SQLException, IOException {
                     Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone())))
                 .build()),
         false);
+    testDataSets(
+        sqlToArrow(
+            conn.createStatement().executeQuery(table.getQuery()),
+            new RootAllocator(Integer.MAX_VALUE),
+            Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone()))),
+        false);
+    testDataSets(
+        sqlToArrow(
+            conn.createStatement().executeQuery(table.getQuery()),
+            Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone()))),
+        false);
+    testDataSets(
+        sqlToArrow(
+            conn.createStatement().executeQuery(table.getQuery()),
+            new JdbcToArrowConfigBuilder(
+                    new RootAllocator(Integer.MAX_VALUE),
+                    Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone())))
+                .build()),
+        false);

Review Comment:
   Good catch. It was my mistake. I have removed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637564158


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   Yeah, I see your point. I am wondering, does this change, I mean using the temporary directory is in other modules too? 
   Just want to get an idea and see what we could do.
   
   cc @lidavidm WDYT? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2156643841

   Ummm....
   
   There is a JUnit 5 bug on Windows (Server 2022) with JDK 11, specifically in the CI used by Arrow.
   
   - https://github.com/junit-team/junit5/issues/2811
   - https://github.com/apache/arrow/actions/runs/9437309980/job/25993017637?pr=42038#step:5:2647
   
   Should we consider a rollback because of this, or is there another solution we could explore?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1633261125


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -45,18 +45,17 @@
 import org.apache.avro.io.DatumWriter;
 import org.apache.avro.io.DecoderFactory;
 import org.apache.avro.io.EncoderFactory;
-import org.junit.Before;
-import org.junit.ClassRule;
-import org.junit.rules.TemporaryFolder;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.io.TempDir;
 
 public class AvroTestBase {
 
-  @ClassRule
-  public static final TemporaryFolder TMP = new TemporaryFolder();
+  @TempDir
+  public File TMP;

Review Comment:
   I understand that in JUnit 5, the alternative for temporary directories is to use `@TempDir`. So I modified it accordingly.
   
   - [Annotation Interface TempDir](https://junit.org/junit5/docs/5.10.2/api/org.junit.jupiter.api/org/junit/jupiter/api/io/TempDir.html)
   
   > `@TempDir` can be used to annotate a field in a test class or a parameter in a lifecycle method or test method of type [Path](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/Path.html) or [File](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/File.html) that should be resolved into a temporary directory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1633303507


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/h2/JdbcToArrowCharSetTest.java:
##########
@@ -60,24 +57,9 @@ public class JdbcToArrowCharSetTest extends AbstractJdbcToArrowTest {
     "h2/test1_charset_kr_h2.yml"
   };
 
-  /**
-   * Constructor which populates the table object for each test iteration.
-   *
-   * @param table Table oject
-   */
-  public JdbcToArrowCharSetTest(Table table) {
+  public void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   I reflected



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637569425


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   Even for us, it seems impossible to apply this method. In reality, when running multiple tests, we observed that some tests fail randomly.
   
   First Failed Case:
   - AvroSkipFieldTest.testSkipEnumField
   - AvroToArrowTest.testIntType
    
   Second Failed Case:
   - AvroToArrowTest.testLongType
   - AvroToArrowTest.testEnumAttributes
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637569425


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   Even for us, it seems impossible to apply this method. In reality, when running multiple tests, we observed that some tests fail randomly. 😱
   
   First Failed Case:
   - AvroSkipFieldTest.testSkipEnumField
   - AvroToArrowTest.testIntType
    
   Second Failed Case:
   - AvroToArrowTest.testLongType
   - AvroToArrowTest.testEnumAttributes
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637573978


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   I think we might be able to avoid the error by using `@TempDir` in a different way. In fact, the PR for Arrow does not encounter the same error with `TempDir`.
   
   - #42093 
   
   I will modify the code and test it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2165962433

   @vibhatha @lidavidm lol. Finally, it looks I've resolved the problem!
   
   I would appreciate it if you could review this when you have free time.
   
   - [comment](https://github.com/apache/arrow/pull/42038#discussion_r1638386051)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1636764318


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -45,18 +45,17 @@
 import org.apache.avro.io.DatumWriter;
 import org.apache.avro.io.DecoderFactory;
 import org.apache.avro.io.EncoderFactory;
-import org.junit.Before;
-import org.junit.ClassRule;
-import org.junit.rules.TemporaryFolder;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.io.TempDir;
 
 public class AvroTestBase {
 
-  @ClassRule
-  public static final TemporaryFolder TMP = new TemporaryFolder();
+  @TempDir
+  public File TMP;

Review Comment:
   Oh, that seems to be related, though can we re-check by running CIs? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1633248953


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/AbstractJdbcToArrowTest.java:
##########
@@ -91,15 +89,15 @@ protected static Table getTable(String ymlFilePath, @SuppressWarnings("rawtypes"
             clss.getClassLoader().getResourceAsStream(ymlFilePath), Table.class);
   }
 
-
   /**
    * This method creates Connection object and DB table and also populate data into table for test.
    *
    * @throws SQLException on error
    * @throws ClassNotFoundException on error
    */
-  @Before
-  public void setUp() throws SQLException, ClassNotFoundException {
+  protected void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   Yes, we need to create a database connection with a different `.yml` for each test, but `@BeforeEach` cannot take arguments for `@ParameterizedTest`.
   
   I described something similar [here](https://github.com/apache/arrow/pull/42029#discussion_r1633208511).
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2156875072

   > Ummm....
   > 
   > There is a JUnit 5 bug on Windows (Server 2022) with JDK 11, specifically in the CI used by Arrow.
   > 
   >     * [@TempDir directory cannot be deleted on Windows with Java 11 junit-team/junit5#2811](https://github.com/junit-team/junit5/issues/2811)
   > 
   >     * https://github.com/apache/arrow/actions/runs/9437309980/job/25993017637?pr=42038#step:5:2647
   > 
   > 
   > Should we consider a rollback because of this, or is there another solution we could explore?
   > error message
   > 
   > ```
   > Error:  Tests run: 5, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 3.553 s <<< FAILURE! -- in org.apache.arrow.adapter.avro.AvroToArrowIteratorTest
   > [INFO] --- maven-surefire-plugin:3.2.5:test (default-test) @ arrow-jdbc ---
   > Error:  org.apache.arrow.adapter.avro.AvroToArrowIteratorTest.testArrayType -- Time elapsed: 0.125 s <<< ERROR!
   > java.io.IOException: Failed to delete temp directory D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427. The following paths could not be deleted (see suppressed exceptions for details): <root>, test.avro
   > 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
   > 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
   > 	at java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
   > 	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258)
   > 	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258)
   > 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
   > 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
   > 	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
   > 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
   > 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   > 	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
   > 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
   > 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
   > 	Suppressed: java.nio.file.DirectoryNotEmptyException: D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427
   > 		at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:271)
   > 		at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
   > 		at java.base/java.nio.file.Files.delete(Files.java:1142)
   > 		at java.base/java.nio.file.Files.walkFileTree(Files.java:2743)
   > 		at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
   > 		... 13 more
   > 	Suppressed: java.nio.file.FileSystemException: D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427\test.avro: The process cannot access the file because it is being used by another process.
   > 
   > 		at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
   > 		at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
   > 		at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
   > 		at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274)
   > 		at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
   > 		at java.base/java.nio.file.Files.delete(Files.java:1142)
   > 		at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
   > 		at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
   > 		... 13 more
   > 		Suppressed: java.nio.file.FileSystemException: D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427\test.avro: The process cannot access the file because it is being used by another process.
   > 
   > 			... 21 more
   > ```
   > 
   > UPDATE:
   > 
   > I tried two kinds of approaches
   > #### Using `try-with-resources` Statements for `FileOutputStream` and `FileInputStream`
   > 
   > ```java
   > try (FileOutputStream outStream = new FileOutputStream(dataFile);
   >      FileInputStream inStream = new FileInputStream(dataFile)) {
   >     BinaryEncoder encoder = new EncoderFactory().directBinaryEncoder(outStream, null);
   >     DatumWriter<Object> writer = new GenericDatumWriter<>(schema);
   >     for (Object value : data) {
   >         writer.write(value, encoder);
   >     }
   >     outStream.flush();
   > 
   >     BinaryDecoder decoder = new DecoderFactory().directBinaryDecoder(inStream, null);
   >     return AvroToArrow.avroToArrow(schema, decoder, config);
   > }
   > ```
   > 
   > #### Separating `FileOutputStream` and `FileInputStream` Usage to Avoid Concurrent Access
   > 
   > ```java
   >   protected VectorSchemaRoot writeAndRead(Schema schema, List data) throws Exception {
   >     File dataFile = new File(TMP, "test.avro");
   > 
   >     try (FileOutputStream out = new FileOutputStream(dataFile)) {
   >       BinaryEncoder encoder = new EncoderFactory().directBinaryEncoder(out, null);
   >       DatumWriter<Object> writer = new GenericDatumWriter<>(schema);
   >       for (Object value : data) {
   >         writer.write(value, encoder);
   >       }
   >       out.flush();
   >     }
   > 
   >     try (FileInputStream in = new FileInputStream(dataFile)) {
   >       BinaryDecoder decoder = new DecoderFactory().directBinaryDecoder(in, null);
   >       return AvroToArrow.avroToArrow(schema, decoder, config);
   >     }
   >   }
   > ```
   
   @llama90 thanks for taking a very thorough assessment on this. I think the [issue](https://github.com/apache/arrow/actions/runs/9437483721/job/25993406708?pr=42038#step:5:10715) still persists. Furthermore looking into the JUNIT5 bug report, it seems that there isn't a solid fix yet. So my suggestion would be to fix it once that's resolved. 
   
   Although, we can create two sub-issues for this ticket, one for functional conversion (orc) and other for non-functional bits. 
   So that we can track the failing one later. But I have doubts about that approach. 
   
   cc @lidavidm Appreciate your feedback. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1636658726


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/h2/JdbcToArrowTimeZoneTest.java:
##########
@@ -129,6 +121,25 @@ public void testJdbcToArrowValues() throws SQLException, IOException {
                     Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone())))
                 .build()),
         false);
+    testDataSets(
+        sqlToArrow(
+            conn.createStatement().executeQuery(table.getQuery()),
+            new RootAllocator(Integer.MAX_VALUE),
+            Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone()))),
+        false);
+    testDataSets(
+        sqlToArrow(
+            conn.createStatement().executeQuery(table.getQuery()),
+            Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone()))),
+        false);
+    testDataSets(
+        sqlToArrow(
+            conn.createStatement().executeQuery(table.getQuery()),
+            new JdbcToArrowConfigBuilder(
+                    new RootAllocator(Integer.MAX_VALUE),
+                    Calendar.getInstance(TimeZone.getTimeZone(table.getTimezone())))
+                .build()),
+        false);

Review Comment:
   could I get bit of an explanation about this part? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1632070214


##########
java/adapter/jdbc/src/test/java/org/apache/arrow/adapter/jdbc/AbstractJdbcToArrowTest.java:
##########
@@ -91,15 +89,15 @@ protected static Table getTable(String ymlFilePath, @SuppressWarnings("rawtypes"
             clss.getClassLoader().getResourceAsStream(ymlFilePath), Table.class);
   }
 
-
   /**
    * This method creates Connection object and DB table and also populate data into table for test.
    *
    * @throws SQLException on error
    * @throws ClassNotFoundException on error
    */
-  @Before
-  public void setUp() throws SQLException, ClassNotFoundException {
+  protected void initializeDatabase(Table table) throws SQLException, ClassNotFoundException {

Review Comment:
   @vibhatha I'll look into it more, but if you have any suggestions for a better structure, I would really appreciate your advice.
   
   cc @lidavidm
   
   ```mermaid
   classDiagram
       direction LR
       class AbstractJdbcToArrowTest {
       }
   
       class JdbcToArrowCharSetTest {
       }
   
       class JdbcToArrowDataTypesTest {
       }
   
       class JdbcToArrowMapDataTypeTest {
       }
   
       class JdbcToArrowNullTest {
       }
   
       class JdbcToArrowOptionalColumnsTest {
       }
   
       class JdbcToArrowTest {
       }
   
       class JdbcToArrowTimeZoneTest {
       }
   
       class JdbcToArrowVectorIteratorTest {
       }
   
       AbstractJdbcToArrowTest <|-- JdbcToArrowCharSetTest
       AbstractJdbcToArrowTest <|-- JdbcToArrowDataTypesTest
       AbstractJdbcToArrowTest <|-- JdbcToArrowMapDataTypeTest
       AbstractJdbcToArrowTest <|-- JdbcToArrowNullTest
       AbstractJdbcToArrowTest <|-- JdbcToArrowOptionalColumnsTest
       AbstractJdbcToArrowTest <|-- JdbcToArrowTest
       AbstractJdbcToArrowTest <|-- JdbcToArrowTimeZoneTest
       JdbcToArrowTest <|-- JdbcToArrowVectorIteratorTest
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2156095401

   I currently use explicit calls to `initializeDatabase` in each test method. However, I believe there is a more efficient structure for this process.
   
   ### Reason
   
   - In JUnit 4, the method calling sequence is: `constructor` -> `@Before` -> `@ParameterizedTest`.
     - In JUnit 5, the constructor is not used for `@ParameterizedTest`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1639038758


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroToArrowIteratorTest.java:
##########


Review Comment:
   Thanks for pushing this one and evaluating diverse approaches. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2169228541

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 870b315d4b766a671cd5e478e5b9776beb00ff59.
   
   There were 12 benchmark results indicating a performance regression:
   
   - Commit Run on `test-mac-arm` at [2024-06-14 14:37:43Z](https://conbench.ursa.dev/compare/runs/3b1a4e29c56b49aeb665d879827144dc...9f5f1292d4084c768ae802217a021d40/)
     - [`tpch` (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-07, scale_factor=1](https://conbench.ursa.dev/compare/benchmarks/0666c40b04757c5480006e6b5d10dcc9...0666c5f09f167a9b80008449842558e7)
     - [`tpch` (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-17, scale_factor=1](https://conbench.ursa.dev/compare/benchmarks/0666c41a89f67196800003e4be3c6b8c...0666c600cf7c738d80006ae02fe34203)
   - and 10 more (see the report linked below)
   
   The [full Conbench report](https://github.com/apache/arrow/runs/26263048135) has more details. It also includes information about 59 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on code in PR #42038:
URL: https://github.com/apache/arrow/pull/42038#discussion_r1637900791


##########
java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroTestBase.java:
##########
@@ -58,6 +59,11 @@ public void init() {
     config = new AvroToArrowConfigBuilder(allocator).build();
   }
 
+  @AfterEach
+  public void tearDown() {
+    System.gc();

Review Comment:
   Thanks for working on this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2165743808

   > Yes, I tried to check (`mvn spotless:apply`) before PR, but sometimes I missed it 🥲
   
   Yeah that happens, and after pre-commit that will be warned before committing. 
   
   > I can't test the environment for Windows locally, I am repeatedly testing it through CI.
   
   I understand. Thanks. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-42030: [Java] Update Unit Tests for Adapter Module [arrow]

Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #42038:
URL: https://github.com/apache/arrow/pull/42038#issuecomment-2165366074

   @llama90 the CI failures are due to spotless issue. Could you please try `mvn spotless:apply' before building? 
   We will integrate `pre-commit` sooner, so that this won't be an issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org