You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nastra (via GitHub)" <gi...@apache.org> on 2023/03/21 16:43:56 UTC

[GitHub] [iceberg] nastra opened a new issue, #7160: Move JUnit4 tests to JUnit5

nastra opened a new issue, #7160:
URL: https://github.com/apache/iceberg/issues/7160

   ### Feature Request / Improvement
   
   In our contributing guidelines we point people to writing new unit tests using JUni5. 
   
   We should consider/evaluate what the best approach would be to eventually move existing tests from JUnit4 tests to JUnit5. 
   
   ### Query engine
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-2712999987

   @tomtongue yes I believe only Spark 3.4 is left, but it wasn't a priority to migrate it to JUnit5 (since it's a big task). However, feel free to work on this as that will definitely make it easier to backport code to Spark 3.4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1507534391

   Thank you Weijie for the detailed explanation, this is super helpful! Regarding Junit5 vs AssertJ `Assertions`, what did Flink community do? Do you think using AssertJ `Assertions` (which seems to be the consensus above) is preferred over Junit5?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] nastra commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1479289308

   It's worth looking at https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4. There are a few items to watch out for:
   
   For example in JUnit4 the method signature of `assertEquals()` was `assertEquals(String message, Object expected, Object actual)` but in JUnit5 it is `assertEquals(Object expected, Object actual, String message)`. There are also a few more methods affected by this. See https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4-failure-message-arguments for some details.
   In those cases I'm not sure what's better (just converting from JUnit4 to JUnit5 assertions or switching assertion checks to AssertJ where it makes sense). Both will cause larger diffs.
   
   For modules that don't use JUnit5 yet, those would require
   ```
   test {
       useJUnitPlatform()
     }
   ```
   
   Things like `@Rule public TemporaryFolder temp = new TemporaryFolder();` don't work in JUni5 and would have to be converted to `@TempDir Path temp;`.
   
   It's worth doing the conversion module-by-module and starting in a module that has a small amount of tests (like `iceberg-pig` or `iceberg-gcp`).
   
   @jackye1995 any thoughts on the outlined approach?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

Posted by "attilakreiner (via GitHub)" <gi...@apache.org>.
attilakreiner commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-2213539368

   Hey Team, I am new here, I was looking for a beginner task and I've seen you are in the process of migrating from Junit 4 to 5.
   
   I just gave it a try and migrated over the tests in the :iceberg-data module. (The `:iceberg-flink` module also got affected; and a little bit `:iceberg-spark` and `:iceberg-mr` too). I believe I have completely removed Junit 4 from the `:iceberg-data` module.
   
   Pls review this PR: https://github.com/apache/iceberg/pull/10657
   
   I'd highly appreciate any feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-2712820271

   @nastra I believe the remaining migration is only in Spark 3.4. If possible, can I proceed them to the JUnit 5 with AssertJ style? (and, I think this also should help the backport of V3 spec features to Spark 3.4)
   Spark 3.3 is still based in JUnit 4, but it's recently removed from the repository via https://github.com/apache/iceberg/pull/12279. So I think Spark 3.4 migration is enough. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1782099308

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-2713013874

   @nastra Thank you. Yes, it's still a big task and I understand the current prioritization. I will start it again gradually from adding the base tests like other versions for the further help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] mitrofmep commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "mitrofmep (via GitHub)" <gi...@apache.org>.
mitrofmep commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1478910257

   Hello! I'd like to help with this issue. Just checked [#209](https://github.com/apache/iceberg-docs/pull/209). Am I right, that it needs to check all the test classes, where only "import org.junit.*" used, without import jupiter api? For example, like in `core/src/test/java/org/apache/iceberg/avro/AvroDataTest.java`. And if possible, to refactor them using junit5?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1481383833

   I just noticed one thing, given the fact that Iceberg prefers to not import static methods directly, we will end up having 2 `Assertions` classes if we use both Junit5 and AssertJ. Given that we are already using AssertJ, we probably need to stick with that instead of mixing both.
   
   This means we want to migrate to Junit5 for framework and usages of things like tempdir, before, after, etc. but use AssertJ for all assertions. Should we have a thread on devlist just to confirm this as a community guideline?
   
   > there are scripts available to convert assertions
   
   That's super cool, maybe we should try it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1480330202

   Yes we definitely want to convert module by module. 
   
   I do not have strong opinion regarding using Junit `assertEquals` vs AssertJ, it seems like they will incur almost the same amount of changes:
   
   ```
   Assertions.assertThat("a").isEqualTo("b").describedAs("xxx");
   ```
   
   vs 
   
   ```
   Assertions.assertEquals("a", "b", "xxx")
   ```
   
   AssertJ might have the advantage that it can format the description with `.describedAs("xxx %s %s", x, y)` so it's a bit easier to use. And it might simplify some test cases. I don't quite get why we want to lazily load a description message with the feature that Junit5 provides. So maybe using AssertJ could be preferred.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] nastra commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1480718595

   I think in a lot of cases we don't necessarily need `.describedAs(..)` / `.as(..)` as the context that is being provided when a check fails is usually enough. Taking the below example:
   ```
   Assert.assertTrue("OSS file should exist", ossClient().get().doesObjectExist(uri.bucket(), uri.key()));
   Assert.assertEquals("Should have expected location", location, out.location());
   Assert.assertEquals("Should have expected length", dataSize, ossDataLength(uri));
   Assert.assertArrayEquals("Should have expected content", data, ossDataContent(uri, dataSize));
   ```
   
   which could be rewritten to:
   ```
   Assertions.assertThat(ossClient().get().doesObjectExist(uri.bucket(), uri.key())).as("OSS file should exist").isTrue();
   Assertions.assertThat(out.location()).isEqualTo(location);
   Assertions.assertThat(ossDataLength(uri)).isEqualTo(dataSize);
   Assertions.assertThat(ossDataContent(uri, dataSize)).isEqualTo(data);
   ```
   
   That being said, we should just double-check when transitioning/reviewing where it makes sense to have a failure description / overriding error msg.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] reswqa commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1507850136

   Thanks @jackye1995 for the quick reply!
   
   > Regarding Junit5 vs AssertJ Assertions, what did Flink community do?
   Earlier, many different types of assertions were mixed in Flink's codebase, such as `Junit4` assertion, `Hamcrest`,`some custom assertion`, etc. 
   
   While migrating to `Junit5`, Flink community decided that use `AssertJ` as the only assertion framework.
   
   > Do you think using AssertJ Assertions (which seems to be the consensus above) is preferred over Junit5?
   In my personal experience, `AssertJ` is really powerful and the benefit speaks for itself.
   
   - Thanks to its chained assertion style, we can avoid writing a lot of redundant code.
   - It has truly helpful error messages.
   - It provides a dedicated assertion API for many types, which will greatly reduce the burden of writing tests and increase the readability of the code.
   - Provides assertions for Guava types like Multimap, Table, Optional or ByteSource.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] reswqa commented on issue #7160: Move JUnit4 tests to JUnit5

Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on issue #7160:
URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1507348739

   Hi guys,
   
   I am very happy to see this discussion on migrating test framework to `Junit5`, which will bring great benefits to the testing part of the project. For example, stronger support for parameterized test and better support for lambda expressions.
   
   Since the `Apache Flink` community has already started migrating tests to `Junit5` and `Assertj` a long time ago, maybe we can learn something from it. In short, It adopted a `module-by-module` migration solution, and you can refer to this umbrella [ticket](https://issues.apache.org/jira/browse/FLINK-25325)). 
   
   
   > It's worth looking at https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4.
   
   It is really a good reference to start. At the same time, I would like to share the [migration guidance]([JUnit5 migration guide](https://docs.google.com/document/d/1514Wa_aNB9bJUen4xm5uiuXOooOJTtXqS_Jqk9KJitU/edit)) of the flink community for your reference.
   
   
   > It's also worth mentioning that there are scripts available to convert assertions: https://assertj.github.io/doc/#assertj-migration
   This script does most of the boring migration stuff. However, according to my experience, there are still many situations that it cannot handle, such as converting `try{} catch(expected exception)` to `Assertions.assertThatThrowBy`. However, there are not many such exceptions. We can run this script first and spend a little more time to manually handle some special cases.
   
   > Things like @Rule public TemporaryFolder temp = new TemporaryFolder(); don't work in JUni5 and would have to be converted to @TempDir Path temp;.
   
   Yes, something like this does the more tedious part of the migration process. 
   `Juint5` has much better support for parameterized tests to avoid separating parameterized and non-parameterized parts into different test classes. But this also brings some troubles to our migration, because many of our previous tests were parameterized for the whole class, but now we have to inject parameters for each method.
   In order to avoid the inconvenience caused by the above problems as much as possible, we have done some work in flink:
   - Introduce `ParameterizedTestExtension` and some annotations to make it easier to migrate test class that require parameterization for all methods.
   - Introduce `TempDirUtils` to provide some common methods in `org.junit.rules.TemporaryFolder` but for `Junit5`.
   For more details, please refer to this [ticket](https://issues.apache.org/jira/browse/FLINK-25315).
   
   Of course, I am also very willing to participate in the migration of iceberg's test framework, and I look forward to taking this opportunity to integrate into this great community.
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org