You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/04/14 08:06:45 UTC
[GitHub] [iceberg] snazy opened a new issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
snazy opened a new issue #2475:
URL: https://github.com/apache/iceberg/issues/2475
From the CI run below, it might be that the JDK8 test runs has issues when the tasks `:iceberg-hive3:test` and `:iceberg-mr:test` run in parallel. Only JDK8 Gradle runs do include `:iceberg-hive3`, which explains why only the JDK8 CI test runs are affected (seen only the JDK8 test runs timing out).
I don't know about the inner workings of both project tests, but I suspect those two projects somehow interfere with each other.
If it's just those two test tasks, it might help to add a `mustRunAfter(':iceberg-mr:test')` to `:iceberg-hive3:test`.
[Example failuter](https://github.com/apache/iceberg/pull/2450/checks?check_run_id=2335040662)
From the [raw log](https://pipelines.actions.githubusercontent.com/aep6esCMku3CvU43ZNtyjk8wsCZTONMFPo274492W2mxr3EUmN/_apis/pipelines/1/runs/8135/signedlogcontent/5?urlExpires=2021-04-14T07%3A11%3A28.2210078Z&urlSigningMethod=HMACV1&urlSignature=fsG4gTD4g8cfObkeHFeoyh6r9re9eWy3RnbavenSdyM%3D):
```
2021-04-13T15:46:50.1563347Z Starting a Gradle Daemon (subsequent builds will be faster)
...
2021-04-13T15:51:44.2558196Z > Task :iceberg-hive3:test
...
2021-04-13T15:58:24.8564436Z > Task :iceberg-mr:test
...
2021-04-13T16:58:31.6625242Z > Task :iceberg-hive3:test
(reports test failure)
2021-04-13T20:50:20.3558417Z
(:iceberg-hive3:test reports next test failure)
...
2021-04-13T20:50:20.8582685Z > Task :iceberg-mr:test
(reports test failure)
2021-04-13T21:07:21.6565277Z > Task :iceberg-mr:test FAILED
...
2021-04-13T21:11:26.6561908Z > Task :iceberg-hive3:test FAILED
...
2021-04-13T21:11:26.6581909Z BUILD FAILED in 5h 24m 39s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819628854
Some update on this one:
* #2448 didn't help :(
* Adding `mustRunAfter()` to `build.gradle` for the two suspect `test` tasks mentioned in the description didn't help :(
* Adding `mustRunAfter()` to `build.gradle` for every `test` task (so that at most one `test` task runs at a time) didn't help :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819739173
Aha! So it seems that [this commit](https://github.com/apache/iceberg/pull/2450/commits/2cda7cdc7c4579963becbe6b7c2bb74e5055fc2d) fixed it. I'll setup a PR with just that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819327841
> @snazy: Could it be that it is the same as #2448
I doubt. 2448 "just" adds timeouts to tests, so they can "fail earlier" instead of running infinitely. But I think, the underlying problem's the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy edited a comment on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy edited a comment on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819646004
I'm currently trying to reduce that maxHeapSize setting [here](https://github.com/apache/iceberg/blob/master/build.gradle#L950-L951). Maybe it's hitting the instance limits (2 vCPUs, 7GB vRAM, 14G disk) - I'm just poking around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819642997
The last run I've seen it got stuck in `:iceberg-hive3:test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819639170
Do you have links to runs with #2448?
I would love to see the `test logs` for the tests, and #2448 aims to timeout the test themselves before the github timeout comes. So github can collect the logs for the run.
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819325378
@snazy: Could it be that it is the same as #2448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819647800
Maybe we should add timeout for all of the `TestHiveIcebergStorageHandler*` tests. The failure on the commit was about the failing `TestHiveIcebergStorageHandlerWithMultipleCatalogs`. For the others I do not see which test is stuck.
Also I have noticed that the annotation based timeout does not override the rule base timeout. So this will still mean that the timeout will happen after 40 sec.
```
@Rule
public Timeout timeout = new Timeout(40000, TimeUnit.MILLISECONDS);
[..]
@Test(timeout = 100000)
public void testSelectDistinctFromTable() throws IOException {
[..]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer closed issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
RussellSpitzer closed issue #2475:
URL: https://github.com/apache/iceberg/issues/2475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819642534
So we have to add timeout to the `TestHiveIcebergStorageHandlerWithMultipleCatalogs` as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy edited a comment on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy edited a comment on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819642997
The last run I've seen it got stuck in `:iceberg-hive3:test` (see [here](https://github.com/apache/iceberg/pull/2450/checks?check_run_id=2343384901) - that was the run with at most one `test` task at a time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy edited a comment on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy edited a comment on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819642997
The last run I've seen it got stuck in `:iceberg-hive3:test` (see [here](https://github.com/apache/iceberg/pull/2450/checks?check_run_id=2343384901))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819646004
I'm currently trying to reduce that maxHeapSize setting [here](https://github.com/apache/iceberg/blob/master/build.gradle#L950-L951). Maybe it's hitting the instance limits (2 vCPUs, 6GB vRAM) - I'm just poking around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819779537
Thanks to @lhotari for pointing out the GH-runner issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819358405
> > @snazy: Could it be that it is the same as #2448
>
> I doubt. 2448 "just" adds timeouts to tests, so they can "fail earlier" instead of running infinitely. But I think, the underlying problem's the same.
There is a minimal [change](https://github.com/apache/iceberg/pull/2448/files#diff-2da837fbcd9c1789e9c32826e7638ff55c1d8ffb5779017759f51561f35d807cR82) hidden there which might solve the issue:
```
- metastore.start(new HiveConf(hs2Conf));
+ metastore.start(new HiveConf(hs2Conf), 10);
```
The timeout helped to identify why are the tests are waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819682315
[Another run](https://github.com/apache/iceberg/pull/2450/checks?check_run_id=2345407451) with the GitHub-Ubuntu-runner workaround mentioned in #2451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy edited a comment on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy edited a comment on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819640200
> Do you have links to runs with #2448?
It's on the [commit](https://github.com/apache/iceberg/commit/c3640afadc5af759bee531368c8be16e2a2b7f5d) in master, don't have anything else. It's maybe not just these tests but something else (see [here](https://github.com/apache/iceberg/runs/2341885191) and in [here as well](https://github.com/apache/iceberg/pull/2450/checks?check_run_id=2344793278), when that run's finished).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] snazy commented on issue #2475: CI `run-tests (8)` sometimes times out after 6 hours
Posted by GitBox <gi...@apache.org>.
snazy commented on issue #2475:
URL: https://github.com/apache/iceberg/issues/2475#issuecomment-819640200
> Do you have links to runs with #2448?
It's on the [commit](https://github.com/apache/iceberg/commit/c3640afadc5af759bee531368c8be16e2a2b7f5d) in master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org