You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by GitBox <gi...@apache.org> on 2021/08/17 05:27:48 UTC
[GitHub] [parquet-mr] YunLemon opened a new pull request #924: Improve Travis CI build Performance
YunLemon opened a new pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924
According to [Build times out because no output was received](https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received), we should carefully use travis_wait, as it may make the build unstable and extend the build time.
=====================
If there are any inappropriate modifications in this PR, please give me a reply and I will change them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900093413
I think I know the reason behind this.
Parquet uses `arch: arm64`, i.e. Equinix Metal machines and there was a recent update of their hardware - https://blog.travis-ci.com/2021-08-06-oss-equinix
https://github.com/apache/parquet-mr/blob/5154d04f1ccc21811c0668bf68cfd19f91100907/.travis.yml#L22
I'd suggest to watch the build times in the next few weeks and decide what new value of `travis_wait` to use, or to remove it completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] gszadovszky commented on pull request #924: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
gszadovszky commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900048186
@YunLemon, we usually don't open PRs without jiras. Could you please create a jira about this and add the jira id to the header of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-968646400
I think it is safe to merge this PR!
Recent builds have such logs:
1.
```
Still running (3 of 60): mvn install --batch-mode -DskipTests=true -Dmaven.javadoc.skip=true -Dsource.skip=true -Dorg.slf4j.simpleLogger.logFile=mvn-install.log
```
i.e. `mvn install` passes in 2-3 minutes. It is less than 10 mins, so it is OK to drop `travis_wait`!
2.
```
Still running (22 of 60): mvn verify --batch-mode javadoc:javadoc -P ci-test,default -Dorg.slf4j.simpleLogger.logFile=mvn-verify.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/maven/lib/guice.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thrift version 0.15.0
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Thrift version 0.15.0
message msg {
required group binary_col (LIST) {
repeated group list {
required binary element;
}
}
required group int32_col (LIST) {
repeated group list {
required int32 element;
}
}
}
```
i.e. `mvn verify` took 22 mins to run but it has many WARN logs which "keep alive" the console.
For example: https://app.travis-ci.com/github/apache/parquet-mr/jobs/546236418
If the build stucks for any reason then Travis will kill it after 10 mins of silence, instead of waiting for the `60` mins defined with `travis_wait`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900073269
Looking at the [Build History](https://app.travis-ci.com/github/apache/parquet-mr/builds) I see many builds which take 30, 40, 50 mins per job.
The shortest workflow (includes 3 jobs) is 50 mins, while the longest one is almost 2 hours.
The build time depends on how loaded Travis is.
I have added 'travis_wait` because builds were failing for the reasons explained above !
Looking at the build history I think you can try to reduce it to 30 mins.
But anything less would lead to forcefully stopped jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g edited a comment on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
martin-g edited a comment on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900057808
With `-Dorg.slf4j.simpleLogger.logFile=mvn-verify.log` all the output is redirected from stdout to a file.
TravisCI stops the job if there is no output for 10 minutes.
Since Parquet's build takes more than 10 mins you will have to make sure that the tests write something to stderr once in 10 minutes or the build will break.
What problem does this PR address ?
I see nothing concerning at https://app.travis-ci.com/github/apache/parquet-mr/builds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] YunLemon commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
YunLemon commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-968429043
yes, [https://app.travis-ci.com/github/apache/parquet-mr/jobs/541671084](https://app.travis-ci.com/github/apache/parquet-mr/jobs/541671084), I think this build can explain what i am worried about. 60 isn't a good number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] YunLemon commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
YunLemon commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900085802
Ok, I understand. Many jobs did take 30, 40, 50 mins.....But I can't see the logs of some old builds...From the recent builds, these two commands are quick(less than 800s?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] shangxinli merged pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
shangxinli merged pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] shangxinli commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
shangxinli commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-967272796
@YunLemon, Thanks for working on this! Are we still on track to reduce the minutes to wait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] martin-g commented on pull request #924: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900057808
With `-Dorg.slf4j.simpleLogger.logFile=mvn-verify.log` all the output is redirected from stdout to a file.
TravisCI stops the job if there is no output for 10 minutes.
Since Parquet's build takes more then 10 mins you will have to make sure that the tests write something to stderr once in 10 minutes or the build will break.
What problem does this PR address ?
I see nothing concerning at https://app.travis-ci.com/github/apache/parquet-mr/builds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] YunLemon commented on pull request #924: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
YunLemon commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900052260
> @YunLemon, we usually don't open PRs without jiras. Could you please create a jira about this and add the jira id to the header of this PR?
ok, I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] YunLemon commented on pull request #924: PARQUET-2076: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
YunLemon commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900068394
I don't think 60 minutes is a good value. Judging from the build of this PR, the running time of these two commands is less than 10 minutes(116.41s and 658.30s).
If the command gets stuck, then we will wait for 60 minutes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] gszadovszky commented on pull request #924: Improve Travis CI build Performance
Posted by GitBox <gi...@apache.org>.
gszadovszky commented on pull request #924:
URL: https://github.com/apache/parquet-mr/pull/924#issuecomment-900047470
@martin-g, you were the one who introduced `travis_wait` in #890. Could you please check this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org