You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steve Loughran <st...@cloudera.com.INVALID> on 2020/05/12 13:50:37 UTC
why is a hadoop dist taking so long to build
every week or so I do a -Pdist build of hadoop, so I can have an up to date
command line. Today I kicked off the build with -Pdist -DskipTests
-Dmaven.javadoc.skip=true
And it took half an hour to complete -with 1/3 of that time just shading
the minicluster
[INFO] Apache Hadoop Main ................................. SUCCESS [
1.065 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [
3.606 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [
1.374 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [
1.209 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [
0.221 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
1.096 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
4.158 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
0.710 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [
2.969 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
1.591 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [
49.545 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [
1.647 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [
3.279 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [
4.420 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [
0.118 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
32.316 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:19
min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
1.336 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [
4.519 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
2.166 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
13.665 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
0.103 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [
0.097 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [
23.601 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [
19.536 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [
0.080 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
11.128 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
16.997 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
1.623 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
4.663 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
2.247 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
26.548 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
2.960 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [
4.913 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
1.732 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
1.867 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
0.123 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
3.153 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
3.503 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
0.088 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
4.038 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
3.494 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [
2.660 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
1.713 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [
0.091 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
2.062 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
1.544 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
0.601 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
6.887 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
6.869 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
3.005 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
5.289 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
4.543 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
5.421 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
1.516 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [
0.077 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
7.382 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [
2.310 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
0.086 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
24.634 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
0.183 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
0.128 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
1.844 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [
0.109 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
1.188 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [
0.123 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
9.718 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [
18.364 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
1.582 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
2.293 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
1.526 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
2.003 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [
4.660 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
1.782 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
2.608 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
2.700 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
2.079 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
2.756 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
1.692 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
4.494 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
0.085 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [
1.052 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
1.228 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [
2.137 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [
1.913 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [
1.181 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [
1.364 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [
0.079 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [
1.598 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
6.910 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
1.256 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [
4.236 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
1.051 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
2.416 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
2.483 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
1.284 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
1.575 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
5.312 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [
0.074 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [03:17
min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:59
min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
1.076 s]
*[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [10:32
min]*
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
2.611 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
0.533 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [02:55
min]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [
0.168 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
2.697 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
2.792 s]
it used to be that -DskipShade would save the shading, but last time I
tried to skip the shading something failed.
1. has anyone else tried a full dist build recently -and how long did it
take?
2. could they skip the shading?
Re: why is a hadoop dist taking so long to build
Posted by Akira Ajisaka <aa...@apache.org>.
I think YARN-9606 broke the build. Reverted.
Thank you, Ayush and Steve.
-Akira
On Fri, May 15, 2020 at 5:46 AM Ayush Saxena <ay...@gmail.com> wrote:
> Could be due to YARN-9606, got committed today itself.
> https://issues.apache.org/jira/browse/YARN-9606
> The last build in the JIRA had -1 on shadedclient.
>
> https://issues.apache.org/jira/browse/YARN-9606?focusedCommentId=17078443&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17078443
>
> tried locally was working without for me, couldn't check further to be
> completely sure. can give a check...
>
Re: why is a hadoop dist taking so long to build
Posted by Ayush Saxena <ay...@gmail.com>.
Could be due to YARN-9606, got committed today itself.
https://issues.apache.org/jira/browse/YARN-9606
The last build in the JIRA had -1 on shadedclient.
https://issues.apache.org/jira/browse/YARN-9606?focusedCommentId=17078443&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17078443
tried locally was working without for me, couldn't check further to be
completely sure. can give a check...
Re: why is a hadoop dist taking so long to build
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
ok, if it works for you, I Will play further with it.
yetus seems to be unhappy too
https://builds.apache.org/job/hadoop-multibranch/job/PR-2022/1/artifact/out/branch-mvninstall-root.txt
On Tue, 12 May 2020 at 15:51, Ayush Saxena <ay...@gmail.com> wrote:
> Sorry, didn't see the first question.
> Attached the Build_Details for reference.
> With -DskipShade 5:18
> Without -DskipShade : 12:52
>
> -Ayush
>
> On Tue, 12 May 2020 at 19:51, Ayush Saxena <ay...@gmail.com> wrote:
>
>> Hi Steve,
>> Just tried building with -DskipShade, It worked for me.
>> Executed *mvn clean install -DskipTests -DskipShade -Pdist
>> -Dmaven.javadoc.skip=true*
>>
>> Sharing the result :
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary for Apache Hadoop Main 3.4.0-SNAPSHOT:
>> [INFO]
>> [INFO] Apache Hadoop Main ................................. SUCCESS [
>> 1.587 s]
>> [INFO] Apache Hadoop Build Tools .......................... SUCCESS [
>> 19.332 s]
>> [INFO] Apache Hadoop Project POM .......................... SUCCESS [
>> 3.538 s]
>> [INFO] Apache Hadoop Annotations .......................... SUCCESS [
>> 2.756 s]
>> [INFO] Apache Hadoop Assemblies ........................... SUCCESS [
>> 0.348 s]
>> [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
>> 2.319 s]
>> [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
>> 5.573 s]
>> [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
>> 2.083 s]
>> [INFO] Apache Hadoop Auth ................................. SUCCESS [
>> 9.620 s]
>> [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
>> 4.134 s]
>> [INFO] Apache Hadoop Common ............................... SUCCESS [
>> 51.670 s]
>> [INFO] Apache Hadoop NFS .................................. SUCCESS [
>> 3.019 s]
>> [INFO] Apache Hadoop KMS .................................. SUCCESS [
>> 3.651 s]
>> [INFO] Apache Hadoop Registry ............................. SUCCESS [
>> 4.862 s]
>> [INFO] Apache Hadoop Common Project ....................... SUCCESS [
>> 0.109 s]
>> [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
>> 24.093 s]
>> [INFO] Apache Hadoop HDFS ................................. SUCCESS [
>> 46.701 s]
>> [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
>> 2.147 s]
>> [INFO] Apache Hadoop HttpFS ............................... SUCCESS [
>> 4.048 s]
>> [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
>> 3.467 s]
>> [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
>> 6.077 s]
>> [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
>> 0.340 s]
>> [INFO] Apache Hadoop YARN ................................. SUCCESS [
>> 0.349 s]
>> [INFO] Apache Hadoop YARN API ............................. SUCCESS [
>> 16.439 s]
>> [INFO] Apache Hadoop YARN Common .......................... SUCCESS [
>> 20.588 s]
>> [INFO] Apache Hadoop YARN Server .......................... SUCCESS [
>> 0.082 s]
>> [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
>> 6.682 s]
>> [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
>> 13.558 s]
>> [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
>> 1.800 s]
>> [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
>> 4.442 s]
>> [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
>> 2.093 s]
>> [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
>> 19.693 s]
>> [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
>> 3.153 s]
>> [INFO] Apache Hadoop YARN Client .......................... SUCCESS [
>> 3.888 s]
>> [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
>> 2.198 s]
>> [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
>> 1.360 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
>> 0.078 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
>> 8.619 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
>> 6.664 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
>> 0.060 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
>> 2.963 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
>> 3.729 s]
>> [INFO] Apache Hadoop YARN Router .......................... SUCCESS [
>> 2.989 s]
>> [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
>> 2.976 s]
>> [INFO] Apache Hadoop YARN Applications .................... SUCCESS [
>> 0.335 s]
>> [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
>> 2.720 s]
>> [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
>> 1.341 s]
>> [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
>> 0.223 s]
>> [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
>> 8.664 s]
>> [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
>> 5.485 s]
>> [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
>> 3.390 s]
>> [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
>> 3.636 s]
>> [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
>> 3.584 s]
>> [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
>> 5.087 s]
>> [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
>> 2.747 s]
>> [INFO] Apache Hadoop YARN Services ........................ SUCCESS [
>> 0.393 s]
>> [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
>> 4.652 s]
>> [INFO] Apache Hadoop YARN Services API .................... SUCCESS [
>> 2.859 s]
>> [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
>> 0.145 s]
>> [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
>> 39.601 s]
>> [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
>> 0.371 s]
>> [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
>> 0.151 s]
>> [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
>> 1.250 s]
>> [INFO] Apache Hadoop YARN Site ............................ SUCCESS [
>> 0.131 s]
>> [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
>> 1.733 s]
>> [INFO] Apache Hadoop YARN UI .............................. SUCCESS [
>> 0.070 s]
>> [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
>> 7.734 s]
>> [INFO] Apache Hadoop YARN Project ......................... SUCCESS [
>> 7.258 s]
>> [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
>> 0.684 s]
>> [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
>> 3.412 s]
>> [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
>> 2.193 s]
>> [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
>> 3.257 s]
>> [INFO] Apache Hadoop MapReduce ............................ SUCCESS [
>> 6.359 s]
>> [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
>> 1.383 s]
>> [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
>> 2.370 s]
>> [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
>> 1.699 s]
>> [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
>> 1.390 s]
>> [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
>> 3.769 s]
>> [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
>> 1.027 s]
>> [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
>> 4.113 s]
>> [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
>> 0.539 s]
>> [INFO] Apache Hadoop Archives ............................. SUCCESS [
>> 0.831 s]
>> [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
>> 0.635 s]
>> [INFO] Apache Hadoop Rumen ................................ SUCCESS [
>> 3.435 s]
>> [INFO] Apache Hadoop Gridmix .............................. SUCCESS [
>> 1.090 s]
>> [INFO] Apache Hadoop Data Join ............................ SUCCESS [
>> 0.530 s]
>> [INFO] Apache Hadoop Extras ............................... SUCCESS [
>> 0.488 s]
>> [INFO] Apache Hadoop Pipes ................................ SUCCESS [
>> 0.075 s]
>> [INFO] Apache Hadoop OpenStack support .................... SUCCESS [
>> 1.449 s]
>> [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
>> 16.265 s]
>> [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
>> 1.903 s]
>> [INFO] Apache Hadoop Azure support ........................ SUCCESS [
>> 4.522 s]
>> [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
>> 2.036 s]
>> [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
>> 3.199 s]
>> [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
>> 2.126 s]
>> [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
>> 0.710 s]
>> [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
>> 0.681 s]
>> [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
>> 4.655 s]
>> [INFO] Apache Hadoop Tools ................................ SUCCESS [
>> 0.068 s]
>> [INFO] Apache Hadoop Client API ........................... SUCCESS [
>> 2.329 s]
>> [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [
>> 0.646 s]
>> [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
>> 1.650 s]
>> [INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [
>> 2.925 s]
>> [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
>> 0.370 s]
>> [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
>> 0.688 s]
>> [INFO] Apache Hadoop Distribution ......................... SUCCESS [
>> 41.321 s]
>> [INFO] Apache Hadoop Client Modules ....................... SUCCESS [
>> 0.663 s]
>> [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
>> 1.865 s]
>> [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
>> 0.810 s]
>> [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [
>> 0.076 s]
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 09:23 min
>> [INFO] Finished at: 2020-05-12T19:46:31+05:30
>> [INFO]
>> ------------------------------------------------------------------------
>> ayush@ayushpc:~/trunk$ mvn clean install -DskipTests -DskipShade -Pdist
>> -Dmaven.javadoc.skip=true
>>
>>
>> -Ayush
>>
>>
>> On Tue, 12 May 2020 at 19:21, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>>
>>> every week or so I do a -Pdist build of hadoop, so I can have an up to
>>> date
>>> command line. Today I kicked off the build with -Pdist -DskipTests
>>> -Dmaven.javadoc.skip=true
>>>
>>> And it took half an hour to complete -with 1/3 of that time just shading
>>> the minicluster
>>>
>>> [INFO] Apache Hadoop Main ................................. SUCCESS [
>>> 1.065 s]
>>> [INFO] Apache Hadoop Build Tools .......................... SUCCESS [
>>> 3.606 s]
>>> [INFO] Apache Hadoop Project POM .......................... SUCCESS [
>>> 1.374 s]
>>> [INFO] Apache Hadoop Annotations .......................... SUCCESS [
>>> 1.209 s]
>>> [INFO] Apache Hadoop Assemblies ........................... SUCCESS [
>>> 0.221 s]
>>> [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
>>> 1.096 s]
>>> [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
>>> 4.158 s]
>>> [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
>>> 0.710 s]
>>> [INFO] Apache Hadoop Auth ................................. SUCCESS [
>>> 2.969 s]
>>> [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
>>> 1.591 s]
>>> [INFO] Apache Hadoop Common ............................... SUCCESS [
>>> 49.545 s]
>>> [INFO] Apache Hadoop NFS .................................. SUCCESS [
>>> 1.647 s]
>>> [INFO] Apache Hadoop KMS .................................. SUCCESS [
>>> 3.279 s]
>>> [INFO] Apache Hadoop Registry ............................. SUCCESS [
>>> 4.420 s]
>>> [INFO] Apache Hadoop Common Project ....................... SUCCESS [
>>> 0.118 s]
>>> [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
>>> 32.316 s]
>>> [INFO] Apache Hadoop HDFS ................................. SUCCESS
>>> [01:19
>>> min]
>>> [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
>>> 1.336 s]
>>> [INFO] Apache Hadoop HttpFS ............................... SUCCESS [
>>> 4.519 s]
>>> [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
>>> 2.166 s]
>>> [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
>>> 13.665 s]
>>> [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
>>> 0.103 s]
>>> [INFO] Apache Hadoop YARN ................................. SUCCESS [
>>> 0.097 s]
>>> [INFO] Apache Hadoop YARN API ............................. SUCCESS [
>>> 23.601 s]
>>> [INFO] Apache Hadoop YARN Common .......................... SUCCESS [
>>> 19.536 s]
>>> [INFO] Apache Hadoop YARN Server .......................... SUCCESS [
>>> 0.080 s]
>>> [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
>>> 11.128 s]
>>> [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
>>> 16.997 s]
>>> [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
>>> 1.623 s]
>>> [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
>>> 4.663 s]
>>> [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
>>> 2.247 s]
>>> [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
>>> 26.548 s]
>>> [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
>>> 2.960 s]
>>> [INFO] Apache Hadoop YARN Client .......................... SUCCESS [
>>> 4.913 s]
>>> [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
>>> 1.732 s]
>>> [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
>>> 1.867 s]
>>> [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
>>> 0.123 s]
>>> [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
>>> 3.153 s]
>>> [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
>>> 3.503 s]
>>> [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
>>> 0.088 s]
>>> [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
>>> 4.038 s]
>>> [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
>>> 3.494 s]
>>> [INFO] Apache Hadoop YARN Router .......................... SUCCESS [
>>> 2.660 s]
>>> [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
>>> 1.713 s]
>>> [INFO] Apache Hadoop YARN Applications .................... SUCCESS [
>>> 0.091 s]
>>> [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
>>> 2.062 s]
>>> [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
>>> 1.544 s]
>>> [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
>>> 0.601 s]
>>> [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
>>> 6.887 s]
>>> [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
>>> 6.869 s]
>>> [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
>>> 3.005 s]
>>> [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
>>> 5.289 s]
>>> [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
>>> 4.543 s]
>>> [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
>>> 5.421 s]
>>> [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
>>> 1.516 s]
>>> [INFO] Apache Hadoop YARN Services ........................ SUCCESS [
>>> 0.077 s]
>>> [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
>>> 7.382 s]
>>> [INFO] Apache Hadoop YARN Services API .................... SUCCESS [
>>> 2.310 s]
>>> [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
>>> 0.086 s]
>>> [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
>>> 24.634 s]
>>> [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
>>> 0.183 s]
>>> [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
>>> 0.128 s]
>>> [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
>>> 1.844 s]
>>> [INFO] Apache Hadoop YARN Site ............................ SUCCESS [
>>> 0.109 s]
>>> [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
>>> 1.188 s]
>>> [INFO] Apache Hadoop YARN UI .............................. SUCCESS [
>>> 0.123 s]
>>> [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
>>> 9.718 s]
>>> [INFO] Apache Hadoop YARN Project ......................... SUCCESS [
>>> 18.364 s]
>>> [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
>>> 1.582 s]
>>> [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
>>> 2.293 s]
>>> [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
>>> 1.526 s]
>>> [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
>>> 2.003 s]
>>> [INFO] Apache Hadoop MapReduce ............................ SUCCESS [
>>> 4.660 s]
>>> [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
>>> 1.782 s]
>>> [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
>>> 2.608 s]
>>> [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
>>> 2.700 s]
>>> [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
>>> 2.079 s]
>>> [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
>>> 2.756 s]
>>> [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
>>> 1.692 s]
>>> [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
>>> 4.494 s]
>>> [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
>>> 0.085 s]
>>> [INFO] Apache Hadoop Archives ............................. SUCCESS [
>>> 1.052 s]
>>> [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
>>> 1.228 s]
>>> [INFO] Apache Hadoop Rumen ................................ SUCCESS [
>>> 2.137 s]
>>> [INFO] Apache Hadoop Gridmix .............................. SUCCESS [
>>> 1.913 s]
>>> [INFO] Apache Hadoop Data Join ............................ SUCCESS [
>>> 1.181 s]
>>> [INFO] Apache Hadoop Extras ............................... SUCCESS [
>>> 1.364 s]
>>> [INFO] Apache Hadoop Pipes ................................ SUCCESS [
>>> 0.079 s]
>>> [INFO] Apache Hadoop OpenStack support .................... SUCCESS [
>>> 1.598 s]
>>> [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
>>> 6.910 s]
>>> [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
>>> 1.256 s]
>>> [INFO] Apache Hadoop Azure support ........................ SUCCESS [
>>> 4.236 s]
>>> [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
>>> 1.051 s]
>>> [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
>>> 2.416 s]
>>> [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
>>> 2.483 s]
>>> [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
>>> 1.284 s]
>>> [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
>>> 1.575 s]
>>> [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
>>> 5.312 s]
>>> [INFO] Apache Hadoop Tools ................................ SUCCESS [
>>> 0.074 s]
>>> [INFO] Apache Hadoop Client API ........................... SUCCESS
>>> [03:17
>>> min]
>>> [INFO] Apache Hadoop Client Runtime ....................... SUCCESS
>>> [02:59
>>> min]
>>> [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
>>> 1.076 s]
>>> *[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS
>>> [10:32
>>> min]*
>>> [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
>>> 2.611 s]
>>> [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
>>> 0.533 s]
>>> [INFO] Apache Hadoop Distribution ......................... SUCCESS
>>> [02:55
>>> min]
>>> [INFO] Apache Hadoop Client Modules ....................... SUCCESS [
>>> 0.168 s]
>>> [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
>>> 2.697 s]
>>> [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
>>> 2.792 s]
>>>
>>> it used to be that -DskipShade would save the shading, but last time I
>>> tried to skip the shading something failed.
>>>
>>> 1. has anyone else tried a full dist build recently -and how long did it
>>> take?
>>> 2. could they skip the shading?
>>>
>>
Re: why is a hadoop dist taking so long to build
Posted by Ayush Saxena <ay...@gmail.com>.
Sorry, didn't see the first question.
Attached the Build_Details for reference.
With -DskipShade 5:18
Without -DskipShade : 12:52
-Ayush
On Tue, 12 May 2020 at 19:51, Ayush Saxena <ay...@gmail.com> wrote:
> Hi Steve,
> Just tried building with -DskipShade, It worked for me.
> Executed *mvn clean install -DskipTests -DskipShade -Pdist
> -Dmaven.javadoc.skip=true*
>
> Sharing the result :
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary for Apache Hadoop Main 3.4.0-SNAPSHOT:
> [INFO]
> [INFO] Apache Hadoop Main ................................. SUCCESS [
> 1.587 s]
> [INFO] Apache Hadoop Build Tools .......................... SUCCESS [
> 19.332 s]
> [INFO] Apache Hadoop Project POM .......................... SUCCESS [
> 3.538 s]
> [INFO] Apache Hadoop Annotations .......................... SUCCESS [
> 2.756 s]
> [INFO] Apache Hadoop Assemblies ........................... SUCCESS [
> 0.348 s]
> [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
> 2.319 s]
> [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
> 5.573 s]
> [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
> 2.083 s]
> [INFO] Apache Hadoop Auth ................................. SUCCESS [
> 9.620 s]
> [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
> 4.134 s]
> [INFO] Apache Hadoop Common ............................... SUCCESS [
> 51.670 s]
> [INFO] Apache Hadoop NFS .................................. SUCCESS [
> 3.019 s]
> [INFO] Apache Hadoop KMS .................................. SUCCESS [
> 3.651 s]
> [INFO] Apache Hadoop Registry ............................. SUCCESS [
> 4.862 s]
> [INFO] Apache Hadoop Common Project ....................... SUCCESS [
> 0.109 s]
> [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
> 24.093 s]
> [INFO] Apache Hadoop HDFS ................................. SUCCESS [
> 46.701 s]
> [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
> 2.147 s]
> [INFO] Apache Hadoop HttpFS ............................... SUCCESS [
> 4.048 s]
> [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
> 3.467 s]
> [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
> 6.077 s]
> [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
> 0.340 s]
> [INFO] Apache Hadoop YARN ................................. SUCCESS [
> 0.349 s]
> [INFO] Apache Hadoop YARN API ............................. SUCCESS [
> 16.439 s]
> [INFO] Apache Hadoop YARN Common .......................... SUCCESS [
> 20.588 s]
> [INFO] Apache Hadoop YARN Server .......................... SUCCESS [
> 0.082 s]
> [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
> 6.682 s]
> [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
> 13.558 s]
> [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
> 1.800 s]
> [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
> 4.442 s]
> [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
> 2.093 s]
> [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
> 19.693 s]
> [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
> 3.153 s]
> [INFO] Apache Hadoop YARN Client .......................... SUCCESS [
> 3.888 s]
> [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
> 2.198 s]
> [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
> 1.360 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
> 0.078 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
> 8.619 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
> 6.664 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
> 0.060 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
> 2.963 s]
> [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
> 3.729 s]
> [INFO] Apache Hadoop YARN Router .......................... SUCCESS [
> 2.989 s]
> [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
> 2.976 s]
> [INFO] Apache Hadoop YARN Applications .................... SUCCESS [
> 0.335 s]
> [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
> 2.720 s]
> [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
> 1.341 s]
> [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
> 0.223 s]
> [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
> 8.664 s]
> [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
> 5.485 s]
> [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
> 3.390 s]
> [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
> 3.636 s]
> [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
> 3.584 s]
> [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
> 5.087 s]
> [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
> 2.747 s]
> [INFO] Apache Hadoop YARN Services ........................ SUCCESS [
> 0.393 s]
> [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
> 4.652 s]
> [INFO] Apache Hadoop YARN Services API .................... SUCCESS [
> 2.859 s]
> [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
> 0.145 s]
> [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
> 39.601 s]
> [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
> 0.371 s]
> [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
> 0.151 s]
> [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
> 1.250 s]
> [INFO] Apache Hadoop YARN Site ............................ SUCCESS [
> 0.131 s]
> [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
> 1.733 s]
> [INFO] Apache Hadoop YARN UI .............................. SUCCESS [
> 0.070 s]
> [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
> 7.734 s]
> [INFO] Apache Hadoop YARN Project ......................... SUCCESS [
> 7.258 s]
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
> 0.684 s]
> [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
> 3.412 s]
> [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
> 2.193 s]
> [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
> 3.257 s]
> [INFO] Apache Hadoop MapReduce ............................ SUCCESS [
> 6.359 s]
> [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
> 1.383 s]
> [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
> 2.370 s]
> [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
> 1.699 s]
> [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
> 1.390 s]
> [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
> 3.769 s]
> [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
> 1.027 s]
> [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
> 4.113 s]
> [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
> 0.539 s]
> [INFO] Apache Hadoop Archives ............................. SUCCESS [
> 0.831 s]
> [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
> 0.635 s]
> [INFO] Apache Hadoop Rumen ................................ SUCCESS [
> 3.435 s]
> [INFO] Apache Hadoop Gridmix .............................. SUCCESS [
> 1.090 s]
> [INFO] Apache Hadoop Data Join ............................ SUCCESS [
> 0.530 s]
> [INFO] Apache Hadoop Extras ............................... SUCCESS [
> 0.488 s]
> [INFO] Apache Hadoop Pipes ................................ SUCCESS [
> 0.075 s]
> [INFO] Apache Hadoop OpenStack support .................... SUCCESS [
> 1.449 s]
> [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
> 16.265 s]
> [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
> 1.903 s]
> [INFO] Apache Hadoop Azure support ........................ SUCCESS [
> 4.522 s]
> [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
> 2.036 s]
> [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
> 3.199 s]
> [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
> 2.126 s]
> [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
> 0.710 s]
> [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
> 0.681 s]
> [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
> 4.655 s]
> [INFO] Apache Hadoop Tools ................................ SUCCESS [
> 0.068 s]
> [INFO] Apache Hadoop Client API ........................... SUCCESS [
> 2.329 s]
> [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [
> 0.646 s]
> [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
> 1.650 s]
> [INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [
> 2.925 s]
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
> 0.370 s]
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
> 0.688 s]
> [INFO] Apache Hadoop Distribution ......................... SUCCESS [
> 41.321 s]
> [INFO] Apache Hadoop Client Modules ....................... SUCCESS [
> 0.663 s]
> [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
> 1.865 s]
> [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
> 0.810 s]
> [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [
> 0.076 s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 09:23 min
> [INFO] Finished at: 2020-05-12T19:46:31+05:30
> [INFO]
> ------------------------------------------------------------------------
> ayush@ayushpc:~/trunk$ mvn clean install -DskipTests -DskipShade -Pdist
> -Dmaven.javadoc.skip=true
>
>
> -Ayush
>
>
> On Tue, 12 May 2020 at 19:21, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> every week or so I do a -Pdist build of hadoop, so I can have an up to
>> date
>> command line. Today I kicked off the build with -Pdist -DskipTests
>> -Dmaven.javadoc.skip=true
>>
>> And it took half an hour to complete -with 1/3 of that time just shading
>> the minicluster
>>
>> [INFO] Apache Hadoop Main ................................. SUCCESS [
>> 1.065 s]
>> [INFO] Apache Hadoop Build Tools .......................... SUCCESS [
>> 3.606 s]
>> [INFO] Apache Hadoop Project POM .......................... SUCCESS [
>> 1.374 s]
>> [INFO] Apache Hadoop Annotations .......................... SUCCESS [
>> 1.209 s]
>> [INFO] Apache Hadoop Assemblies ........................... SUCCESS [
>> 0.221 s]
>> [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
>> 1.096 s]
>> [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
>> 4.158 s]
>> [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
>> 0.710 s]
>> [INFO] Apache Hadoop Auth ................................. SUCCESS [
>> 2.969 s]
>> [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
>> 1.591 s]
>> [INFO] Apache Hadoop Common ............................... SUCCESS [
>> 49.545 s]
>> [INFO] Apache Hadoop NFS .................................. SUCCESS [
>> 1.647 s]
>> [INFO] Apache Hadoop KMS .................................. SUCCESS [
>> 3.279 s]
>> [INFO] Apache Hadoop Registry ............................. SUCCESS [
>> 4.420 s]
>> [INFO] Apache Hadoop Common Project ....................... SUCCESS [
>> 0.118 s]
>> [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
>> 32.316 s]
>> [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:19
>> min]
>> [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
>> 1.336 s]
>> [INFO] Apache Hadoop HttpFS ............................... SUCCESS [
>> 4.519 s]
>> [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
>> 2.166 s]
>> [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
>> 13.665 s]
>> [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
>> 0.103 s]
>> [INFO] Apache Hadoop YARN ................................. SUCCESS [
>> 0.097 s]
>> [INFO] Apache Hadoop YARN API ............................. SUCCESS [
>> 23.601 s]
>> [INFO] Apache Hadoop YARN Common .......................... SUCCESS [
>> 19.536 s]
>> [INFO] Apache Hadoop YARN Server .......................... SUCCESS [
>> 0.080 s]
>> [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
>> 11.128 s]
>> [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
>> 16.997 s]
>> [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
>> 1.623 s]
>> [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
>> 4.663 s]
>> [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
>> 2.247 s]
>> [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
>> 26.548 s]
>> [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
>> 2.960 s]
>> [INFO] Apache Hadoop YARN Client .......................... SUCCESS [
>> 4.913 s]
>> [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
>> 1.732 s]
>> [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
>> 1.867 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
>> 0.123 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
>> 3.153 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
>> 3.503 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
>> 0.088 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
>> 4.038 s]
>> [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
>> 3.494 s]
>> [INFO] Apache Hadoop YARN Router .......................... SUCCESS [
>> 2.660 s]
>> [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
>> 1.713 s]
>> [INFO] Apache Hadoop YARN Applications .................... SUCCESS [
>> 0.091 s]
>> [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
>> 2.062 s]
>> [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
>> 1.544 s]
>> [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
>> 0.601 s]
>> [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
>> 6.887 s]
>> [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
>> 6.869 s]
>> [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
>> 3.005 s]
>> [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
>> 5.289 s]
>> [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
>> 4.543 s]
>> [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
>> 5.421 s]
>> [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
>> 1.516 s]
>> [INFO] Apache Hadoop YARN Services ........................ SUCCESS [
>> 0.077 s]
>> [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
>> 7.382 s]
>> [INFO] Apache Hadoop YARN Services API .................... SUCCESS [
>> 2.310 s]
>> [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
>> 0.086 s]
>> [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
>> 24.634 s]
>> [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
>> 0.183 s]
>> [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
>> 0.128 s]
>> [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
>> 1.844 s]
>> [INFO] Apache Hadoop YARN Site ............................ SUCCESS [
>> 0.109 s]
>> [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
>> 1.188 s]
>> [INFO] Apache Hadoop YARN UI .............................. SUCCESS [
>> 0.123 s]
>> [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
>> 9.718 s]
>> [INFO] Apache Hadoop YARN Project ......................... SUCCESS [
>> 18.364 s]
>> [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
>> 1.582 s]
>> [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
>> 2.293 s]
>> [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
>> 1.526 s]
>> [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
>> 2.003 s]
>> [INFO] Apache Hadoop MapReduce ............................ SUCCESS [
>> 4.660 s]
>> [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
>> 1.782 s]
>> [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
>> 2.608 s]
>> [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
>> 2.700 s]
>> [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
>> 2.079 s]
>> [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
>> 2.756 s]
>> [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
>> 1.692 s]
>> [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
>> 4.494 s]
>> [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
>> 0.085 s]
>> [INFO] Apache Hadoop Archives ............................. SUCCESS [
>> 1.052 s]
>> [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
>> 1.228 s]
>> [INFO] Apache Hadoop Rumen ................................ SUCCESS [
>> 2.137 s]
>> [INFO] Apache Hadoop Gridmix .............................. SUCCESS [
>> 1.913 s]
>> [INFO] Apache Hadoop Data Join ............................ SUCCESS [
>> 1.181 s]
>> [INFO] Apache Hadoop Extras ............................... SUCCESS [
>> 1.364 s]
>> [INFO] Apache Hadoop Pipes ................................ SUCCESS [
>> 0.079 s]
>> [INFO] Apache Hadoop OpenStack support .................... SUCCESS [
>> 1.598 s]
>> [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
>> 6.910 s]
>> [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
>> 1.256 s]
>> [INFO] Apache Hadoop Azure support ........................ SUCCESS [
>> 4.236 s]
>> [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
>> 1.051 s]
>> [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
>> 2.416 s]
>> [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
>> 2.483 s]
>> [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
>> 1.284 s]
>> [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
>> 1.575 s]
>> [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
>> 5.312 s]
>> [INFO] Apache Hadoop Tools ................................ SUCCESS [
>> 0.074 s]
>> [INFO] Apache Hadoop Client API ........................... SUCCESS [03:17
>> min]
>> [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:59
>> min]
>> [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
>> 1.076 s]
>> *[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS
>> [10:32
>> min]*
>> [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
>> 2.611 s]
>> [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
>> 0.533 s]
>> [INFO] Apache Hadoop Distribution ......................... SUCCESS [02:55
>> min]
>> [INFO] Apache Hadoop Client Modules ....................... SUCCESS [
>> 0.168 s]
>> [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
>> 2.697 s]
>> [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
>> 2.792 s]
>>
>> it used to be that -DskipShade would save the shading, but last time I
>> tried to skip the shading something failed.
>>
>> 1. has anyone else tried a full dist build recently -and how long did it
>> take?
>> 2. could they skip the shading?
>>
>
Re: why is a hadoop dist taking so long to build
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Steve,
Just tried building with -DskipShade, It worked for me.
Executed *mvn clean install -DskipTests -DskipShade -Pdist
-Dmaven.javadoc.skip=true*
Sharing the result :
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.4.0-SNAPSHOT:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [
1.587 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [
19.332 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [
3.538 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [
2.756 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [
0.348 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
2.319 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
5.573 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
2.083 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [
9.620 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
4.134 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [
51.670 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [
3.019 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [
3.651 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [
4.862 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [
0.109 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
24.093 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [
46.701 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
2.147 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [
4.048 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
3.467 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
6.077 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
0.340 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [
0.349 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [
16.439 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [
20.588 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [
0.082 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
6.682 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
13.558 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
1.800 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
4.442 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
2.093 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
19.693 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
3.153 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [
3.888 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
2.198 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
1.360 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
0.078 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
8.619 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
6.664 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
0.060 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
2.963 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
3.729 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [
2.989 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
2.976 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [
0.335 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
2.720 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
1.341 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
0.223 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
8.664 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
5.485 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
3.390 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
3.636 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
3.584 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
5.087 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
2.747 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [
0.393 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
4.652 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [
2.859 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
0.145 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
39.601 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
0.371 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
0.151 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
1.250 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [
0.131 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
1.733 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [
0.070 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
7.734 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [
7.258 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
0.684 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
3.412 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
2.193 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
3.257 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [
6.359 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
1.383 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
2.370 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
1.699 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
1.390 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
3.769 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
1.027 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
4.113 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
0.539 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [
0.831 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
0.635 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [
3.435 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [
1.090 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [
0.530 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [
0.488 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [
0.075 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [
1.449 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
16.265 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
1.903 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [
4.522 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
2.036 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
3.199 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
2.126 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
0.710 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
0.681 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
4.655 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [
0.068 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [
2.329 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [
0.646 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
1.650 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [
2.925 s]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
0.370 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
0.688 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [
41.321 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [
0.663 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
1.865 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
0.810 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [
0.076 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 09:23 min
[INFO] Finished at: 2020-05-12T19:46:31+05:30
[INFO]
------------------------------------------------------------------------
ayush@ayushpc:~/trunk$ mvn clean install -DskipTests -DskipShade -Pdist
-Dmaven.javadoc.skip=true
-Ayush
On Tue, 12 May 2020 at 19:21, Steve Loughran <st...@cloudera.com.invalid>
wrote:
> every week or so I do a -Pdist build of hadoop, so I can have an up to date
> command line. Today I kicked off the build with -Pdist -DskipTests
> -Dmaven.javadoc.skip=true
>
> And it took half an hour to complete -with 1/3 of that time just shading
> the minicluster
>
> [INFO] Apache Hadoop Main ................................. SUCCESS [
> 1.065 s]
> [INFO] Apache Hadoop Build Tools .......................... SUCCESS [
> 3.606 s]
> [INFO] Apache Hadoop Project POM .......................... SUCCESS [
> 1.374 s]
> [INFO] Apache Hadoop Annotations .......................... SUCCESS [
> 1.209 s]
> [INFO] Apache Hadoop Assemblies ........................... SUCCESS [
> 0.221 s]
> [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [
> 1.096 s]
> [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [
> 4.158 s]
> [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
> 0.710 s]
> [INFO] Apache Hadoop Auth ................................. SUCCESS [
> 2.969 s]
> [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
> 1.591 s]
> [INFO] Apache Hadoop Common ............................... SUCCESS [
> 49.545 s]
> [INFO] Apache Hadoop NFS .................................. SUCCESS [
> 1.647 s]
> [INFO] Apache Hadoop KMS .................................. SUCCESS [
> 3.279 s]
> [INFO] Apache Hadoop Registry ............................. SUCCESS [
> 4.420 s]
> [INFO] Apache Hadoop Common Project ....................... SUCCESS [
> 0.118 s]
> [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [
> 32.316 s]
> [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:19
> min]
> [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [
> 1.336 s]
> [INFO] Apache Hadoop HttpFS ............................... SUCCESS [
> 4.519 s]
> [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [
> 2.166 s]
> [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [
> 13.665 s]
> [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [
> 0.103 s]
> [INFO] Apache Hadoop YARN ................................. SUCCESS [
> 0.097 s]
> [INFO] Apache Hadoop YARN API ............................. SUCCESS [
> 23.601 s]
> [INFO] Apache Hadoop YARN Common .......................... SUCCESS [
> 19.536 s]
> [INFO] Apache Hadoop YARN Server .......................... SUCCESS [
> 0.080 s]
> [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [
> 11.128 s]
> [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [
> 16.997 s]
> [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [
> 1.623 s]
> [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [
> 4.663 s]
> [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [
> 2.247 s]
> [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [
> 26.548 s]
> [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [
> 2.960 s]
> [INFO] Apache Hadoop YARN Client .......................... SUCCESS [
> 4.913 s]
> [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [
> 1.732 s]
> [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [
> 1.867 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [
> 0.123 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [
> 3.153 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [
> 3.503 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [
> 0.088 s]
> [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [
> 4.038 s]
> [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [
> 3.494 s]
> [INFO] Apache Hadoop YARN Router .......................... SUCCESS [
> 2.660 s]
> [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [
> 1.713 s]
> [INFO] Apache Hadoop YARN Applications .................... SUCCESS [
> 0.091 s]
> [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [
> 2.062 s]
> [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [
> 1.544 s]
> [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [
> 0.601 s]
> [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [
> 6.887 s]
> [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [
> 6.869 s]
> [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [
> 3.005 s]
> [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [
> 5.289 s]
> [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [
> 4.543 s]
> [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [
> 5.421 s]
> [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [
> 1.516 s]
> [INFO] Apache Hadoop YARN Services ........................ SUCCESS [
> 0.077 s]
> [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [
> 7.382 s]
> [INFO] Apache Hadoop YARN Services API .................... SUCCESS [
> 2.310 s]
> [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [
> 0.086 s]
> [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [
> 24.634 s]
> [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [
> 0.183 s]
> [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [
> 0.128 s]
> [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [
> 1.844 s]
> [INFO] Apache Hadoop YARN Site ............................ SUCCESS [
> 0.109 s]
> [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [
> 1.188 s]
> [INFO] Apache Hadoop YARN UI .............................. SUCCESS [
> 0.123 s]
> [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [
> 9.718 s]
> [INFO] Apache Hadoop YARN Project ......................... SUCCESS [
> 18.364 s]
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [
> 1.582 s]
> [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [
> 2.293 s]
> [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [
> 1.526 s]
> [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [
> 2.003 s]
> [INFO] Apache Hadoop MapReduce ............................ SUCCESS [
> 4.660 s]
> [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [
> 1.782 s]
> [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [
> 2.608 s]
> [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [
> 2.700 s]
> [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [
> 2.079 s]
> [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [
> 2.756 s]
> [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [
> 1.692 s]
> [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [
> 4.494 s]
> [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [
> 0.085 s]
> [INFO] Apache Hadoop Archives ............................. SUCCESS [
> 1.052 s]
> [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [
> 1.228 s]
> [INFO] Apache Hadoop Rumen ................................ SUCCESS [
> 2.137 s]
> [INFO] Apache Hadoop Gridmix .............................. SUCCESS [
> 1.913 s]
> [INFO] Apache Hadoop Data Join ............................ SUCCESS [
> 1.181 s]
> [INFO] Apache Hadoop Extras ............................... SUCCESS [
> 1.364 s]
> [INFO] Apache Hadoop Pipes ................................ SUCCESS [
> 0.079 s]
> [INFO] Apache Hadoop OpenStack support .................... SUCCESS [
> 1.598 s]
> [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [
> 6.910 s]
> [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [
> 1.256 s]
> [INFO] Apache Hadoop Azure support ........................ SUCCESS [
> 4.236 s]
> [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [
> 1.051 s]
> [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [
> 2.416 s]
> [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [
> 2.483 s]
> [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [
> 1.284 s]
> [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [
> 1.575 s]
> [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [
> 5.312 s]
> [INFO] Apache Hadoop Tools ................................ SUCCESS [
> 0.074 s]
> [INFO] Apache Hadoop Client API ........................... SUCCESS [03:17
> min]
> [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:59
> min]
> [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [
> 1.076 s]
> *[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [10:32
> min]*
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [
> 2.611 s]
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [
> 0.533 s]
> [INFO] Apache Hadoop Distribution ......................... SUCCESS [02:55
> min]
> [INFO] Apache Hadoop Client Modules ....................... SUCCESS [
> 0.168 s]
> [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [
> 2.697 s]
> [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [
> 2.792 s]
>
> it used to be that -DskipShade would save the shading, but last time I
> tried to skip the shading something failed.
>
> 1. has anyone else tried a full dist build recently -and how long did it
> take?
> 2. could they skip the shading?
>