You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/11/24 03:43:20 UTC
[GitHub] [bigtop] 821684824 opened a new pull request #703: BIGTOP-3280: Bump Hadoop to 3.2.1
821684824 opened a new pull request #703:
URL: https://github.com/apache/bigtop/pull/703
I have used bigtop to release our production cluster based hadoop 3.2.1(with some 3.3.0 patchs fixed), and some hadoop env related not fixed in default.
1. Fix start/stop error service pid
2. Fix historyserver/timelineserver/proxyserver PIDFILE (affect start failed).
3. Add yarn federation router server support.
4. Support YARN new web ui.
5. Support for hdfs router.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a change in pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #703:
URL: https://github.com/apache/bigtop/pull/703#discussion_r584622808
##########
File path: bigtop-packages/src/common/hadoop/hadoop-yarn-router.svc
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+HADOOP_NAME="router"
+DAEMON="hadoop-yarn-$HADOOP_NAME"
+DESC="Hadoop $HADOOP_NAME"
+EXEC_PATH="/usr/bin/yarn"
Review comment:
"/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh" must be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] qizhu-lucas commented on a change in pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on a change in pull request #703:
URL: https://github.com/apache/bigtop/pull/703#discussion_r584471214
##########
File path: bigtop-packages/src/common/hadoop/install_hadoop.sh
##########
@@ -221,6 +221,7 @@ chmod 644 ${HADOOP_DIR}/lib/*.jar ${MAPREDUCE_DIR}/lib/*.jar ${HDFS_DIR}/lib/*.j
# Install webapps
cp -ra ${BUILD_DIR}/share/hadoop/hdfs/webapps ${HDFS_DIR}/
+cp -ra ${BUILD_DIR}/share/hadoop/yarn/webapps ${YARN_DIR}/
Review comment:
Thanks @iwasakims for review.
I have fixed it to add -Pyarn-ui in bigtop-packages/src/common/hadoop/do-component-build.
##########
File path: bigtop-packages/src/common/hadoop/install_hadoop.sh
##########
@@ -221,6 +221,7 @@ chmod 644 ${HADOOP_DIR}/lib/*.jar ${MAPREDUCE_DIR}/lib/*.jar ${HDFS_DIR}/lib/*.j
# Install webapps
cp -ra ${BUILD_DIR}/share/hadoop/hdfs/webapps ${HDFS_DIR}/
+cp -ra ${BUILD_DIR}/share/hadoop/yarn/webapps ${YARN_DIR}/
Review comment:
Thanks @iwasakims for review.
I have fixed it to add -Pyarn-ui in bigtop-packages/src/common/hadoop/do-component-build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] qizhu-lucas commented on a change in pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on a change in pull request #703:
URL: https://github.com/apache/bigtop/pull/703#discussion_r585220633
##########
File path: bigtop-packages/src/common/hadoop/hadoop-yarn-router.svc
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+HADOOP_NAME="router"
+DAEMON="hadoop-yarn-$HADOOP_NAME"
+DESC="Hadoop $HADOOP_NAME"
+EXEC_PATH="/usr/bin/yarn"
Review comment:
Thanks a lot @iwasakims for patient review.
Updated it in latest pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #703: BIGTOP-3280: Bump Hadoop to 3.2.1
Posted by GitBox <gi...@apache.org>.
guyuqi commented on pull request #703:
URL: https://github.com/apache/bigtop/pull/703#issuecomment-732651679
Thank you for working on it.
As mentioned in BIGTOP-3280, we plan to upgrade Hadoop to 3.3.0.
But anyway, nice to meet you in Bigtop.
It would be appreciated if you join us and contribute more.
Pls join our mailing list: https://bigtop.apache.org/mail-lists.html, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] qizhu-lucas commented on pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on pull request #703:
URL: https://github.com/apache/bigtop/pull/703#issuecomment-782598192
@iwasakims I have rebased it about rpm, but the deb not updated, if you could help with deb.
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #703:
URL: https://github.com/apache/bigtop/pull/703#issuecomment-787865681
Thanks for the update, @qizhu-lucas. While packaging works now, init scripts could not launch daemons. We should use hadoop-daemon.sh and yarn-daemon.sh to start daemons. See other `hadoop-hdfs-*.svc` and `hadoop-yarn-*.svc` as reference.
```
# systemctl start hadoop-hdfs-dfsrouter
# journalctl -xe
...
-- Unit hadoop-hdfs-dfsrouter.service has begun starting up.
Mar 01 10:55:08 148cc1d32442 runuser[31068]: pam_unix(runuser:session): session opened for user hdfs by (uid=0)
Mar 01 10:55:09 148cc1d32442 hadoop-hdfs-dfsrouter[31051]: ERROR: start is not COMMAND nor fully qualified CLASSNAME.
Mar 01 10:55:09 148cc1d32442 hadoop-hdfs-dfsrouter[31051]: Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
Mar 01 10:55:09 148cc1d32442 hadoop-hdfs-dfsrouter[31051]: OPTIONS is none or any of:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #703:
URL: https://github.com/apache/bigtop/pull/703
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #703:
URL: https://github.com/apache/bigtop/pull/703#issuecomment-778055525
@qizhu-lucas Sorry for late reply. This has conflicts now. Could you rebase this?
Since Bigtop supports both .rpm and .deb, you need to fix files under bigtop-packages/src/deb/hadoop too. If you don't have environment to test, I can cover them in follow-up JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #703:
URL: https://github.com/apache/bigtop/pull/703#issuecomment-788643122
I filed [BIGTOP-3513](https://issues.apache.org/jira/browse/BIGTOP-3513) as a follow-up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a change in pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #703:
URL: https://github.com/apache/bigtop/pull/703#discussion_r584622452
##########
File path: bigtop-packages/src/common/hadoop/hadoop-hdfs-dfsrouter.svc
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+HADOOP_NAME="dfsrouter"
+DAEMON="hadoop-hdfs-$HADOOP_NAME"
+DESC="Hadoop $HADOOP_NAME"
+EXEC_PATH="/usr/bin/hdfs"
Review comment:
"/usr/lib/hadoop/sbin/hadoop-daemon.sh" must be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a change in pull request #703: BIGTOP-3456: Add hadoop-yarn-router and hadoop-hdfs-dfsrouter to hadoop services
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #703:
URL: https://github.com/apache/bigtop/pull/703#discussion_r579755839
##########
File path: bigtop-packages/src/common/hadoop/install_hadoop.sh
##########
@@ -221,6 +221,7 @@ chmod 644 ${HADOOP_DIR}/lib/*.jar ${MAPREDUCE_DIR}/lib/*.jar ${HDFS_DIR}/lib/*.j
# Install webapps
cp -ra ${BUILD_DIR}/share/hadoop/hdfs/webapps ${HDFS_DIR}/
+cp -ra ${BUILD_DIR}/share/hadoop/yarn/webapps ${YARN_DIR}/
Review comment:
`./gradlew hadoop-pkg` failed with the following error.
```
+ cp -ra /home/centos/srcs/bigtop/build/hadoop/rpm/BUILD/hadoop-3.2.2-src/build/share/hadoop/yarn/webapps /home/centos/srcs/bigtop/build/hadoop/rpm/BUILDROOT/hadoop-3.2.2-1.el7.x86_64/usr/lib/hadoop-yarn/
cp: cannot stat '/home/centos/srcs/bigtop/build/hadoop/rpm/BUILD/hadoop-3.2.2-src/build/share/hadoop/yarn/webapps': No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.PviJ3q (%install)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org