You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/10/22 10:37:24 UTC
[GitHub] [hadoop] adoroszlai opened a new pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
adoroszlai opened a new pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577
## What changes were proposed in this pull request?
1. Update to Hadoop 3.3.1 release.
2. Change to `curl` for download because `wget` thinks `apache.org` certificate is expired.
https://issues.apache.org/jira/browse/HADOOP-17713
## How was this patch tested?
Built the new image locally:
```
$ ./build.sh
...
=> => writing image sha256:23f613502b2b883ee4978b997409ca6c24116722e67caa6a9f521aef04c0a6bc 0.0s
=> => naming to docker.io/apache/hadoop:3 0.0s
```
and tested it with the included sample Docker Compose cluster:
```
$ docker-compose up -d --scale datanode=3 --scale nodemanager=3
$ docker-compose exec resourcemanager yarn jar \
/opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.1.jar pi 3 3
...
Job Finished in 36.514 seconds
Estimated value of Pi is 3.55555555555555555556
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#issuecomment-963080996
Thanks @ayushtkn and @aajisaka for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#discussion_r740990384
##########
File path: build.sh
##########
@@ -18,7 +18,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e
mkdir -p build
if [ ! -d "$DIR/build/apache-rat-0.13" ]; then
- wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -O "$DIR/build/apache-rat.tar.gz"
+ curl -LSs "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -o "$DIR/build/apache-rat.tar.gz"
Review comment:
Thanks. I forgot about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#discussion_r740970427
##########
File path: build.sh
##########
@@ -18,7 +18,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e
mkdir -p build
if [ ! -d "$DIR/build/apache-rat-0.13" ]; then
- wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -O "$DIR/build/apache-rat.tar.gz"
+ curl -LSs "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -o "$DIR/build/apache-rat.tar.gz"
Review comment:
CDN can be used: https://dlcdn.apache.org/creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#issuecomment-957348441
> Now CDN (https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz) can be used for downloading Hadoop binary tarball. Would you update the URL?
Thanks @aajisaka for the info, updated the URL (in bfd6f64).
> ```
> bash-4.2$ hdfs dfs -ls /
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> ```
>
> I think the log4j.properties file below is being added to wrong place, The HADOOP_CONF_DIR is /etc/hadoop not /opt/hadoop/....
Thanks @ayushtkn for testing the patch. After setting `HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop` (in adf2000) the command works fine:
```
bash-4.2$ hdfs dfs -ls /
Found 21 items
-rwxr-xr-x 1 root root 0 2021-11-02 11:15 /.dockerenv
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#issuecomment-957348441
> Now CDN (https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz) can be used for downloading Hadoop binary tarball. Would you update the URL?
Thanks @aajisaka for the info, updated the URL (in bfd6f64).
> ```
> bash-4.2$ hdfs dfs -ls /
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> ```
>
> I think the log4j.properties file below is being added to wrong place, The HADOOP_CONF_DIR is /etc/hadoop not /opt/hadoop/....
Thanks @ayushtkn for testing the patch. After setting `HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop` (in adf2000) the command works fine:
```
bash-4.2$ hdfs dfs -ls /
Found 21 items
-rwxr-xr-x 1 root root 0 2021-11-02 11:15 /.dockerenv
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#discussion_r740990384
##########
File path: build.sh
##########
@@ -18,7 +18,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e
mkdir -p build
if [ ! -d "$DIR/build/apache-rat-0.13" ]; then
- wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -O "$DIR/build/apache-rat.tar.gz"
+ curl -LSs "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -o "$DIR/build/apache-rat.tar.gz"
Review comment:
Thanks. I forgot about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#issuecomment-957348441
> Now CDN (https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz) can be used for downloading Hadoop binary tarball. Would you update the URL?
Thanks @aajisaka for the info, updated the URL (in bfd6f64).
> ```
> bash-4.2$ hdfs dfs -ls /
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> ```
>
> I think the log4j.properties file below is being added to wrong place, The HADOOP_CONF_DIR is /etc/hadoop not /opt/hadoop/....
Thanks @ayushtkn for testing the patch. After setting `HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop` (in adf2000) the command works fine:
```
bash-4.2$ hdfs dfs -ls /
Found 21 items
-rwxr-xr-x 1 root root 0 2021-11-02 11:15 /.dockerenv
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#discussion_r740970427
##########
File path: build.sh
##########
@@ -18,7 +18,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e
mkdir -p build
if [ ! -d "$DIR/build/apache-rat-0.13" ]; then
- wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -O "$DIR/build/apache-rat.tar.gz"
+ curl -LSs "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -o "$DIR/build/apache-rat.tar.gz"
Review comment:
CDN can be used: https://dlcdn.apache.org/creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#discussion_r740990384
##########
File path: build.sh
##########
@@ -18,7 +18,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e
mkdir -p build
if [ ! -d "$DIR/build/apache-rat-0.13" ]; then
- wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -O "$DIR/build/apache-rat.tar.gz"
+ curl -LSs "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -o "$DIR/build/apache-rat.tar.gz"
Review comment:
Thanks. I forgot about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577#discussion_r740970427
##########
File path: build.sh
##########
@@ -18,7 +18,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e
mkdir -p build
if [ ! -d "$DIR/build/apache-rat-0.13" ]; then
- wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -O "$DIR/build/apache-rat.tar.gz"
+ curl -LSs "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz" -o "$DIR/build/apache-rat.tar.gz"
Review comment:
CDN can be used: https://dlcdn.apache.org/creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #3577: HADOOP-17713. Update apache/hadoop:3 docker image to 3.3.1 release
Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #3577:
URL: https://github.com/apache/hadoop/pull/3577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org