You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2022/07/26 15:11:00 UTC
[jira] [Commented] (SPARK-39500) Ivy doesn't work correctly on IPv6-only environment
[ https://issues.apache.org/jira/browse/SPARK-39500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571488#comment-17571488 ]
Dongjoon Hyun commented on SPARK-39500:
---------------------------------------
For a record, we need the following to use the official IPv6 Maven Repo.
{code:java}
export DEFAULT_ARTIFACT_REPOSITORY=https://ipv6.repo1.maven.org/maven2/{code}
> Ivy doesn't work correctly on IPv6-only environment
> ---------------------------------------------------
>
> Key: SPARK-39500
> URL: https://issues.apache.org/jira/browse/SPARK-39500
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Dongjoon Hyun
> Priority: Major
>
> Ivy doesn't work correctly on IPv6.
> {code}
> SparkSubmitUtils.resolveMavenCoordinates(
> "org.apache.logging.log4j:log4j-api:2.17.2",
> SparkSubmitUtils.buildIvySettings(None, Some("/tmp/ivy")),
> transitive = true)
> {code}
> {code}
> % bin/spark-shell
> 22/06/16 22:22:12 WARN Utils: Your hostname, m1ipv6.local resolves to a loopback address: 127.0.0.1; using 2600:1700:232e:3de0:0:0:0:b instead (on interface en0)
> 22/06/16 22:22:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
> ===== https://ipv6.repo1.maven.org/maven2/
> =====https://maven-central.storage-download.googleapis.com/maven2/
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
> 22/06/16 22:22:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Spark context Web UI available at http://unknown1498776019fa.attlocal.net:4040
> Spark context available as 'sc' (master = local[*], app id = local-1655443334687).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT
> /_/
> Using Scala version 2.12.16 (OpenJDK 64-Bit Server VM, Java 17.0.3)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> :paste -raw
> // Entering paste mode (ctrl-D to finish)
> package org.apache.spark.deploy
> object Download {
> SparkSubmitUtils.resolveMavenCoordinates(
> "org.apache.logging.log4j:log4j-api:2.17.2",
> SparkSubmitUtils.buildIvySettings(None, Some("/tmp/ivy")),
> transitive = true)
> }
> // Exiting paste mode, now interpreting.
> scala> org.apache.spark.deploy.Download
> ===== https://ipv6.repo1.maven.org/maven2/
> =====https://maven-central.storage-download.googleapis.com/maven2/
> :: loading settings :: url = jar:file:/Users/dongjoon/APACHE/spark/assembly/target/scala-2.12/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> Ivy Default Cache set to: /tmp/ivy/cache
> The jars for the packages stored in: /tmp/ivy/jars
> org.apache.logging.log4j#log4j-api added as a dependency
> :: resolving dependencies :: org.apache.spark#spark-submit-parent-f47b503f-897e-4b92-95da-3806c32c220f;1.0
> confs: [default]
> :: resolution report :: resolve 95ms :: artifacts dl 0ms
> :: modules in use:
> ---------------------------------------------------------------------
> | | modules || artifacts |
> | conf | number| search|dwnlded|evicted|| number|dwnlded|
> ---------------------------------------------------------------------
> | default | 1 | 0 | 0 | 0 || 0 | 0 |
> ---------------------------------------------------------------------
> :: problems summary ::
> :::: WARNINGS
> module not found: org.apache.logging.log4j#log4j-api;2.17.2
> ==== local-m2-cache: tried
> file:/Users/dongjoon/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
> -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
> file:/Users/dongjoon/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
> ==== local-ivy-cache: tried
> /tmp/ivy/local/org.apache.logging.log4j/log4j-api/2.17.2/ivys/ivy.xml
> -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
> /tmp/ivy/local/org.apache.logging.log4j/log4j-api/2.17.2/jars/log4j-api.jar
> ==== ipv6: tried
> https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
> -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
> https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
> ==== central: tried
> https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
> -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
> https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
> ==== spark-packages: tried
> https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
> -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
> https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
> ::::::::::::::::::::::::::::::::::::::::::::::
> :: UNRESOLVED DEPENDENCIES ::
> ::::::::::::::::::::::::::::::::::::::::::::::
> :: org.apache.logging.log4j#log4j-api;2.17.2: not found
> ::::::::::::::::::::::::::::::::::::::::::::::
> :::: ERRORS
> Server access error at url https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable)
> Server access error at url https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable)
> Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable)
> Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable)
> Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable)
> Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable)
> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> java.lang.RuntimeException: [unresolved dependency: org.apache.logging.log4j#log4j-api;2.17.2: not found]
> at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1466)
> at org.apache.spark.deploy.Download$.<init>(<pastie>:4)
> at org.apache.spark.deploy.Download$.<clinit>(<pastie>)
> ... 47 elided
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org