You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "flyrain (via GitHub)" <gi...@apache.org> on 2025/01/24 02:11:39 UTC
[PR] Fix the Polaris test credentials [polaris]
flyrain opened a new pull request, #866:
URL: https://github.com/apache/polaris/pull/866
Fixed #865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #866:
URL: https://github.com/apache/polaris/pull/866#issuecomment-2619505165
Thanks @eric-maynard and @adutra for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the Polaris test credentials [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #866:
URL: https://github.com/apache/polaris/pull/866#issuecomment-2611518594
The test failed due to it use the test profile in application.properties. There are two ways to fix this
1. Using a different profile or even different application.properties file for the tests.
2. Creating a new profile for local run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the Polaris test credentials [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1928388407
##########
quarkus/defaults/src/main/resources/application.properties:
##########
@@ -130,6 +131,8 @@ polaris.authentication.token-broker.max-token-generation=PT1H
%test.quarkus.log.category."org.apache.polaris.service.storage.PolarisStorageIntegrationProviderImpl".level=ERROR
%test.quarkus.http.limits.max-body-size=1000000
%test.quarkus.otel.sdk.disabled=true
+%test.polaris.authentication.authenticator.type=test
Review Comment:
My idea was to do exactly the opposite: make all tests (automated + regression) use the default authenticator. Cf #804.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain merged PR #866:
URL: https://github.com/apache/polaris/pull/866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1931083161
##########
regtests/run.sh:
##########
@@ -26,6 +26,7 @@ if [ -z "${SPARK_HOME}" ]; then
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH"
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding
Review Comment:
fixed
##########
regtests/run_spark_sql.sh:
##########
@@ -48,6 +48,7 @@ cd ${REGTEST_HOME}
export SPARK_VERSION=spark-3.5.3
export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1929523209
##########
regtests/run_spark_sql.sh:
##########
@@ -56,7 +56,7 @@ if [ -z "${SPARK_HOME}"]; then
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
-SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:polaris}"
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
Review Comment:
This change may not be needed anymore since I removed the default values for these environment variables in #804.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #866:
URL: https://github.com/apache/polaris/pull/866#issuecomment-2613458142
Repurpose this PR as #804 will resolve the credential issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #866:
URL: https://github.com/apache/polaris/pull/866#issuecomment-2618725197
I also have a VPN on but wasn't having issues. I checked that this change does not introduce any regressions on my end, so LGTM. :shipit:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the Polaris test credentials [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1929277862
##########
quarkus/defaults/src/main/resources/application.properties:
##########
@@ -130,6 +131,8 @@ polaris.authentication.token-broker.max-token-generation=PT1H
%test.quarkus.log.category."org.apache.polaris.service.storage.PolarisStorageIntegrationProviderImpl".level=ERROR
%test.quarkus.http.limits.max-body-size=1000000
%test.quarkus.otel.sdk.disabled=true
+%test.polaris.authentication.authenticator.type=test
Review Comment:
That makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1929581424
##########
regtests/run_spark_sql.sh:
##########
@@ -56,7 +56,7 @@ if [ -z "${SPARK_HOME}"]; then
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
-SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:polaris}"
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
Review Comment:
Tested again based on #804. This change is still needed for my local run. I think this is due a recent change of my local VPN setting. However I couldn't touch VPN settings by myself. Given this script is mainly used for local test, I think this change should be good.
Here is the error message without this change:
```
Exception in thread "main" java.io.IOException: Failed to send RPC /jars/software.amazon.awssdk_http-client-spi-2.23.19.jar to mac.lan/192.168.86.27:55366: io.netty.channel.StacklessClosedChannelException
at org.apache.spark.network.client.TransportClient$2.handleFailure(TransportClient.java:164)
at org.apache.spark.network.client.TransportClient$StdChannelListener.operationComplete(TransportClient.java:372)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:999)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:860)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:877)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:302)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.netty.channel.StacklessClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the Polaris test credentials [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1927996302
##########
regtests/run_spark_sql.sh:
##########
@@ -48,14 +48,15 @@ cd ${REGTEST_HOME}
export SPARK_VERSION=spark-3.5.3
export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messy up driver local IP address binding
./setup.sh
if [ -z "${SPARK_HOME}"]; then
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
-SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:polaris}"
Review Comment:
Looks like this change is optional. It passed without the change for me locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1929523078
##########
regtests/run_spark_sql.sh:
##########
@@ -48,6 +48,7 @@ cd ${REGTEST_HOME}
export SPARK_VERSION=spark-3.5.3
export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messy up driver local IP address binding
Review Comment:
```suggestion
export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid Spark driver IP binding failure due to VPN setting [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #866:
URL: https://github.com/apache/polaris/pull/866#discussion_r1931007214
##########
regtests/run.sh:
##########
@@ -26,6 +26,7 @@ if [ -z "${SPARK_HOME}" ]; then
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH"
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding
Review Comment:
nit: space after `#`
##########
regtests/run_spark_sql.sh:
##########
@@ -48,6 +48,7 @@ cd ${REGTEST_HOME}
export SPARK_VERSION=spark-3.5.3
export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding
Review Comment:
nit: space after `#`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org