A very short writedown of the following error, which apperently this user also encountered and documented (github). Be aware, that this error code might appear in several scenarios. It just happened, that in my specific situation, it was an easy solve.
Setup
I run a Spark image (spark:3.5.0-scala2.12-java11-python3-ubuntu) in a Docker Compose. To start a master server, I defined the command
command: "/opt/spark/bin/spark-class org.apache.spark.deploy.master.Master --ip ${SPARK_MASTER_HOST} --port 7077 --webui-port ${SPARK_MASTER_WEBUI_PORT}"
However, the container shut downs almost immediately with the following error code
24/05/07 13:54:03 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
java.nio.channels.UnresolvedAddressException
at java.base/sun.nio.ch.Net.checkAddress(Unknown Source)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:600)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:579)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
24/05/07 13:54:03 INFO ShutdownHookManager: Shutdown hook called
Solution
So actually ChatGPT gave me the right hint, he said:
The error message you’re encountering, java.nio.channels.UnresolvedAddressException, typically occurs when Spark (or an underlying networking library like Netty) is trying to bind to a network address that cannot be resolved. This error generally indicates that the application is attempting to use an incorrect or non-existent hostname or IP address.
Possible Causes and Solutions
- Invalid Hostname/IP Address:
- Ensure that the hostname or IP address you’re using is valid and resolvable. Check the network settings and any environment variables used to specify the addresses.
- In a Docker environment, confirm that all services and containers can correctly resolve the specified hostnames.
- Misconfigured Bind Address:
- The bind address is the network interface Spark attempts to bind to. Verify the values of environment variables like
SPARK_MASTER_HOST,SPARK_LOCAL_IP, or any other bind address parameters.
- The bind address is the network interface Spark attempts to bind to. Verify the values of environment variables like
That gave me the necessary hint. I just defined SPARK_MASTER_HOST wrongly and should have defined it as 0.0.0.0 as the application is running in a Docker Container.
