You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ramesh Mathikumar <me...@googlemail.com> on 2020/10/27 20:10:28 UTC
Spark Portable Runner + Docker
Hi Group -- Has anyone got this to work? For me it does not either in the IDE or in Colab. Whats the community take on this one?
Re: Spark Portable Runner + Docker
Posted by Kyle Weaver <kc...@google.com>.
There are already replies on the other thread. Please keep the discussion
there so it's easy to follow.
https://lists.apache.org/thread.html/r4245ea5320d362434ce5cafafbacd4cee393197478e4604b1e09a35d%40%3Cuser.beam.apache.org%3E
On Wed, Oct 28, 2020 at 12:57 PM Ramesh Mathikumar <me...@googlemail.com>
wrote:
> Hi Alex -- Please se the details you are looking for.
>
> I am running a sample pipeline and my environment is this.
>
> python "SaiStudy - Apache-Beam-Spark.py" --runner=PortableRunner
> --job_endpoint=192.168.99.102:8099
>
> My Spark is running on a Docker Container and I can see that the
> JobService is running at 8099.
>
> I am getting the following error: grpc._channel._MultiThreadedRendezvous:
> <_MultiThreadedRendezvous of RPC that terminated with: status =
> StatusCode.UNAVAILABLE details = "failed to connect to all addresses"
> debug_error_string =
> "{"created":"@1603539936.536000000","description":"Failed to pick
> subchannel","file":"src/core/ext/filters/client_channel/client_chann
> el.cc","file_line":4090,"referenced_errors":[{"created":"@1603539936.536000000","description":"failed
> to connect to all addresses","file":"src/core/ext/filters/cli
> ent_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>
> When I curl to ip:port, I can see the following error from the docker logs
> Oct 24, 2020 11:34:50 AM
> org.apache.beam.vendor.grpc.v1p26p0.io.grpc.netty.NettyServerTransport
> notifyTerminated INFO: Transport failed
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2Exception:
> Unexpected HTTP/1.x request: GET / at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:103)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:302)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProt
> ection(ByteToMessageDecoder.java:505) at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:37
> 4) at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044
> ) at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
>
> Help Please.
>
> On 2020/10/28 15:37:22, Alexey Romanenko <ar...@gmail.com>
> wrote:
> > Hi Ramesh,
> >
> > By “+ Docker” do you mean Docker SDK Harness or running a Spark in
> Docker? For the former I believe it works fine.
> >
> > Could you share more details of what kind of error you are facing?
> >
> > > On 27 Oct 2020, at 21:10, Ramesh Mathikumar <me...@googlemail.com>
> wrote:
> > >
> > > Hi Group -- Has anyone got this to work? For me it does not either in
> the IDE or in Colab. Whats the community take on this one?
> >
> >
>
Re: Spark Portable Runner + Docker
Posted by Ramesh Mathikumar <me...@googlemail.com>.
Hi Alex -- Please se the details you are looking for.
I am running a sample pipeline and my environment is this.
python "SaiStudy - Apache-Beam-Spark.py" --runner=PortableRunner --job_endpoint=192.168.99.102:8099
My Spark is running on a Docker Container and I can see that the JobService is running at 8099.
I am getting the following error: grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1603539936.536000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_chann el.cc","file_line":4090,"referenced_errors":[{"created":"@1603539936.536000000","description":"failed to connect to all addresses","file":"src/core/ext/filters/cli ent_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
When I curl to ip:port, I can see the following error from the docker logs Oct 24, 2020 11:34:50 AM org.apache.beam.vendor.grpc.v1p26p0.io.grpc.netty.NettyServerTransport notifyTerminated INFO: Transport failed org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET / at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:103) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:302) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProt
ection(ByteToMessageDecoder.java:505) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:37
4) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044
) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)
Help Please.
On 2020/10/28 15:37:22, Alexey Romanenko <ar...@gmail.com> wrote:
> Hi Ramesh,
>
> By “+ Docker” do you mean Docker SDK Harness or running a Spark in Docker? For the former I believe it works fine.
>
> Could you share more details of what kind of error you are facing?
>
> > On 27 Oct 2020, at 21:10, Ramesh Mathikumar <me...@googlemail.com> wrote:
> >
> > Hi Group -- Has anyone got this to work? For me it does not either in the IDE or in Colab. Whats the community take on this one?
>
>
Re: Spark Portable Runner + Docker
Posted by Alexey Romanenko <ar...@gmail.com>.
Hi Ramesh,
By “+ Docker” do you mean Docker SDK Harness or running a Spark in Docker? For the former I believe it works fine.
Could you share more details of what kind of error you are facing?
> On 27 Oct 2020, at 21:10, Ramesh Mathikumar <me...@googlemail.com> wrote:
>
> Hi Group -- Has anyone got this to work? For me it does not either in the IDE or in Colab. Whats the community take on this one?