You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "David Li (Jira)" <ji...@apache.org> on 2020/12/29 14:11:00 UTC
[jira] [Commented] (ARROW-11062) When writing to flight stream,
Spark's mapPartitions is not working
[ https://issues.apache.org/jira/browse/ARROW-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256009#comment-17256009 ]
David Li commented on ARROW-11062:
----------------------------------
Hi Ravi, can you provide the output of {{mvn dependency:tree}} for your project to start with?
> When writing to flight stream, Spark's mapPartitions is not working
> -------------------------------------------------------------------
>
> Key: ARROW-11062
> URL: https://issues.apache.org/jira/browse/ARROW-11062
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
> Affects Versions: 2.0.0
> Reporter: Ravi Shankar
> Priority: Major
> Fix For: 2.0.0
>
>
> Hi,
> I have the following method:
>
> val outRDD = myRdd.mapPartitions { it =>
> val l = Location.forGrpcInsecure("10.0.0.113", 12233);
> val allocator = it.allocator.newChildAllocator("SparkFlightConnector", 0, Long.MaxValue)
> val client = FlightClient.builder(allocator, l).build();
> val desc = FlightDescriptor.path("wonderful")
> val stream = client.startPut(desc,it.root, new AsyncPutListener)
> it.foreach { root =>
> // doPut on the populated VectorSchemaRoot
> stream.putNext()
> }
> stream.completed()
> // Need to call this, or exceptions from the server get swallowed
> stream.getResult
>
> // println(it.next().contentToTSVString())
> client.close()
>
> Iterator.empty
> }.count
>
> Following is the error:
>
> Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
> at io.grpc.Metadata$Key.validateName(Metadata.java:742)
> at io.grpc.Metadata$Key.<init>(Metadata.java:750)
> at io.grpc.Metadata$Key.<init>(Metadata.java:668)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
> at io.grpc.Metadata$Key.of(Metadata.java:705)
> at io.grpc.Metadata$Key.of(Metadata.java:701)
> at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
>
> When I googles, its sayong some guava related jar issue. I did maven dependency tree and did not find anything wrong. Please help.
>
> Best,
> Ravion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)