You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 姜戎 <21...@qq.com> on 2020/04/30 08:02:57 UTC
kafka rdd save to hive errer
failed to get records for compacted ...after polling for120000
partition 0 offset min=0 max=1427265
get offsetrange 0 until 500000 to make rdd
Re: kafka rdd save to hive errer
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
E.g., as per
https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
val df = spark
.read
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1")
.option("startingOffsets", "earliest")
.option("endingOffsets", "latest")
.load()
On Sun, May 3, 2020 at 1:50 AM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> Hello 姜戎 ,
>
> Unfortunately there's not enough information in your email for us to help
> you. Are you trying to use Spark Batch to read from Kafka? Have you tried
> setting "endingOffsets" to "latest" instead of an arbitrary number?
>
> Kind regards,
>
> Liam Clarke-Hutchinson
>
>
> On Fri, May 1, 2020 at 2:36 AM 姜戎 <21...@qq.com> wrote:
>
>> failed to get records for compacted ...after polling for120000
>> partition 0 offset min=0 max=1427265
>> get offsetrange 0 until 500000 to make rdd
>
>
Re: kafka rdd save to hive errer
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hello 姜戎 ,
Unfortunately there's not enough information in your email for us to help
you. Are you trying to use Spark Batch to read from Kafka? Have you tried
setting "endingOffsets" to "latest" instead of an arbitrary number?
Kind regards,
Liam Clarke-Hutchinson
On Fri, May 1, 2020 at 2:36 AM 姜戎 <21...@qq.com> wrote:
> failed to get records for compacted ...after polling for120000
> partition 0 offset min=0 max=1427265
> get offsetrange 0 until 500000 to make rdd