You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by 田磊 <to...@163.com> on 2021/07/24 03:32:28 UTC

flink写hbase

大家好:
我flink map算子并行度36处理完数据后用sink用2个并行度写hbase的话,处理完几万条数据后会报table 对象空指针的异常。按理来说之前已经处理了几万条数据,操作的都是hbase的同一张表,不可能table对象获取空指针啊。如果将sink的并行度调整为1的话,不会再有空指针的问题,但是数据写到不到10万条的时候sink的处理速度很慢,出现了严重的反压,导致上游处理数据的速度为0,任务失败。最后将并行度改为map算子的并行度改为24,sink的并行度改为2,虽然说也有反压。但是目前程序没有出现任务失败的情况。有大佬给解释一下什么原因么,有点想不通。
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:129)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1031)
at com.chinaunicom.audit.photo.history.handle.customsink.HbaseNumberAndQualitySink.invoke(HbaseNumberAndQualitySink.java:312)
at com.chinaunicom.audit.photo.history.handle.customsink.HbaseNumberAndQualitySink.invoke(HbaseNumberAndQualitySink.java:48)
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191)
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204)
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:583)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
at java.lang.Thread.run(Thread.java:745)