You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by K Fred <yu...@gmail.com> on 2018/11/08 03:01:06 UTC

Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation.

Hi,

I am running into an issue using HBaseIO with FlinkRunner.
The exception is below:
Exception in thread "main" java.lang.IllegalStateException: Unable to
return a default Coder for
ParDo(HBaseProfile)/ParMultiDo(HBaseProfile).output [PCollection]. Correct
one of the following root causes:
  No Coder has been manually specified;  you may do so using .setCoder().
  Inferring a Coder from the CoderRegistry failed: Unable to provide a
Coder for org.apache.hadoop.hbase.client.Mutation.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
  Using the default output Coder from the producing PTransform failed:
PTransform.getOutputCoder called.
at
org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:278)
at
org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:115)
at
org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:190)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:536)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:370)
at ac.cn.iie.process.ProfileProcess.process(ProfileProcess.java:91)
at ac.cn.iie.Bootstrap.main(Bootstrap.java:25)

I have no idea about this question. Just guess that may be related to Coder
register. So i regist Coder provider manually. The detail is :
https://stackoverflow.com/questions/53200867/unable-to-provide-a-coder-for-org-apache-hadoop-hbase-client-mutation-using-hbas/53200894#53200894

Does anyone have better solution? I think it shouldn't register by user. It
camn register automatically by pipeline.
-- 

Fred

Re: Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation.

Posted by Lukasz Cwik <lc...@google.com>.
Good to know that the suggested fix work. Feel free to upvote the answer so
that others can see what worked.

On Fri, Nov 9, 2018 at 12:29 AM K Fred <yu...@gmail.com> wrote:

> Hi Lukasz,
>
> I have solved this issue by setting ServicesResourceTransformer on my
> maven-shade transformets list.
> And there are another way to solve this question by adding
> HBaseMutationCoder in code manually.
>
> Thanks.
>
> On Fri, Nov 9, 2018 at 3:16 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Posted my answer on SO, reposting here for visibility:
>>
>> Note that the HBaseCoderProviderRegistrar[1] already registers the
>> HBaseMutationCoder for the Mutation type automatically already.
>>
>> Using the maven-shade plugin without handling service files contained in
>> META-INF/ inside your output jar is a common pitfall for users. You want to
>> add the ServicesResourceTransformer to your maven-shade transformers list.
>>
>> See Apache documentation[2] and ServiceLoader javadoc[3] for more
>> details. There are many other questions like this on StackOverflow.
>>
>>
>>   [1]:
>> https://github.com/apache/beam/blob/95f1a5aeba85cd4a6e0eb1ce6eee141e3667d86d/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java#L30
>>   [2]:
>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>>   [3]:
>> https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
>>
>>
>> On Thu, Nov 8, 2018 at 10:13 AM K Fred <yu...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am running into an issue using HBaseIO with FlinkRunner.
>>> The exception is below:
>>> Exception in thread "main" java.lang.IllegalStateException: Unable to
>>> return a default Coder for
>>> ParDo(HBaseProfile)/ParMultiDo(HBaseProfile).output [PCollection]. Correct
>>> one of the following root causes:
>>>   No Coder has been manually specified;  you may do so using .setCoder().
>>>   Inferring a Coder from the CoderRegistry failed: Unable to provide a
>>> Coder for org.apache.hadoop.hbase.client.Mutation.
>>>   Building a Coder using a registered CoderProvider failed.
>>>   See suppressed exceptions for detailed failures.
>>>   Using the default output Coder from the producing PTransform failed:
>>> PTransform.getOutputCoder called.
>>> at
>>> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
>>> at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:278)
>>> at
>>> org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:115)
>>> at
>>> org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:190)
>>> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:536)
>>> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>>> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:370)
>>> at ac.cn.iie.process.ProfileProcess.process(ProfileProcess.java:91)
>>> at ac.cn.iie.Bootstrap.main(Bootstrap.java:25)
>>>
>>> I have no idea about this question. Just guess that may be related to
>>> Coder register. So i regist Coder provider manually. The detail is :
>>> https://stackoverflow.com/questions/53200867/unable-to-provide-a-coder-for-org-apache-hadoop-hbase-client-mutation-using-hbas/53200894#53200894
>>>
>>> Does anyone have better solution? I think it shouldn't register by user.
>>> It camn register automatically by pipeline.
>>> --
>>>
>>> Fred
>>>
>> --
>
> Fred
>

Re: Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation.

Posted by K Fred <yu...@gmail.com>.
Hi Lukasz,

I have solved this issue by setting ServicesResourceTransformer on my
maven-shade transformets list.
And there are another way to solve this question by adding
HBaseMutationCoder in code manually.

Thanks.

On Fri, Nov 9, 2018 at 3:16 AM Lukasz Cwik <lc...@google.com> wrote:

> Posted my answer on SO, reposting here for visibility:
>
> Note that the HBaseCoderProviderRegistrar[1] already registers the
> HBaseMutationCoder for the Mutation type automatically already.
>
> Using the maven-shade plugin without handling service files contained in
> META-INF/ inside your output jar is a common pitfall for users. You want to
> add the ServicesResourceTransformer to your maven-shade transformers list.
>
> See Apache documentation[2] and ServiceLoader javadoc[3] for more details.
> There are many other questions like this on StackOverflow.
>
>
>   [1]:
> https://github.com/apache/beam/blob/95f1a5aeba85cd4a6e0eb1ce6eee141e3667d86d/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java#L30
>   [2]:
> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>   [3]:
> https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
>
>
> On Thu, Nov 8, 2018 at 10:13 AM K Fred <yu...@gmail.com> wrote:
>
>> Hi,
>>
>> I am running into an issue using HBaseIO with FlinkRunner.
>> The exception is below:
>> Exception in thread "main" java.lang.IllegalStateException: Unable to
>> return a default Coder for
>> ParDo(HBaseProfile)/ParMultiDo(HBaseProfile).output [PCollection]. Correct
>> one of the following root causes:
>>   No Coder has been manually specified;  you may do so using .setCoder().
>>   Inferring a Coder from the CoderRegistry failed: Unable to provide a
>> Coder for org.apache.hadoop.hbase.client.Mutation.
>>   Building a Coder using a registered CoderProvider failed.
>>   See suppressed exceptions for detailed failures.
>>   Using the default output Coder from the producing PTransform failed:
>> PTransform.getOutputCoder called.
>> at
>> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
>> at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:278)
>> at
>> org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:115)
>> at
>> org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:190)
>> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:536)
>> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:370)
>> at ac.cn.iie.process.ProfileProcess.process(ProfileProcess.java:91)
>> at ac.cn.iie.Bootstrap.main(Bootstrap.java:25)
>>
>> I have no idea about this question. Just guess that may be related to
>> Coder register. So i regist Coder provider manually. The detail is :
>> https://stackoverflow.com/questions/53200867/unable-to-provide-a-coder-for-org-apache-hadoop-hbase-client-mutation-using-hbas/53200894#53200894
>>
>> Does anyone have better solution? I think it shouldn't register by user.
>> It camn register automatically by pipeline.
>> --
>>
>> Fred
>>
> --

Fred

Re: Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation.

Posted by Lukasz Cwik <lc...@google.com>.
Posted my answer on SO, reposting here for visibility:

Note that the HBaseCoderProviderRegistrar[1] already registers the
HBaseMutationCoder for the Mutation type automatically already.

Using the maven-shade plugin without handling service files contained in
META-INF/ inside your output jar is a common pitfall for users. You want to
add the ServicesResourceTransformer to your maven-shade transformers list.

See Apache documentation[2] and ServiceLoader javadoc[3] for more details.
There are many other questions like this on StackOverflow.


  [1]:
https://github.com/apache/beam/blob/95f1a5aeba85cd4a6e0eb1ce6eee141e3667d86d/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java#L30
  [2]:
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
  [3]:
https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html


On Thu, Nov 8, 2018 at 10:13 AM K Fred <yu...@gmail.com> wrote:

> Hi,
>
> I am running into an issue using HBaseIO with FlinkRunner.
> The exception is below:
> Exception in thread "main" java.lang.IllegalStateException: Unable to
> return a default Coder for
> ParDo(HBaseProfile)/ParMultiDo(HBaseProfile).output [PCollection]. Correct
> one of the following root causes:
>   No Coder has been manually specified;  you may do so using .setCoder().
>   Inferring a Coder from the CoderRegistry failed: Unable to provide a
> Coder for org.apache.hadoop.hbase.client.Mutation.
>   Building a Coder using a registered CoderProvider failed.
>   See suppressed exceptions for detailed failures.
>   Using the default output Coder from the producing PTransform failed:
> PTransform.getOutputCoder called.
> at
> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:278)
> at
> org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:115)
> at
> org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:190)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:536)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:370)
> at ac.cn.iie.process.ProfileProcess.process(ProfileProcess.java:91)
> at ac.cn.iie.Bootstrap.main(Bootstrap.java:25)
>
> I have no idea about this question. Just guess that may be related to
> Coder register. So i regist Coder provider manually. The detail is :
> https://stackoverflow.com/questions/53200867/unable-to-provide-a-coder-for-org-apache-hadoop-hbase-client-mutation-using-hbas/53200894#53200894
>
> Does anyone have better solution? I think it shouldn't register by user.
> It camn register automatically by pipeline.
> --
>
> Fred
>