You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Juan Cuzmar <jc...@protonmail.com> on 2023/03/23 07:16:11 UTC
Recommended way to set coder for JdbcIO with Apache Beam
Hello all,
I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
Here's a snippet of my code:
.apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
.withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
...
.withRowMapper(resultSet -> {
// Mapping logic here
}).withOutputParallelization(false)
).setCoder(SerializableCoder.of(InboundData.class))
When I run the code, I encounter the following error:
java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
...
I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
Thank you for your time and assistance.
Best regards!
Re: Recommended way to set coder for JdbcIO with Apache Beam
Posted by Juan Cuzmar <jc...@protonmail.com>.
Hello!
i created this example:
https://github.com/j1cs/coder-error-beam
and the issue you told me:
https://github.com/apache/beam/issues/26003
Juan Cuzmar.
------- Original Message -------
On Monday, March 27th, 2023 at 10:58 AM, Alexey Romanenko <ar...@gmail.com> wrote:
> Hmm, it worked fine for me on a primitive pipeline and default AvroCode for simple InboundData implementation.
>
> Could you create a GitHub issue for that (https://github.com/apache/beam/issues)? It would be very helpful to provide the steps there how to reproduce this issue.
>
> Thanks,
> Alexey
>
> > On 23 Mar 2023, at 19:08, Juan Cuzmar jcuzmar@protonmail.com wrote:
> >
> > unfortunately didn't work.
> > I added the @DefaultCoder as you told me and i removed the .withCoder from my JdbcIO pipe and showed the same error:
> >
> > java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform. Provide a coder via withCoder, or ensure that one can be inferred from the provided RowMapper.
> >
> > Juan Cuzmar.
> >
> > ------- Original Message -------
> > On Thursday, March 23rd, 2023 at 2:17 PM, Alexey Romanenko aromanenko.dev@gmail.com wrote:
> >
> > > Well, perhaps it’s a bug (I’ll try to check it on my side later).
> > >
> > > Can you try to annotate InboundData class with “@DefaultCoder(AvroCoder.class)” for example and see if it will work?
> > >
> > > > On 23 Mar 2023, at 15:13, Juan Cuzmar jcuzmar@protonmail.com wrote:
> > > >
> > > > Alexey yes! here it is:
> > > >
> > > > import lombok.*;
> > > > import lombok.extern.jackson.Jacksonized;
> > > >
> > > > import java.io.Serializable;
> > > > import java.util.List;
> > > > import java.util.Map;
> > > >
> > > > @Jacksonized
> > > > @Builder
> > > > @Getter
> > > > @Value
> > > > @EqualsAndHashCode
> > > > @ToString
> > > > public class InboundData implements Serializable {
> > > >
> > > > String payload;
> > > > Map<String, String> attributes;
> > > > List<Store> stores;
> > > > }
> > > >
> > > > ------- Original Message -------
> > > > On Thursday, March 23rd, 2023 at 8:14 AM, Alexey Romanenko aromanenko.dev@gmail.com wrote:
> > > >
> > > > > Could you share a class declaration of your InboundData class? Is it just a POJO?
> > > > >
> > > > > —
> > > > > Alexey
> > > > >
> > > > > > On 23 Mar 2023, at 08:16, Juan Cuzmar jcuzmar@protonmail.com wrote:
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
> > > > > >
> > > > > > Here's a snippet of my code:
> > > > > >
> > > > > > .apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
> > > > > > .withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
> > > > > > ...
> > > > > > .withRowMapper(resultSet -> {
> > > > > > // Mapping logic here
> > > > > > }).withOutputParallelization(false)
> > > > > > ).setCoder(SerializableCoder.of(InboundData.class))
> > > > > >
> > > > > > When I run the code, I encounter the following error:
> > > > > >
> > > > > > java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
> > > > > > ...
> > > > > >
> > > > > > I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
> > > > > >
> > > > > > If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
> > > > > >
> > > > > > Thank you for your time and assistance.
> > > > > >
> > > > > > Best regards!
Re: Recommended way to set coder for JdbcIO with Apache Beam
Posted by Alexey Romanenko <ar...@gmail.com>.
Hmm, it worked fine for me on a primitive pipeline and default AvroCode for simple InboundData implementation.
Could you create a GitHub issue for that (https://github.com/apache/beam/issues)? It would be very helpful to provide the steps there how to reproduce this issue.
Thanks,
Alexey
> On 23 Mar 2023, at 19:08, Juan Cuzmar <jc...@protonmail.com> wrote:
>
> unfortunately didn't work.
> I added the @DefaultCoder as you told me and i removed the .withCoder from my JdbcIO pipe and showed the same error:
>
> java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform. Provide a coder via withCoder, or ensure that one can be inferred from the provided RowMapper.
>
>
>
>
> Juan Cuzmar.
>
> ------- Original Message -------
> On Thursday, March 23rd, 2023 at 2:17 PM, Alexey Romanenko <ar...@gmail.com> wrote:
>
>
>> Well, perhaps it’s a bug (I’ll try to check it on my side later).
>>
>> Can you try to annotate InboundData class with “@DefaultCoder(AvroCoder.class)” for example and see if it will work?
>>
>>> On 23 Mar 2023, at 15:13, Juan Cuzmar jcuzmar@protonmail.com wrote:
>>>
>>> Alexey yes! here it is:
>>>
>>> import lombok.*;
>>> import lombok.extern.jackson.Jacksonized;
>>>
>>> import java.io.Serializable;
>>> import java.util.List;
>>> import java.util.Map;
>>>
>>> @Jacksonized
>>> @Builder
>>> @Getter
>>> @Value
>>> @EqualsAndHashCode
>>> @ToString
>>> public class InboundData implements Serializable {
>>>
>>> String payload;
>>> Map<String, String> attributes;
>>> List<Store> stores;
>>> }
>>>
>>> ------- Original Message -------
>>> On Thursday, March 23rd, 2023 at 8:14 AM, Alexey Romanenko aromanenko.dev@gmail.com wrote:
>>>
>>>> Could you share a class declaration of your InboundData class? Is it just a POJO?
>>>>
>>>> —
>>>> Alexey
>>>>
>>>>> On 23 Mar 2023, at 08:16, Juan Cuzmar jcuzmar@protonmail.com wrote:
>>>>>
>>>>> Hello all,
>>>>>
>>>>> I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
>>>>>
>>>>> Here's a snippet of my code:
>>>>>
>>>>> .apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
>>>>> .withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
>>>>> ...
>>>>> .withRowMapper(resultSet -> {
>>>>> // Mapping logic here
>>>>> }).withOutputParallelization(false)
>>>>> ).setCoder(SerializableCoder.of(InboundData.class))
>>>>>
>>>>> When I run the code, I encounter the following error:
>>>>>
>>>>> java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
>>>>> ...
>>>>>
>>>>> I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
>>>>>
>>>>> If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
>>>>>
>>>>> Thank you for your time and assistance.
>>>>>
>>>>> Best regards!
Re: Recommended way to set coder for JdbcIO with Apache Beam
Posted by Juan Cuzmar <jc...@protonmail.com>.
unfortunately didn't work.
I added the @DefaultCoder as you told me and i removed the .withCoder from my JdbcIO pipe and showed the same error:
java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform. Provide a coder via withCoder, or ensure that one can be inferred from the provided RowMapper.
Juan Cuzmar.
------- Original Message -------
On Thursday, March 23rd, 2023 at 2:17 PM, Alexey Romanenko <ar...@gmail.com> wrote:
> Well, perhaps it’s a bug (I’ll try to check it on my side later).
>
> Can you try to annotate InboundData class with “@DefaultCoder(AvroCoder.class)” for example and see if it will work?
>
> > On 23 Mar 2023, at 15:13, Juan Cuzmar jcuzmar@protonmail.com wrote:
> >
> > Alexey yes! here it is:
> >
> > import lombok.*;
> > import lombok.extern.jackson.Jacksonized;
> >
> > import java.io.Serializable;
> > import java.util.List;
> > import java.util.Map;
> >
> > @Jacksonized
> > @Builder
> > @Getter
> > @Value
> > @EqualsAndHashCode
> > @ToString
> > public class InboundData implements Serializable {
> >
> > String payload;
> > Map<String, String> attributes;
> > List<Store> stores;
> > }
> >
> > ------- Original Message -------
> > On Thursday, March 23rd, 2023 at 8:14 AM, Alexey Romanenko aromanenko.dev@gmail.com wrote:
> >
> > > Could you share a class declaration of your InboundData class? Is it just a POJO?
> > >
> > > —
> > > Alexey
> > >
> > > > On 23 Mar 2023, at 08:16, Juan Cuzmar jcuzmar@protonmail.com wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
> > > >
> > > > Here's a snippet of my code:
> > > >
> > > > .apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
> > > > .withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
> > > > ...
> > > > .withRowMapper(resultSet -> {
> > > > // Mapping logic here
> > > > }).withOutputParallelization(false)
> > > > ).setCoder(SerializableCoder.of(InboundData.class))
> > > >
> > > > When I run the code, I encounter the following error:
> > > >
> > > > java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
> > > > ...
> > > >
> > > > I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
> > > >
> > > > If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
> > > >
> > > > Thank you for your time and assistance.
> > > >
> > > > Best regards!
Re: Recommended way to set coder for JdbcIO with Apache Beam
Posted by Alexey Romanenko <ar...@gmail.com>.
Well, perhaps it’s a bug (I’ll try to check it on my side later).
Can you try to annotate InboundData class with “@DefaultCoder(AvroCoder.class)” for example and see if it will work?
> On 23 Mar 2023, at 15:13, Juan Cuzmar <jc...@protonmail.com> wrote:
>
> Alexey yes! here it is:
>
>
> import lombok.*;
> import lombok.extern.jackson.Jacksonized;
>
> import java.io.Serializable;
> import java.util.List;
> import java.util.Map;
>
> @Jacksonized
> @Builder
> @Getter
> @Value
> @EqualsAndHashCode
> @ToString
> public class InboundData implements Serializable {
>
> String payload;
> Map<String, String> attributes;
> List<Store> stores;
> }
>
>
> ------- Original Message -------
> On Thursday, March 23rd, 2023 at 8:14 AM, Alexey Romanenko <ar...@gmail.com> wrote:
>
>
>> Could you share a class declaration of your InboundData class? Is it just a POJO?
>>
>> —
>> Alexey
>>
>>> On 23 Mar 2023, at 08:16, Juan Cuzmar jcuzmar@protonmail.com wrote:
>>>
>>> Hello all,
>>>
>>> I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
>>>
>>> Here's a snippet of my code:
>>>
>>> .apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
>>> .withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
>>> ...
>>> .withRowMapper(resultSet -> {
>>> // Mapping logic here
>>> }).withOutputParallelization(false)
>>> ).setCoder(SerializableCoder.of(InboundData.class))
>>>
>>> When I run the code, I encounter the following error:
>>>
>>> java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
>>> ...
>>>
>>> I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
>>>
>>> If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
>>>
>>> Thank you for your time and assistance.
>>>
>>> Best regards!
Re: Recommended way to set coder for JdbcIO with Apache Beam
Posted by Juan Cuzmar <jc...@protonmail.com>.
Alexey yes! here it is:
import lombok.*;
import lombok.extern.jackson.Jacksonized;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
@Jacksonized
@Builder
@Getter
@Value
@EqualsAndHashCode
@ToString
public class InboundData implements Serializable {
String payload;
Map<String, String> attributes;
List<Store> stores;
}
------- Original Message -------
On Thursday, March 23rd, 2023 at 8:14 AM, Alexey Romanenko <ar...@gmail.com> wrote:
> Could you share a class declaration of your InboundData class? Is it just a POJO?
>
> —
> Alexey
>
> > On 23 Mar 2023, at 08:16, Juan Cuzmar jcuzmar@protonmail.com wrote:
> >
> > Hello all,
> >
> > I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
> >
> > Here's a snippet of my code:
> >
> > .apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
> > .withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
> > ...
> > .withRowMapper(resultSet -> {
> > // Mapping logic here
> > }).withOutputParallelization(false)
> > ).setCoder(SerializableCoder.of(InboundData.class))
> >
> > When I run the code, I encounter the following error:
> >
> > java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
> > ...
> >
> > I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
> >
> > If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
> >
> > Thank you for your time and assistance.
> >
> > Best regards!
Re: Recommended way to set coder for JdbcIO with Apache Beam
Posted by Alexey Romanenko <ar...@gmail.com>.
Could you share a class declaration of your InboundData class? Is it just a POJO?
—
Alexey
> On 23 Mar 2023, at 08:16, Juan Cuzmar <jc...@protonmail.com> wrote:
>
> Hello all,
>
> I hope this message finds you well. I am currently working with Apache Beam's JdbcIO and need some guidance regarding setting a coder for the input data without resorting to the deprecated withCoder method. I've been trying to resolve this issue and would appreciate any insights or recommendations from the community.
>
> Here's a snippet of my code:
>
> .apply("Inserting", JdbcIO.<PubsubMessage, InboundData>readAll()
> .withDataSourceProviderFn(DataSourceProvider.of(dbConfig))
> ...
> .withRowMapper(resultSet -> {
> // Mapping logic here
> }).withOutputParallelization(false)
> ).setCoder(SerializableCoder.of(InboundData.class))
>
> When I run the code, I encounter the following error:
>
>
> java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll() transform.
> ...
>
>
> I understand that the withCoder method is marked as deprecated and should be avoided. Therefore, I would like to know the recommended way to set the coder for JdbcIO, considering the deprecated status of withCoder.
>
> If there are any specific coding practices or conventions that I should follow when posting code snippets or emphasizing certain aspects, please let me know, and I'll be sure to adhere to them in the future.
>
> Thank you for your time and assistance.
>
> Best regards!