You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ahmet Altay <al...@google.com> on 2022/02/28 06:05:08 UTC
Fwd: Beam Website Feedback
Forwarding to user@ list. And adding relevant folks who could reply @Pablo
Estrada <pa...@google.com> @Robert Burke <re...@google.com>
(Folks, if you respond please reply to the list and Abe together. Abe is
having problems with replying back to the user lists.
---------- Forwarded message ---------
From: 阿部智大 <ab...@videomarket.co.jp>
Date: Sun, Feb 27, 2022 at 10:00 PM
Subject: Re: Beam Website Feedback
To: Ahmet Altay <al...@google.com>
Thank you for your reply.
The content I tried to send is as follows.
ーーー
In the following steps, we are creating a process to write the data on
MySQL to the BQ table.
1. Get data from MySQL with "SELECT * FROM table" with Query function of
databaseio package
2. Write to the BQ table specified by the Write function of the bigquery
package
However, there is a difference between the number of data that can be
actually acquired by MySQL and the number that is inserted into the BQ
table.
Is this a bug?
I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
results.
2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
> Hi,
>
> What is your question?
>
> Ahmet
>
> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
> wrote:
>
>> Hi Beam Users,
>>
>>
>> I'm a Japanese developer.
>>
>> I have a question about using the Go SDK, so I want to send an email to
>> user@beam.apache.org, but I can't because MAILER-DAEMON is returned.
>> Could you tell me how to send it?
>>
>> Thanks & Regards
>
>
Re: Beam Website Feedback
Posted by Ahmet Altay <al...@google.com>.
On Sun, Feb 27, 2022 at 11:20 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Hello Abe,
>
> Can you check if you are subscribed to the user mailing list, it seems
> you are to dev@ maybe the issue is the missing user@ suscription.
> You can do this by sending an email to:
> user-subscribe@beam.apache.org
>
> Ahmet and the others, it might be a good idea to mention the user@
> mailing list in the contribute doc too, I just quickly checked and did
> not see it there.
> https://beam.apache.org/contribute/
It is the first thing in that list: "ask or answer questions on
user@beam.apache.org or stackoverflow". Do you have something else in mind?
>
>
> Regards,
> Ismaël
>
> On Mon, Feb 28, 2022 at 7:05 AM Ahmet Altay <al...@google.com> wrote:
> >
> > Forwarding to user@ list. And adding relevant folks who could reply
> @Pablo Estrada @Robert Burke
> >
> > (Folks, if you respond please reply to the list and Abe together. Abe is
> having problems with replying back to the user lists.
> >
> > ---------- Forwarded message ---------
> > From: 阿部智大 <ab...@videomarket.co.jp>
> > Date: Sun, Feb 27, 2022 at 10:00 PM
> > Subject: Re: Beam Website Feedback
> > To: Ahmet Altay <al...@google.com>
> >
> >
> > Thank you for your reply.
> >
> > The content I tried to send is as follows.
> >
> >
> > ーーー
> > In the following steps, we are creating a process to write the data on
> MySQL to the BQ table.
> >
> > 1. Get data from MySQL with "SELECT * FROM table" with Query function of
> databaseio package
> > 2. Write to the BQ table specified by the Write function of the bigquery
> package
> >
> > However, there is a difference between the number of data that can be
> actually acquired by MySQL and the number that is inserted into the BQ
> table.
> > Is this a bug?
> >
> > I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
> results.
> >
> > 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
> >>
> >> Hi,
> >>
> >> What is your question?
> >>
> >> Ahmet
> >>
> >> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
> wrote:
> >>>
> >>> Hi Beam Users,
> >>>
> >>>
> >>> I'm a Japanese developer.
> >>>
> >>> I have a question about using the Go SDK, so I want to send an email
> to user@beam.apache.org, but I can't because MAILER-DAEMON is returned.
> >>> Could you tell me how to send it?
> >>>
> >>> Thanks & Regards
>
Re: Beam Website Feedback
Posted by Ismaël Mejía <ie...@gmail.com>.
Hello Abe,
Can you check if you are subscribed to the user mailing list, it seems
you are to dev@ maybe the issue is the missing user@ suscription.
You can do this by sending an email to:
user-subscribe@beam.apache.org
Ahmet and the others, it might be a good idea to mention the user@
mailing list in the contribute doc too, I just quickly checked and did
not see it there.
https://beam.apache.org/contribute/
Regards,
Ismaël
On Mon, Feb 28, 2022 at 7:05 AM Ahmet Altay <al...@google.com> wrote:
>
> Forwarding to user@ list. And adding relevant folks who could reply @Pablo Estrada @Robert Burke
>
> (Folks, if you respond please reply to the list and Abe together. Abe is having problems with replying back to the user lists.
>
> ---------- Forwarded message ---------
> From: 阿部智大 <ab...@videomarket.co.jp>
> Date: Sun, Feb 27, 2022 at 10:00 PM
> Subject: Re: Beam Website Feedback
> To: Ahmet Altay <al...@google.com>
>
>
> Thank you for your reply.
>
> The content I tried to send is as follows.
>
>
> ーーー
> In the following steps, we are creating a process to write the data on MySQL to the BQ table.
>
> 1. Get data from MySQL with "SELECT * FROM table" with Query function of databaseio package
> 2. Write to the BQ table specified by the Write function of the bigquery package
>
> However, there is a difference between the number of data that can be actually acquired by MySQL and the number that is inserted into the BQ table.
> Is this a bug?
>
> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same results.
>
> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>
>> Hi,
>>
>> What is your question?
>>
>> Ahmet
>>
>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp> wrote:
>>>
>>> Hi Beam Users,
>>>
>>>
>>> I'm a Japanese developer.
>>>
>>> I have a question about using the Go SDK, so I want to send an email to user@beam.apache.org, but I can't because MAILER-DAEMON is returned.
>>> Could you tell me how to send it?
>>>
>>> Thanks & Regards
Re: Beam Website Feedback
Posted by 阿部智大 <ab...@videomarket.co.jp>.
Sorry.
I upgraded the version to 2.7.0 and still get the same error in jdbcio.
2022年3月15日(火) 4:58 Kyle Weaver <kc...@google.com>:
> > When I access the repositories / orgapachebeam-1254 that I told you, the
> 404 not found page is displayed.
>
> Beam 2.37.0 is now released, so you can use it the normal way.
>
> On Sun, Mar 6, 2022 at 6:12 PM 阿部智大 <ab...@videomarket.co.jp>
> wrote:
>
>> Hello Robert,
>>
>> I'm sorry for the late reply.
>> Also, thank you for your reply.
>>
>> I forgot to send it to user as well.
>> It should now be added to the thread.
>>
>> When I access the repositories / orgapachebeam-1254 that I told you, the
>> 404 not found page is displayed.
>>
>> Does this mean that I had another solution by the time I answered?
>>
>> Thanks & Regards
>> Abe
>>
>> 2022年3月3日(木) 3:55 Robert Burke <re...@google.com>:
>>
>>> Turns out this is exactly possible, as the Jar download paths have a
>>> common suffix
>>> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
>>> convenience override.
>>> It's not a 2.37.0 release blocker due to the newness of the auto
>>> download and start functionality, it's not a regression, but the
>>> convenience override should be handled for 2.38.0
>>> Rather than another flag that will clutter a user's help, we'll be
>>> having a variable or a Set function on the appropriate package for users to
>>> call (or populated electively with their own flag), and the
>>> https://beam.apache.org/blog/validate-beam-release/ will be updated to
>>> address this ability for Cross Language transforms.
>>>
>>> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
>>>
>>>> In replying to an off thread question from Abe, I realized that my
>>>> suggestion doesn't work since the SDK doesn't know how to source/grab the
>>>> RC versions of the Cross Language Jars. They end up getting a 404 response
>>>> as a result.
>>>>
>>>> > panic: tried cross-language for
>>>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
>>>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
>>>> and failed
>>>> > expanding external transform
>>>> > received non 200 response code, got 404
>>>>
>>>> I'll see if we can't improve that kind of error message for future
>>>> releases.
>>>>
>>>> What is the best way for a user to access RC jars?
>>>>
>>>> I guess one would need to download them from the staging repo for the
>>>> RC:
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>>>>
>>>> In this case, it's likely one of the jars in here:
>>>>
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
>>>> And specifically
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>>>>
>>>>
>>>> One should be able to start up the service with the jar, then using the
>>>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
>>>> the call, so it queries the service you started up instead.
>>>>
>>>> [1]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>>>>
>>>> I've asked the team to look into simplifying this. We should in
>>>> principle should get it to work against a specific RC repo with a flag,
>>>> rather than forcing the full manual startup.
>>>>
>>>> Cheers,
>>>> Robert Burke
>>>> Beam Go Busybody
>>>>
>>>>
>>>>
>>>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>>>>
>>>>> Hi Abe!
>>>>>
>>>>> Thank you for your interest in the Go SDK! I have two responses for
>>>>> you.
>>>>>
>>>>> First, the current native databaseio and bigqueryio transform in the
>>>>> Go SDK aren't production ready at this time. They support scaling, which
>>>>> may lead to the limits that you're seeing. Outside of the native TextIO
>>>>> with the ReadSDF* calls, those old transforms don't scale well. They were
>>>>> originally community contributions, but additional maintenance or
>>>>> improvements haven't occurred in some time. This is a known issue and will
>>>>> be resolved in time.
>>>>>
>>>>> Second, you are in luck!
>>>>> In the in progress v2.37.0-RC2 release, there are several Cross
>>>>> Language IOs
>>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>>>>> to databases, including a JDBCIO
>>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>>>>> and a BigQueryIO
>>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>>>>> These should automatically incorporate the Java versions of these
>>>>> transforms into your pipeline, just call and configure the "wrapper" Read
>>>>> and Write calls to use them in your pipeline. Using these does require
>>>>> that a recent edition of Java be installed on your system, but should
>>>>> otherwise not require manual set up of Java jars and similar.
>>>>>
>>>>> Cross language transforms require a runner that supports them, which
>>>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>>>>> Portable runner. The default Go Direct runner doesn't support them at this
>>>>> time, and will fail.
>>>>>
>>>>> Robert Burke
>>>>> Beam Go Busybody
>>>>>
>>>>> [1]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>>>>> [2]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>>>>> [3]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>>>>
>>>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>>>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>>>>
>>>>>> (Folks, if you respond please reply to the list and Abe together. Abe
>>>>>> is having problems with replying back to the user lists.
>>>>>>
>>>>>> ---------- Forwarded message ---------
>>>>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>>>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>>>>> Subject: Re: Beam Website Feedback
>>>>>> To: Ahmet Altay <al...@google.com>
>>>>>>
>>>>>>
>>>>>> Thank you for your reply.
>>>>>>
>>>>>> The content I tried to send is as follows.
>>>>>>
>>>>>>
>>>>>> ーーー
>>>>>> In the following steps, we are creating a process to write the data
>>>>>> on MySQL to the BQ table.
>>>>>>
>>>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function
>>>>>> of databaseio package
>>>>>> 2. Write to the BQ table specified by the Write function of the
>>>>>> bigquery package
>>>>>>
>>>>>> However, there is a difference between the number of data that can be
>>>>>> actually acquired by MySQL and the number that is inserted into the BQ
>>>>>> table.
>>>>>> Is this a bug?
>>>>>>
>>>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the
>>>>>> same results.
>>>>>>
>>>>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> What is your question?
>>>>>>>
>>>>>>> Ahmet
>>>>>>>
>>>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Beam Users,
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm a Japanese developer.
>>>>>>>>
>>>>>>>> I have a question about using the Go SDK, so I want to send an
>>>>>>>> email to user@beam.apache.org, but I can't because MAILER-DAEMON
>>>>>>>> is returned.
>>>>>>>> Could you tell me how to send it?
>>>>>>>>
>>>>>>>> Thanks & Regards
>>>>>>>
>>>>>>>
Re: Beam Website Feedback
Posted by Kyle Weaver <kc...@google.com>.
> When I access the repositories / orgapachebeam-1254 that I told you, the
404 not found page is displayed.
Beam 2.37.0 is now released, so you can use it the normal way.
On Sun, Mar 6, 2022 at 6:12 PM 阿部智大 <ab...@videomarket.co.jp> wrote:
> Hello Robert,
>
> I'm sorry for the late reply.
> Also, thank you for your reply.
>
> I forgot to send it to user as well.
> It should now be added to the thread.
>
> When I access the repositories / orgapachebeam-1254 that I told you, the
> 404 not found page is displayed.
>
> Does this mean that I had another solution by the time I answered?
>
> Thanks & Regards
> Abe
>
> 2022年3月3日(木) 3:55 Robert Burke <re...@google.com>:
>
>> Turns out this is exactly possible, as the Jar download paths have a
>> common suffix
>> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
>> convenience override.
>> It's not a 2.37.0 release blocker due to the newness of the auto download
>> and start functionality, it's not a regression, but the convenience
>> override should be handled for 2.38.0
>> Rather than another flag that will clutter a user's help, we'll be having
>> a variable or a Set function on the appropriate package for users to call
>> (or populated electively with their own flag), and the
>> https://beam.apache.org/blog/validate-beam-release/ will be updated to
>> address this ability for Cross Language transforms.
>>
>> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
>>
>>> In replying to an off thread question from Abe, I realized that my
>>> suggestion doesn't work since the SDK doesn't know how to source/grab the
>>> RC versions of the Cross Language Jars. They end up getting a 404 response
>>> as a result.
>>>
>>> > panic: tried cross-language for
>>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
>>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
>>> and failed
>>> > expanding external transform
>>> > received non 200 response code, got 404
>>>
>>> I'll see if we can't improve that kind of error message for future
>>> releases.
>>>
>>> What is the best way for a user to access RC jars?
>>>
>>> I guess one would need to download them from the staging repo for the
>>> RC:
>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>>>
>>> In this case, it's likely one of the jars in here:
>>>
>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
>>> And specifically
>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>>>
>>>
>>> One should be able to start up the service with the jar, then using the
>>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
>>> the call, so it queries the service you started up instead.
>>>
>>> [1]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>>>
>>> I've asked the team to look into simplifying this. We should in
>>> principle should get it to work against a specific RC repo with a flag,
>>> rather than forcing the full manual startup.
>>>
>>> Cheers,
>>> Robert Burke
>>> Beam Go Busybody
>>>
>>>
>>>
>>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>>>
>>>> Hi Abe!
>>>>
>>>> Thank you for your interest in the Go SDK! I have two responses for you.
>>>>
>>>> First, the current native databaseio and bigqueryio transform in the Go
>>>> SDK aren't production ready at this time. They support scaling, which may
>>>> lead to the limits that you're seeing. Outside of the native TextIO with
>>>> the ReadSDF* calls, those old transforms don't scale well. They were
>>>> originally community contributions, but additional maintenance or
>>>> improvements haven't occurred in some time. This is a known issue and will
>>>> be resolved in time.
>>>>
>>>> Second, you are in luck!
>>>> In the in progress v2.37.0-RC2 release, there are several Cross
>>>> Language IOs
>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>>>> to databases, including a JDBCIO
>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>>>> and a BigQueryIO
>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>>>> These should automatically incorporate the Java versions of these
>>>> transforms into your pipeline, just call and configure the "wrapper" Read
>>>> and Write calls to use them in your pipeline. Using these does require
>>>> that a recent edition of Java be installed on your system, but should
>>>> otherwise not require manual set up of Java jars and similar.
>>>>
>>>> Cross language transforms require a runner that supports them, which
>>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>>>> Portable runner. The default Go Direct runner doesn't support them at this
>>>> time, and will fail.
>>>>
>>>> Robert Burke
>>>> Beam Go Busybody
>>>>
>>>> [1]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>>>> [2]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>>>> [3]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>>>
>>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>>>
>>>>> (Folks, if you respond please reply to the list and Abe together. Abe
>>>>> is having problems with replying back to the user lists.
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>>>> Subject: Re: Beam Website Feedback
>>>>> To: Ahmet Altay <al...@google.com>
>>>>>
>>>>>
>>>>> Thank you for your reply.
>>>>>
>>>>> The content I tried to send is as follows.
>>>>>
>>>>>
>>>>> ーーー
>>>>> In the following steps, we are creating a process to write the data on
>>>>> MySQL to the BQ table.
>>>>>
>>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function
>>>>> of databaseio package
>>>>> 2. Write to the BQ table specified by the Write function of the
>>>>> bigquery package
>>>>>
>>>>> However, there is a difference between the number of data that can be
>>>>> actually acquired by MySQL and the number that is inserted into the BQ
>>>>> table.
>>>>> Is this a bug?
>>>>>
>>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>>>>> results.
>>>>>
>>>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What is your question?
>>>>>>
>>>>>> Ahmet
>>>>>>
>>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Beam Users,
>>>>>>>
>>>>>>>
>>>>>>> I'm a Japanese developer.
>>>>>>>
>>>>>>> I have a question about using the Go SDK, so I want to send an email
>>>>>>> to user@beam.apache.org, but I can't because MAILER-DAEMON is
>>>>>>> returned.
>>>>>>> Could you tell me how to send it?
>>>>>>>
>>>>>>> Thanks & Regards
>>>>>>
>>>>>>
Re: Beam Website Feedback
Posted by 阿部智大 <ab...@videomarket.co.jp>.
Hello Robert,
I'm sorry for the late reply.
Also, thank you for your reply.
I forgot to send it to user as well.
It should now be added to the thread.
When I access the repositories / orgapachebeam-1254 that I told you, the
404 not found page is displayed.
Does this mean that I had another solution by the time I answered?
Thanks & Regards
Abe
2022年3月3日(木) 3:55 Robert Burke <re...@google.com>:
> Turns out this is exactly possible, as the Jar download paths have a
> common suffix
> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
> convenience override.
> It's not a 2.37.0 release blocker due to the newness of the auto download
> and start functionality, it's not a regression, but the convenience
> override should be handled for 2.38.0
> Rather than another flag that will clutter a user's help, we'll be having
> a variable or a Set function on the appropriate package for users to call
> (or populated electively with their own flag), and the
> https://beam.apache.org/blog/validate-beam-release/ will be updated to
> address this ability for Cross Language transforms.
>
> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
>
>> In replying to an off thread question from Abe, I realized that my
>> suggestion doesn't work since the SDK doesn't know how to source/grab the
>> RC versions of the Cross Language Jars. They end up getting a 404 response
>> as a result.
>>
>> > panic: tried cross-language for
>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
>> and failed
>> > expanding external transform
>> > received non 200 response code, got 404
>>
>> I'll see if we can't improve that kind of error message for future
>> releases.
>>
>> What is the best way for a user to access RC jars?
>>
>> I guess one would need to download them from the staging repo for the
>> RC:
>> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>>
>> In this case, it's likely one of the jars in here:
>>
>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
>> And specifically
>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>>
>>
>> One should be able to start up the service with the jar, then using the
>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
>> the call, so it queries the service you started up instead.
>>
>> [1]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>>
>> I've asked the team to look into simplifying this. We should in principle
>> should get it to work against a specific RC repo with a flag, rather than
>> forcing the full manual startup.
>>
>> Cheers,
>> Robert Burke
>> Beam Go Busybody
>>
>>
>>
>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>>
>>> Hi Abe!
>>>
>>> Thank you for your interest in the Go SDK! I have two responses for you.
>>>
>>> First, the current native databaseio and bigqueryio transform in the Go
>>> SDK aren't production ready at this time. They support scaling, which may
>>> lead to the limits that you're seeing. Outside of the native TextIO with
>>> the ReadSDF* calls, those old transforms don't scale well. They were
>>> originally community contributions, but additional maintenance or
>>> improvements haven't occurred in some time. This is a known issue and will
>>> be resolved in time.
>>>
>>> Second, you are in luck!
>>> In the in progress v2.37.0-RC2 release, there are several Cross
>>> Language IOs
>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>>> to databases, including a JDBCIO
>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>>> and a BigQueryIO
>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>>> These should automatically incorporate the Java versions of these
>>> transforms into your pipeline, just call and configure the "wrapper" Read
>>> and Write calls to use them in your pipeline. Using these does require
>>> that a recent edition of Java be installed on your system, but should
>>> otherwise not require manual set up of Java jars and similar.
>>>
>>> Cross language transforms require a runner that supports them, which
>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>>> Portable runner. The default Go Direct runner doesn't support them at this
>>> time, and will fail.
>>>
>>> Robert Burke
>>> Beam Go Busybody
>>>
>>> [1]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>>> [2]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>>> [3]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>>
>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>>
>>>> (Folks, if you respond please reply to the list and Abe together. Abe
>>>> is having problems with replying back to the user lists.
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>>> Subject: Re: Beam Website Feedback
>>>> To: Ahmet Altay <al...@google.com>
>>>>
>>>>
>>>> Thank you for your reply.
>>>>
>>>> The content I tried to send is as follows.
>>>>
>>>>
>>>> ーーー
>>>> In the following steps, we are creating a process to write the data on
>>>> MySQL to the BQ table.
>>>>
>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function
>>>> of databaseio package
>>>> 2. Write to the BQ table specified by the Write function of the
>>>> bigquery package
>>>>
>>>> However, there is a difference between the number of data that can be
>>>> actually acquired by MySQL and the number that is inserted into the BQ
>>>> table.
>>>> Is this a bug?
>>>>
>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>>>> results.
>>>>
>>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> What is your question?
>>>>>
>>>>> Ahmet
>>>>>
>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>>> wrote:
>>>>>
>>>>>> Hi Beam Users,
>>>>>>
>>>>>>
>>>>>> I'm a Japanese developer.
>>>>>>
>>>>>> I have a question about using the Go SDK, so I want to send an email
>>>>>> to user@beam.apache.org, but I can't because MAILER-DAEMON is
>>>>>> returned.
>>>>>> Could you tell me how to send it?
>>>>>>
>>>>>> Thanks & Regards
>>>>>
>>>>>
Re: Beam Website Feedback
Posted by Robert Burke <re...@google.com>.
Turns out this is exactly possible, as the Jar download paths have a common
suffix
https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
convenience override.
It's not a 2.37.0 release blocker due to the newness of the auto download
and start functionality, it's not a regression, but the convenience
override should be handled for 2.38.0
Rather than another flag that will clutter a user's help, we'll be having a
variable or a Set function on the appropriate package for users to call (or
populated electively with their own flag), and the
https://beam.apache.org/blog/validate-beam-release/ will be updated to
address this ability for Cross Language transforms.
On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
> In replying to an off thread question from Abe, I realized that my
> suggestion doesn't work since the SDK doesn't know how to source/grab the
> RC versions of the Cross Language Jars. They end up getting a 404 response
> as a result.
>
> > panic: tried cross-language for
> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
> and failed
> > expanding external transform
> > received non 200 response code, got 404
>
> I'll see if we can't improve that kind of error message for future
> releases.
>
> What is the best way for a user to access RC jars?
>
> I guess one would need to download them from the staging repo for the RC:
> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>
> In this case, it's likely one of the jars in here:
>
> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
> And specifically
> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>
>
> One should be able to start up the service with the jar, then using the
> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
> the call, so it queries the service you started up instead.
>
> [1]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>
> I've asked the team to look into simplifying this. We should in principle
> should get it to work against a specific RC repo with a flag, rather than
> forcing the full manual startup.
>
> Cheers,
> Robert Burke
> Beam Go Busybody
>
>
>
> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>
>> Hi Abe!
>>
>> Thank you for your interest in the Go SDK! I have two responses for you.
>>
>> First, the current native databaseio and bigqueryio transform in the Go
>> SDK aren't production ready at this time. They support scaling, which may
>> lead to the limits that you're seeing. Outside of the native TextIO with
>> the ReadSDF* calls, those old transforms don't scale well. They were
>> originally community contributions, but additional maintenance or
>> improvements haven't occurred in some time. This is a known issue and will
>> be resolved in time.
>>
>> Second, you are in luck!
>> In the in progress v2.37.0-RC2 release, there are several Cross Language
>> IOs
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>> to databases, including a JDBCIO
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>> and a BigQueryIO
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>> These should automatically incorporate the Java versions of these
>> transforms into your pipeline, just call and configure the "wrapper" Read
>> and Write calls to use them in your pipeline. Using these does require
>> that a recent edition of Java be installed on your system, but should
>> otherwise not require manual set up of Java jars and similar.
>>
>> Cross language transforms require a runner that supports them, which
>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>> Portable runner. The default Go Direct runner doesn't support them at this
>> time, and will fail.
>>
>> Robert Burke
>> Beam Go Busybody
>>
>> [1]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>> [2]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>> [3]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>
>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>
>>> (Folks, if you respond please reply to the list and Abe together. Abe is
>>> having problems with replying back to the user lists.
>>>
>>> ---------- Forwarded message ---------
>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>> Subject: Re: Beam Website Feedback
>>> To: Ahmet Altay <al...@google.com>
>>>
>>>
>>> Thank you for your reply.
>>>
>>> The content I tried to send is as follows.
>>>
>>>
>>> ーーー
>>> In the following steps, we are creating a process to write the data on
>>> MySQL to the BQ table.
>>>
>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function of
>>> databaseio package
>>> 2. Write to the BQ table specified by the Write function of the bigquery
>>> package
>>>
>>> However, there is a difference between the number of data that can be
>>> actually acquired by MySQL and the number that is inserted into the BQ
>>> table.
>>> Is this a bug?
>>>
>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>>> results.
>>>
>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>
>>>> Hi,
>>>>
>>>> What is your question?
>>>>
>>>> Ahmet
>>>>
>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>> wrote:
>>>>
>>>>> Hi Beam Users,
>>>>>
>>>>>
>>>>> I'm a Japanese developer.
>>>>>
>>>>> I have a question about using the Go SDK, so I want to send an email
>>>>> to user@beam.apache.org, but I can't because MAILER-DAEMON is
>>>>> returned.
>>>>> Could you tell me how to send it?
>>>>>
>>>>> Thanks & Regards
>>>>
>>>>
Re: Beam Website Feedback
Posted by Robert Burke <re...@google.com>.
In replying to an off thread question from Abe, I realized that my
suggestion doesn't work since the SDK doesn't know how to source/grab the
RC versions of the Cross Language Jars. They end up getting a 404 response
as a result.
> panic: tried cross-language for
beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
and failed
> expanding external transform
> received non 200 response code, got 404
I'll see if we can't improve that kind of error message for future releases.
What is the best way for a user to access RC jars?
I guess one would need to download them from the staging repo for the RC:
https://repository.apache.org/content/repositories/orgapachebeam-1254/
In this case, it's likely one of the jars in here:
https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
And specifically
https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
One should be able to start up the service with the jar, then using the
address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
the call, so it queries the service you started up instead.
[1]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
I've asked the team to look into simplifying this. We should in principle
should get it to work against a specific RC repo with a flag, rather than
forcing the full manual startup.
Cheers,
Robert Burke
Beam Go Busybody
On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
> Hi Abe!
>
> Thank you for your interest in the Go SDK! I have two responses for you.
>
> First, the current native databaseio and bigqueryio transform in the Go
> SDK aren't production ready at this time. They support scaling, which may
> lead to the limits that you're seeing. Outside of the native TextIO with
> the ReadSDF* calls, those old transforms don't scale well. They were
> originally community contributions, but additional maintenance or
> improvements haven't occurred in some time. This is a known issue and will
> be resolved in time.
>
> Second, you are in luck!
> In the in progress v2.37.0-RC2 release, there are several Cross Language
> IOs
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
> to databases, including a JDBCIO
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
> and a BigQueryIO
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
> These should automatically incorporate the Java versions of these
> transforms into your pipeline, just call and configure the "wrapper" Read
> and Write calls to use them in your pipeline. Using these does require
> that a recent edition of Java be installed on your system, but should
> otherwise not require manual set up of Java jars and similar.
>
> Cross language transforms require a runner that supports them, which
> currently include Flink, Spark, Google Cloud Dataflow, and the Python
> Portable runner. The default Go Direct runner doesn't support them at this
> time, and will fail.
>
> Robert Burke
> Beam Go Busybody
>
> [1]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
> [2]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
> [3]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>
> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>
>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>
>> (Folks, if you respond please reply to the list and Abe together. Abe is
>> having problems with replying back to the user lists.
>>
>> ---------- Forwarded message ---------
>> From: 阿部智大 <ab...@videomarket.co.jp>
>> Date: Sun, Feb 27, 2022 at 10:00 PM
>> Subject: Re: Beam Website Feedback
>> To: Ahmet Altay <al...@google.com>
>>
>>
>> Thank you for your reply.
>>
>> The content I tried to send is as follows.
>>
>>
>> ーーー
>> In the following steps, we are creating a process to write the data on
>> MySQL to the BQ table.
>>
>> 1. Get data from MySQL with "SELECT * FROM table" with Query function of
>> databaseio package
>> 2. Write to the BQ table specified by the Write function of the bigquery
>> package
>>
>> However, there is a difference between the number of data that can be
>> actually acquired by MySQL and the number that is inserted into the BQ
>> table.
>> Is this a bug?
>>
>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>> results.
>>
>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>
>>> Hi,
>>>
>>> What is your question?
>>>
>>> Ahmet
>>>
>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>> wrote:
>>>
>>>> Hi Beam Users,
>>>>
>>>>
>>>> I'm a Japanese developer.
>>>>
>>>> I have a question about using the Go SDK, so I want to send an email to
>>>> user@beam.apache.org, but I can't because MAILER-DAEMON is returned.
>>>> Could you tell me how to send it?
>>>>
>>>> Thanks & Regards
>>>
>>>
Re: Beam Website Feedback
Posted by Robert Burke <re...@google.com>.
Hi Abe!
Thank you for your interest in the Go SDK! I have two responses for you.
First, the current native databaseio and bigqueryio transform in the Go
SDK aren't production ready at this time. They support scaling, which may
lead to the limits that you're seeing. Outside of the native TextIO with
the ReadSDF* calls, those old transforms don't scale well. They were
originally community contributions, but additional maintenance or
improvements haven't occurred in some time. This is a known issue and will
be resolved in time.
Second, you are in luck!
In the in progress v2.37.0-RC2 release, there are several Cross Language IOs
<https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/>
[1] related
to databases, including a JDBCIO
<https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio>
[2]
and a BigQueryIO
<https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio>
[3].
These should automatically incorporate the Java versions of these
transforms into your pipeline, just call and configure the "wrapper" Read
and Write calls to use them in your pipeline. Using these does require
that a recent edition of Java be installed on your system, but should
otherwise not require manual set up of Java jars and similar.
Cross language transforms require a runner that supports them, which
currently include Flink, Spark, Google Cloud Dataflow, and the Python
Portable runner. The default Go Direct runner doesn't support them at this
time, and will fail.
Robert Burke
Beam Go Busybody
[1]
https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
[2]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
[3]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>
> (Folks, if you respond please reply to the list and Abe together. Abe is
> having problems with replying back to the user lists.
>
> ---------- Forwarded message ---------
> From: 阿部智大 <ab...@videomarket.co.jp>
> Date: Sun, Feb 27, 2022 at 10:00 PM
> Subject: Re: Beam Website Feedback
> To: Ahmet Altay <al...@google.com>
>
>
> Thank you for your reply.
>
> The content I tried to send is as follows.
>
>
> ーーー
> In the following steps, we are creating a process to write the data on
> MySQL to the BQ table.
>
> 1. Get data from MySQL with "SELECT * FROM table" with Query function of
> databaseio package
> 2. Write to the BQ table specified by the Write function of the bigquery
> package
>
> However, there is a difference between the number of data that can be
> actually acquired by MySQL and the number that is inserted into the BQ
> table.
> Is this a bug?
>
> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
> results.
>
> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>
>> Hi,
>>
>> What is your question?
>>
>> Ahmet
>>
>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>> wrote:
>>
>>> Hi Beam Users,
>>>
>>>
>>> I'm a Japanese developer.
>>>
>>> I have a question about using the Go SDK, so I want to send an email to
>>> user@beam.apache.org, but I can't because MAILER-DAEMON is returned.
>>> Could you tell me how to send it?
>>>
>>> Thanks & Regards
>>
>>