You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Robert Burke <re...@google.com> on 2022/03/02 17:59:54 UTC
Re: Beam Website Feedback
In replying to an off thread question from Abe, I realized that my
suggestion doesn't work since the SDK doesn't know how to source/grab the
RC versions of the Cross Language Jars. They end up getting a 404 response
as a result.
> panic: tried cross-language for
beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
and failed
> expanding external transform
> received non 200 response code, got 404
I'll see if we can't improve that kind of error message for future releases.
What is the best way for a user to access RC jars?
I guess one would need to download them from the staging repo for the RC:
https://repository.apache.org/content/repositories/orgapachebeam-1254/
In this case, it's likely one of the jars in here:
https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
And specifically
https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
One should be able to start up the service with the jar, then using the
address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
the call, so it queries the service you started up instead.
[1]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
I've asked the team to look into simplifying this. We should in principle
should get it to work against a specific RC repo with a flag, rather than
forcing the full manual startup.
Cheers,
Robert Burke
Beam Go Busybody
On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
> Hi Abe!
>
> Thank you for your interest in the Go SDK! I have two responses for you.
>
> First, the current native databaseio and bigqueryio transform in the Go
> SDK aren't production ready at this time. They support scaling, which may
> lead to the limits that you're seeing. Outside of the native TextIO with
> the ReadSDF* calls, those old transforms don't scale well. They were
> originally community contributions, but additional maintenance or
> improvements haven't occurred in some time. This is a known issue and will
> be resolved in time.
>
> Second, you are in luck!
> In the in progress v2.37.0-RC2 release, there are several Cross Language
> IOs
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
> to databases, including a JDBCIO
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
> and a BigQueryIO
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
> These should automatically incorporate the Java versions of these
> transforms into your pipeline, just call and configure the "wrapper" Read
> and Write calls to use them in your pipeline. Using these does require
> that a recent edition of Java be installed on your system, but should
> otherwise not require manual set up of Java jars and similar.
>
> Cross language transforms require a runner that supports them, which
> currently include Flink, Spark, Google Cloud Dataflow, and the Python
> Portable runner. The default Go Direct runner doesn't support them at this
> time, and will fail.
>
> Robert Burke
> Beam Go Busybody
>
> [1]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
> [2]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
> [3]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>
> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>
>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>
>> (Folks, if you respond please reply to the list and Abe together. Abe is
>> having problems with replying back to the user lists.
>>
>> ---------- Forwarded message ---------
>> From: 阿部智大 <ab...@videomarket.co.jp>
>> Date: Sun, Feb 27, 2022 at 10:00 PM
>> Subject: Re: Beam Website Feedback
>> To: Ahmet Altay <al...@google.com>
>>
>>
>> Thank you for your reply.
>>
>> The content I tried to send is as follows.
>>
>>
>> ーーー
>> In the following steps, we are creating a process to write the data on
>> MySQL to the BQ table.
>>
>> 1. Get data from MySQL with "SELECT * FROM table" with Query function of
>> databaseio package
>> 2. Write to the BQ table specified by the Write function of the bigquery
>> package
>>
>> However, there is a difference between the number of data that can be
>> actually acquired by MySQL and the number that is inserted into the BQ
>> table.
>> Is this a bug?
>>
>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>> results.
>>
>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>
>>> Hi,
>>>
>>> What is your question?
>>>
>>> Ahmet
>>>
>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>> wrote:
>>>
>>>> Hi Beam Users,
>>>>
>>>>
>>>> I'm a Japanese developer.
>>>>
>>>> I have a question about using the Go SDK, so I want to send an email to
>>>> user@beam.apache.org, but I can't because MAILER-DAEMON is returned.
>>>> Could you tell me how to send it?
>>>>
>>>> Thanks & Regards
>>>
>>>
Re: Beam Website Feedback
Posted by 阿部智大 <ab...@videomarket.co.jp>.
Sorry.
I upgraded the version to 2.7.0 and still get the same error in jdbcio.
2022年3月15日(火) 4:58 Kyle Weaver <kc...@google.com>:
> > When I access the repositories / orgapachebeam-1254 that I told you, the
> 404 not found page is displayed.
>
> Beam 2.37.0 is now released, so you can use it the normal way.
>
> On Sun, Mar 6, 2022 at 6:12 PM 阿部智大 <ab...@videomarket.co.jp>
> wrote:
>
>> Hello Robert,
>>
>> I'm sorry for the late reply.
>> Also, thank you for your reply.
>>
>> I forgot to send it to user as well.
>> It should now be added to the thread.
>>
>> When I access the repositories / orgapachebeam-1254 that I told you, the
>> 404 not found page is displayed.
>>
>> Does this mean that I had another solution by the time I answered?
>>
>> Thanks & Regards
>> Abe
>>
>> 2022年3月3日(木) 3:55 Robert Burke <re...@google.com>:
>>
>>> Turns out this is exactly possible, as the Jar download paths have a
>>> common suffix
>>> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
>>> convenience override.
>>> It's not a 2.37.0 release blocker due to the newness of the auto
>>> download and start functionality, it's not a regression, but the
>>> convenience override should be handled for 2.38.0
>>> Rather than another flag that will clutter a user's help, we'll be
>>> having a variable or a Set function on the appropriate package for users to
>>> call (or populated electively with their own flag), and the
>>> https://beam.apache.org/blog/validate-beam-release/ will be updated to
>>> address this ability for Cross Language transforms.
>>>
>>> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
>>>
>>>> In replying to an off thread question from Abe, I realized that my
>>>> suggestion doesn't work since the SDK doesn't know how to source/grab the
>>>> RC versions of the Cross Language Jars. They end up getting a 404 response
>>>> as a result.
>>>>
>>>> > panic: tried cross-language for
>>>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
>>>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
>>>> and failed
>>>> > expanding external transform
>>>> > received non 200 response code, got 404
>>>>
>>>> I'll see if we can't improve that kind of error message for future
>>>> releases.
>>>>
>>>> What is the best way for a user to access RC jars?
>>>>
>>>> I guess one would need to download them from the staging repo for the
>>>> RC:
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>>>>
>>>> In this case, it's likely one of the jars in here:
>>>>
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
>>>> And specifically
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>>>>
>>>>
>>>> One should be able to start up the service with the jar, then using the
>>>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
>>>> the call, so it queries the service you started up instead.
>>>>
>>>> [1]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>>>>
>>>> I've asked the team to look into simplifying this. We should in
>>>> principle should get it to work against a specific RC repo with a flag,
>>>> rather than forcing the full manual startup.
>>>>
>>>> Cheers,
>>>> Robert Burke
>>>> Beam Go Busybody
>>>>
>>>>
>>>>
>>>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>>>>
>>>>> Hi Abe!
>>>>>
>>>>> Thank you for your interest in the Go SDK! I have two responses for
>>>>> you.
>>>>>
>>>>> First, the current native databaseio and bigqueryio transform in the
>>>>> Go SDK aren't production ready at this time. They support scaling, which
>>>>> may lead to the limits that you're seeing. Outside of the native TextIO
>>>>> with the ReadSDF* calls, those old transforms don't scale well. They were
>>>>> originally community contributions, but additional maintenance or
>>>>> improvements haven't occurred in some time. This is a known issue and will
>>>>> be resolved in time.
>>>>>
>>>>> Second, you are in luck!
>>>>> In the in progress v2.37.0-RC2 release, there are several Cross
>>>>> Language IOs
>>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>>>>> to databases, including a JDBCIO
>>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>>>>> and a BigQueryIO
>>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>>>>> These should automatically incorporate the Java versions of these
>>>>> transforms into your pipeline, just call and configure the "wrapper" Read
>>>>> and Write calls to use them in your pipeline. Using these does require
>>>>> that a recent edition of Java be installed on your system, but should
>>>>> otherwise not require manual set up of Java jars and similar.
>>>>>
>>>>> Cross language transforms require a runner that supports them, which
>>>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>>>>> Portable runner. The default Go Direct runner doesn't support them at this
>>>>> time, and will fail.
>>>>>
>>>>> Robert Burke
>>>>> Beam Go Busybody
>>>>>
>>>>> [1]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>>>>> [2]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>>>>> [3]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>>>>
>>>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>>>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>>>>
>>>>>> (Folks, if you respond please reply to the list and Abe together. Abe
>>>>>> is having problems with replying back to the user lists.
>>>>>>
>>>>>> ---------- Forwarded message ---------
>>>>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>>>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>>>>> Subject: Re: Beam Website Feedback
>>>>>> To: Ahmet Altay <al...@google.com>
>>>>>>
>>>>>>
>>>>>> Thank you for your reply.
>>>>>>
>>>>>> The content I tried to send is as follows.
>>>>>>
>>>>>>
>>>>>> ーーー
>>>>>> In the following steps, we are creating a process to write the data
>>>>>> on MySQL to the BQ table.
>>>>>>
>>>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function
>>>>>> of databaseio package
>>>>>> 2. Write to the BQ table specified by the Write function of the
>>>>>> bigquery package
>>>>>>
>>>>>> However, there is a difference between the number of data that can be
>>>>>> actually acquired by MySQL and the number that is inserted into the BQ
>>>>>> table.
>>>>>> Is this a bug?
>>>>>>
>>>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the
>>>>>> same results.
>>>>>>
>>>>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> What is your question?
>>>>>>>
>>>>>>> Ahmet
>>>>>>>
>>>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Beam Users,
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm a Japanese developer.
>>>>>>>>
>>>>>>>> I have a question about using the Go SDK, so I want to send an
>>>>>>>> email to user@beam.apache.org, but I can't because MAILER-DAEMON
>>>>>>>> is returned.
>>>>>>>> Could you tell me how to send it?
>>>>>>>>
>>>>>>>> Thanks & Regards
>>>>>>>
>>>>>>>
Re: Beam Website Feedback
Posted by Kyle Weaver <kc...@google.com>.
> When I access the repositories / orgapachebeam-1254 that I told you, the
404 not found page is displayed.
Beam 2.37.0 is now released, so you can use it the normal way.
On Sun, Mar 6, 2022 at 6:12 PM 阿部智大 <ab...@videomarket.co.jp> wrote:
> Hello Robert,
>
> I'm sorry for the late reply.
> Also, thank you for your reply.
>
> I forgot to send it to user as well.
> It should now be added to the thread.
>
> When I access the repositories / orgapachebeam-1254 that I told you, the
> 404 not found page is displayed.
>
> Does this mean that I had another solution by the time I answered?
>
> Thanks & Regards
> Abe
>
> 2022年3月3日(木) 3:55 Robert Burke <re...@google.com>:
>
>> Turns out this is exactly possible, as the Jar download paths have a
>> common suffix
>> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
>> convenience override.
>> It's not a 2.37.0 release blocker due to the newness of the auto download
>> and start functionality, it's not a regression, but the convenience
>> override should be handled for 2.38.0
>> Rather than another flag that will clutter a user's help, we'll be having
>> a variable or a Set function on the appropriate package for users to call
>> (or populated electively with their own flag), and the
>> https://beam.apache.org/blog/validate-beam-release/ will be updated to
>> address this ability for Cross Language transforms.
>>
>> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
>>
>>> In replying to an off thread question from Abe, I realized that my
>>> suggestion doesn't work since the SDK doesn't know how to source/grab the
>>> RC versions of the Cross Language Jars. They end up getting a 404 response
>>> as a result.
>>>
>>> > panic: tried cross-language for
>>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
>>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
>>> and failed
>>> > expanding external transform
>>> > received non 200 response code, got 404
>>>
>>> I'll see if we can't improve that kind of error message for future
>>> releases.
>>>
>>> What is the best way for a user to access RC jars?
>>>
>>> I guess one would need to download them from the staging repo for the
>>> RC:
>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>>>
>>> In this case, it's likely one of the jars in here:
>>>
>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
>>> And specifically
>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>>>
>>>
>>> One should be able to start up the service with the jar, then using the
>>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
>>> the call, so it queries the service you started up instead.
>>>
>>> [1]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>>>
>>> I've asked the team to look into simplifying this. We should in
>>> principle should get it to work against a specific RC repo with a flag,
>>> rather than forcing the full manual startup.
>>>
>>> Cheers,
>>> Robert Burke
>>> Beam Go Busybody
>>>
>>>
>>>
>>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>>>
>>>> Hi Abe!
>>>>
>>>> Thank you for your interest in the Go SDK! I have two responses for you.
>>>>
>>>> First, the current native databaseio and bigqueryio transform in the Go
>>>> SDK aren't production ready at this time. They support scaling, which may
>>>> lead to the limits that you're seeing. Outside of the native TextIO with
>>>> the ReadSDF* calls, those old transforms don't scale well. They were
>>>> originally community contributions, but additional maintenance or
>>>> improvements haven't occurred in some time. This is a known issue and will
>>>> be resolved in time.
>>>>
>>>> Second, you are in luck!
>>>> In the in progress v2.37.0-RC2 release, there are several Cross
>>>> Language IOs
>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>>>> to databases, including a JDBCIO
>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>>>> and a BigQueryIO
>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>>>> These should automatically incorporate the Java versions of these
>>>> transforms into your pipeline, just call and configure the "wrapper" Read
>>>> and Write calls to use them in your pipeline. Using these does require
>>>> that a recent edition of Java be installed on your system, but should
>>>> otherwise not require manual set up of Java jars and similar.
>>>>
>>>> Cross language transforms require a runner that supports them, which
>>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>>>> Portable runner. The default Go Direct runner doesn't support them at this
>>>> time, and will fail.
>>>>
>>>> Robert Burke
>>>> Beam Go Busybody
>>>>
>>>> [1]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>>>> [2]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>>>> [3]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>>>
>>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>>>
>>>>> (Folks, if you respond please reply to the list and Abe together. Abe
>>>>> is having problems with replying back to the user lists.
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>>>> Subject: Re: Beam Website Feedback
>>>>> To: Ahmet Altay <al...@google.com>
>>>>>
>>>>>
>>>>> Thank you for your reply.
>>>>>
>>>>> The content I tried to send is as follows.
>>>>>
>>>>>
>>>>> ーーー
>>>>> In the following steps, we are creating a process to write the data on
>>>>> MySQL to the BQ table.
>>>>>
>>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function
>>>>> of databaseio package
>>>>> 2. Write to the BQ table specified by the Write function of the
>>>>> bigquery package
>>>>>
>>>>> However, there is a difference between the number of data that can be
>>>>> actually acquired by MySQL and the number that is inserted into the BQ
>>>>> table.
>>>>> Is this a bug?
>>>>>
>>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>>>>> results.
>>>>>
>>>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What is your question?
>>>>>>
>>>>>> Ahmet
>>>>>>
>>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Beam Users,
>>>>>>>
>>>>>>>
>>>>>>> I'm a Japanese developer.
>>>>>>>
>>>>>>> I have a question about using the Go SDK, so I want to send an email
>>>>>>> to user@beam.apache.org, but I can't because MAILER-DAEMON is
>>>>>>> returned.
>>>>>>> Could you tell me how to send it?
>>>>>>>
>>>>>>> Thanks & Regards
>>>>>>
>>>>>>
Re: Beam Website Feedback
Posted by 阿部智大 <ab...@videomarket.co.jp>.
Hello Robert,
I'm sorry for the late reply.
Also, thank you for your reply.
I forgot to send it to user as well.
It should now be added to the thread.
When I access the repositories / orgapachebeam-1254 that I told you, the
404 not found page is displayed.
Does this mean that I had another solution by the time I answered?
Thanks & Regards
Abe
2022年3月3日(木) 3:55 Robert Burke <re...@google.com>:
> Turns out this is exactly possible, as the Jar download paths have a
> common suffix
> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
> convenience override.
> It's not a 2.37.0 release blocker due to the newness of the auto download
> and start functionality, it's not a regression, but the convenience
> override should be handled for 2.38.0
> Rather than another flag that will clutter a user's help, we'll be having
> a variable or a Set function on the appropriate package for users to call
> (or populated electively with their own flag), and the
> https://beam.apache.org/blog/validate-beam-release/ will be updated to
> address this ability for Cross Language transforms.
>
> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
>
>> In replying to an off thread question from Abe, I realized that my
>> suggestion doesn't work since the SDK doesn't know how to source/grab the
>> RC versions of the Cross Language Jars. They end up getting a 404 response
>> as a result.
>>
>> > panic: tried cross-language for
>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
>> and failed
>> > expanding external transform
>> > received non 200 response code, got 404
>>
>> I'll see if we can't improve that kind of error message for future
>> releases.
>>
>> What is the best way for a user to access RC jars?
>>
>> I guess one would need to download them from the staging repo for the
>> RC:
>> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>>
>> In this case, it's likely one of the jars in here:
>>
>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
>> And specifically
>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>>
>>
>> One should be able to start up the service with the jar, then using the
>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
>> the call, so it queries the service you started up instead.
>>
>> [1]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>>
>> I've asked the team to look into simplifying this. We should in principle
>> should get it to work against a specific RC repo with a flag, rather than
>> forcing the full manual startup.
>>
>> Cheers,
>> Robert Burke
>> Beam Go Busybody
>>
>>
>>
>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>>
>>> Hi Abe!
>>>
>>> Thank you for your interest in the Go SDK! I have two responses for you.
>>>
>>> First, the current native databaseio and bigqueryio transform in the Go
>>> SDK aren't production ready at this time. They support scaling, which may
>>> lead to the limits that you're seeing. Outside of the native TextIO with
>>> the ReadSDF* calls, those old transforms don't scale well. They were
>>> originally community contributions, but additional maintenance or
>>> improvements haven't occurred in some time. This is a known issue and will
>>> be resolved in time.
>>>
>>> Second, you are in luck!
>>> In the in progress v2.37.0-RC2 release, there are several Cross
>>> Language IOs
>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>>> to databases, including a JDBCIO
>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>>> and a BigQueryIO
>>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>>> These should automatically incorporate the Java versions of these
>>> transforms into your pipeline, just call and configure the "wrapper" Read
>>> and Write calls to use them in your pipeline. Using these does require
>>> that a recent edition of Java be installed on your system, but should
>>> otherwise not require manual set up of Java jars and similar.
>>>
>>> Cross language transforms require a runner that supports them, which
>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>>> Portable runner. The default Go Direct runner doesn't support them at this
>>> time, and will fail.
>>>
>>> Robert Burke
>>> Beam Go Busybody
>>>
>>> [1]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>>> [2]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>>> [3]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>>
>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>>
>>>> (Folks, if you respond please reply to the list and Abe together. Abe
>>>> is having problems with replying back to the user lists.
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>>> Subject: Re: Beam Website Feedback
>>>> To: Ahmet Altay <al...@google.com>
>>>>
>>>>
>>>> Thank you for your reply.
>>>>
>>>> The content I tried to send is as follows.
>>>>
>>>>
>>>> ーーー
>>>> In the following steps, we are creating a process to write the data on
>>>> MySQL to the BQ table.
>>>>
>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function
>>>> of databaseio package
>>>> 2. Write to the BQ table specified by the Write function of the
>>>> bigquery package
>>>>
>>>> However, there is a difference between the number of data that can be
>>>> actually acquired by MySQL and the number that is inserted into the BQ
>>>> table.
>>>> Is this a bug?
>>>>
>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>>>> results.
>>>>
>>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> What is your question?
>>>>>
>>>>> Ahmet
>>>>>
>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>>> wrote:
>>>>>
>>>>>> Hi Beam Users,
>>>>>>
>>>>>>
>>>>>> I'm a Japanese developer.
>>>>>>
>>>>>> I have a question about using the Go SDK, so I want to send an email
>>>>>> to user@beam.apache.org, but I can't because MAILER-DAEMON is
>>>>>> returned.
>>>>>> Could you tell me how to send it?
>>>>>>
>>>>>> Thanks & Regards
>>>>>
>>>>>
Re: Beam Website Feedback
Posted by Robert Burke <re...@google.com>.
Turns out this is exactly possible, as the Jar download paths have a common
suffix
https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the
convenience override.
It's not a 2.37.0 release blocker due to the newness of the auto download
and start functionality, it's not a regression, but the convenience
override should be handled for 2.38.0
Rather than another flag that will clutter a user's help, we'll be having a
variable or a Set function on the appropriate package for users to call (or
populated electively with their own flag), and the
https://beam.apache.org/blog/validate-beam-release/ will be updated to
address this ability for Cross Language transforms.
On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <re...@google.com> wrote:
> In replying to an off thread question from Abe, I realized that my
> suggestion doesn't work since the SDK doesn't know how to source/grab the
> RC versions of the Cross Language Jars. They end up getting a 404 response
> as a result.
>
> > panic: tried cross-language for
> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against
> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService
> and failed
> > expanding external transform
> > received non 200 response code, got 404
>
> I'll see if we can't improve that kind of error message for future
> releases.
>
> What is the best way for a user to access RC jars?
>
> I guess one would need to download them from the staging repo for the RC:
> https://repository.apache.org/content/repositories/orgapachebeam-1254/
>
> In this case, it's likely one of the jars in here:
>
> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/
> And specifically
> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar
>
>
> One should be able to start up the service with the jar, then using the
> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to
> the call, so it queries the service you started up instead.
>
> [1]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead
>
> I've asked the team to look into simplifying this. We should in principle
> should get it to work against a specific RC repo with a flag, rather than
> forcing the full manual startup.
>
> Cheers,
> Robert Burke
> Beam Go Busybody
>
>
>
> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <re...@google.com> wrote:
>
>> Hi Abe!
>>
>> Thank you for your interest in the Go SDK! I have two responses for you.
>>
>> First, the current native databaseio and bigqueryio transform in the Go
>> SDK aren't production ready at this time. They support scaling, which may
>> lead to the limits that you're seeing. Outside of the native TextIO with
>> the ReadSDF* calls, those old transforms don't scale well. They were
>> originally community contributions, but additional maintenance or
>> improvements haven't occurred in some time. This is a known issue and will
>> be resolved in time.
>>
>> Second, you are in luck!
>> In the in progress v2.37.0-RC2 release, there are several Cross Language
>> IOs
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/> [1] related
>> to databases, including a JDBCIO
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio> [2]
>> and a BigQueryIO
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio> [3].
>> These should automatically incorporate the Java versions of these
>> transforms into your pipeline, just call and configure the "wrapper" Read
>> and Write calls to use them in your pipeline. Using these does require
>> that a recent edition of Java be installed on your system, but should
>> otherwise not require manual set up of Java jars and similar.
>>
>> Cross language transforms require a runner that supports them, which
>> currently include Flink, Spark, Google Cloud Dataflow, and the Python
>> Portable runner. The default Go Direct runner doesn't support them at this
>> time, and will fail.
>>
>> Robert Burke
>> Beam Go Busybody
>>
>> [1]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang@v2.37.0-RC2
>> [2]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/jdbcio
>> [3]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.37.0-RC2/go/pkg/beam/io/xlang/bigqueryio
>>
>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo
>>> Estrada <pa...@google.com> @Robert Burke <re...@google.com>
>>>
>>> (Folks, if you respond please reply to the list and Abe together. Abe is
>>> having problems with replying back to the user lists.
>>>
>>> ---------- Forwarded message ---------
>>> From: 阿部智大 <ab...@videomarket.co.jp>
>>> Date: Sun, Feb 27, 2022 at 10:00 PM
>>> Subject: Re: Beam Website Feedback
>>> To: Ahmet Altay <al...@google.com>
>>>
>>>
>>> Thank you for your reply.
>>>
>>> The content I tried to send is as follows.
>>>
>>>
>>> ーーー
>>> In the following steps, we are creating a process to write the data on
>>> MySQL to the BQ table.
>>>
>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function of
>>> databaseio package
>>> 2. Write to the BQ table specified by the Write function of the bigquery
>>> package
>>>
>>> However, there is a difference between the number of data that can be
>>> actually acquired by MySQL and the number that is inserted into the BQ
>>> table.
>>> Is this a bug?
>>>
>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same
>>> results.
>>>
>>> 2022年2月28日(月) 14:58 Ahmet Altay <al...@google.com>:
>>>
>>>> Hi,
>>>>
>>>> What is your question?
>>>>
>>>> Ahmet
>>>>
>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <ab...@videomarket.co.jp>
>>>> wrote:
>>>>
>>>>> Hi Beam Users,
>>>>>
>>>>>
>>>>> I'm a Japanese developer.
>>>>>
>>>>> I have a question about using the Go SDK, so I want to send an email
>>>>> to user@beam.apache.org, but I can't because MAILER-DAEMON is
>>>>> returned.
>>>>> Could you tell me how to send it?
>>>>>
>>>>> Thanks & Regards
>>>>
>>>>