You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "sroutweave (via GitHub)" <gi...@apache.org> on 2023/09/14 06:53:44 UTC
[GitHub] [beam] sroutweave opened a new issue, #28446: Postgres not connecting in apache beam with go sdk[Bug]:
sroutweave opened a new issue, #28446:
URL: https://github.com/apache/beam/issues/28446
### What happened?
We are trying to connect to postgres DB using apache beam go sdk . But it's throwing error each time. Tried both I/O connector databaseio & Jdbcio. Can we get any help on this please?
With databaseio connector
----------------------------
`caused by:
source failed
caused by:
DoFn[UID:4, PID:org.postgresql.Driver.Query/databaseio.queryFn, Name: github.com/apache/beam/sdks/v2/go/pkg/beam/io/databaseio.queryFn] failed:
failed to open database: org.postgresql.Driver
caused by:
sql: unknown driver "org.postgresql.Driver" (forgotten import?)
`
With Jdbcio connector
-----------------------
`panic: tried cross-language for beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService;org.postgresql:postgresql:42.3.3 and failed
expanding external transform
error in starting expansion service, StartService(): context deadline exceeded
`
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Postgres not connecting in apache beam with go sdk [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-5441977575
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johannaojeling commented on issue #28446: Postgres not connecting in apache beam with go sdk[Bug]:
Posted by "johannaojeling (via GitHub)" <gi...@apache.org>.
johannaojeling commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-1722209708
@sroutweave the native `databaseio` connector uses the `database/sql` package, which requires a [supported driver](https://github.com/golang/go/wiki/SQLDrivers) to be registered. For Postgres, you could use for instance [pq](https://github.com/lib/pq) or [pgx](https://github.com/jackc/pgx).
In your Go program, you will need to add a blank import to the selected driver package in order for its driver to get registered. The pq driver will be registered under the name `postgres` and pgx under `pgx`. You will need to pass the same name as the driver argument to the `databaseio.Read` function, instead of the JDBC name `org.postgresql.Driver` for the correct driver to be picked up.
Example program:
```go
package main
import (
"context"
"flag"
"log"
"reflect"
"github.com/apache/beam/sdks/v2/go/pkg/beam"
"github.com/apache/beam/sdks/v2/go/pkg/beam/io/databaseio"
"github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx"
"github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug"
_ "github.com/jackc/pgx/v5/stdlib" // driver: "pgx"
//_ "github.com/lib/pq" // driver: "postgres"
)
var (
driver = flag.String("driver", "pgx", "Database driver")
dsn = flag.String("dsn", "host=localhost port=5432 user=postgres password=pwd dbname=postgres sslmode=disable", "Database DSN")
table = flag.String("table", "users", "Database table")
)
type User struct {
ID string
Name string
Email string
}
func main() {
flag.Parse()
beam.Init()
p, s := beam.NewPipelineWithRoot()
rows := databaseio.Read(s, *driver, *dsn, *table, reflect.TypeOf(User{}))
debug.Print(s, rows)
ctx := context.Background()
if err := beamx.Run(ctx, p); err != nil {
log.Fatalf("Failed to execute job: %v", err)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Postgres not connecting in apache beam with go sdk [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28446: [Bug]: Postgres not connecting in apache beam with go sdk
URL: https://github.com/apache/beam/issues/28446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johannaojeling commented on issue #28446: Postgres not connecting in apache beam with go sdk[Bug]:
Posted by "johannaojeling (via GitHub)" <gi...@apache.org>.
johannaojeling commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-1724023680
> I am trying to connect cloudsql postgres db instance . And for that what should be the driver? I didn't find in the supported driver list. Can you please help in that? @johannaojeling
There are different ways to connect to a Cloud SQL database depending on your requirements but you should be able to use one of those mentioned Postgres drivers. Please see the [Google Cloud docs](https://cloud.google.com/sql/docs/postgres/connect-overview#go) for more information and code/configuration examples. The Go snippets use the pgx driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sroutweave commented on issue #28446: Postgres not connecting in apache beam with go sdk[Bug]:
Posted by "sroutweave (via GitHub)" <gi...@apache.org>.
sroutweave commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-1723111577
> ```go
> _ "github.com/jackc/pgx/v5/stdlib"
> ```
I am trying to connect cloudsql db instance . And for that what should be the driver? I didn't find in the supported driver list. Can you please help in that? @johannaojeling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sroutweave commented on issue #28446: Postgres not connecting in apache beam with go sdk[Bug]:
Posted by "sroutweave (via GitHub)" <gi...@apache.org>.
sroutweave commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-1726970474
> > I am trying to connect cloudsql postgres db instance . And for that what should be the driver? I didn't find in the supported driver list. Can you please help in that? @johannaojeling
>
> There are different ways to connect to a Cloud SQL database depending on your requirements but you should be able to use one of those mentioned Postgres drivers. Please see the [Google Cloud docs](https://cloud.google.com/sql/docs/postgres/connect-overview#go) for more information and code/configuration examples. The Go snippets use the pgx driver.
So when I tried with pgx & postgres driver for cloudsql postgres db instance to connect even though the tables are there it shows error as table doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johannaojeling commented on issue #28446: Postgres not connecting in apache beam with go sdk[Bug]:
Posted by "johannaojeling (via GitHub)" <gi...@apache.org>.
johannaojeling commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-1732930466
@sroutweave it would be helpful if you could tell how to reproduce the problem, including information such as:
- relevant code and versions
- how you are running the pipeline (locally, Dataflow, ..?)
- how your Cloud SQL instance is configured (e.g. private vs public IP) and how you usually connect to it
- logs showing the error message
- any other details that may be useful for people help you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Postgres not connecting in apache beam with go sdk [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28446:
URL: https://github.com/apache/beam/issues/28446#issuecomment-5104514418
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org