You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 21:58:26 UTC
[GitHub] [beam] kennknowles opened a new issue, #18995: datastoreio doesn't work with entities containing keys
kennknowles opened a new issue, #18995:
URL: https://github.com/apache/beam/issues/18995
If I trie to map over an entity struct that contains a *datastore.Key field, I get the error:
panic: type value main.Foo must be a concrete type
It looks like this is because typex.IsConcrete doesn't support pointers.
Imported from Jira [BEAM-5200](https://issues.apache.org/jira/browse/BEAM-5200). Original Jira may contain additional context.
Reported by: ribrdb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] datastoreio doesn't work with entities containing keys [beam]
Posted by "remko (via GitHub)" <gi...@apache.org>.
remko commented on issue #18995:
URL: https://github.com/apache/beam/issues/18995#issuecomment-3288686558
This is probably related to ticket #18846 (which uses datastore.Key as an example)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] datastoreio doesn't work with entities containing keys [beam]
Posted by "remko-bw (via GitHub)" <gi...@apache.org>.
remko-bw commented on issue #18995:
URL: https://github.com/apache/beam/issues/18995#issuecomment-3288686199
This is probably related to ticket #18846 (which uses datastore.Key as an example)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] datastoreio doesn't work with entities containing keys [beam]
Posted by "remko (via GitHub)" <gi...@apache.org>.
remko commented on issue #18995:
URL: https://github.com/apache/beam/issues/18995#issuecomment-3308727364
I think I have found a (tedious) workaround for this issue, and that is to register a custom coder for every struct type that has a `*datastore.Key`. That stops the binary from crashing, and seems to work. This is obviously not very convenient.
@kennknowles Can the priority of this issue be re-evaluated? The datastore connector isn't really usable without support for Key fields in structs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] datastoreio doesn't work with entities containing keys [beam]
Posted by "remko (via GitHub)" <gi...@apache.org>.
remko commented on issue #18995:
URL: https://github.com/apache/beam/issues/18995#issuecomment-3255391557
The JIRA ticket says:
```
This issue was marked "stale-P2" and has not received a public comment in 14 days. It is now automatically moved to P3. If you are still affected by it, you can comment and move it back to P2.
```
I am affected by this, so should this become a P2 again? I'm unable to use the datastore connector because almost all my kinds have embedded Keys in them, and I get a stack overflow when starting the pipeline.
This is my program:
```
package main
import (
"context"
"reflect"
"cloud.google.com/go/datastore"
"github.com/apache/beam/sdks/v2/go/pkg/beam"
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime"
"github.com/apache/beam/sdks/v2/go/pkg/beam/io/datastoreio"
"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism"
_ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/gcs"
_ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/local"
)
type Foo struct {
Bar *datastore.Key `datastore:"bar"`
}
var fooKey string
func init() {
fooKey = runtime.RegisterType(reflect.TypeOf((*Foo)(nil)).Elem())
}
func main() {
beam.Init()
p := beam.NewPipeline()
s := p.Root()
datastoreio.Read(s, "myproject", "Foo", 256, reflect.TypeOf(Foo{}), fooKey)
prism.Execute(context.Background(), p)
}
```
I attached the output of the command.
[stack.txt](https://github.com/user-attachments/files/22146666/stack.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org