You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "RonnelOvalles (via GitHub)" <gi...@apache.org> on 2023/01/30 19:53:38 UTC
[GitHub] [pulsar] RonnelOvalles opened a new issue, #19363: [Bug] pulsar-sink nested avro
RonnelOvalles opened a new issue, #19363:
URL: https://github.com/apache/pulsar/issues/19363
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Pulsar 2.9.3
### Minimal reproduce step
Create Producer
`import pulsar
from pulsar.schema import *
import datetime as dt
from faker import Faker
from pulsar.schema import AvroSchema
class Payload(Record):
campaigId = Integer(required=True)
class MetaData(Record):
user_id = Integer(required=True)
username = String(required=True)
payload = Payload(required=True)
def produceToTopic(producer):
fake=Faker()
producer.send(MetaData(user_id=fake.random_int(),
username=fake.name(),
payload=Payload(campaigId=fake.random_int())
))
client = pulsar.Client('pulsar://localhost:6650')
producer = client.create_producer(
topic='persistent://public/default/test5',
schema=AvroSchema(MetaData))
t = dt.datetime.now()
maxTime=3600
currentTime=0
while currentTime<maxTime:
delta = dt.datetime.now() - t
#messages are sent every 2 senconds
if delta.seconds >= 2:
currentTime=currentTime+delta.seconds
t = dt.datetime.now()
produceToTopic(producer)
client.close()`
Create table in postgres
`CREATE TABLE test (
user_id INT PRIMARY KEY,
username VARCHAR ( 300 ) NOT NULL,
payload VARCHAR ( 300 ) NOT NULL
);`
Sink to Postgres DB
`./bin/pulsar-admin sinks create \
--archive ./connectors/pulsar-io-jdbc-postgres-2.9.3.nar \
--inputs public/default/test5 \
--name test-sink-v8 \
--sink-config-file ./connectors/campaignCreatedSink1.yaml \
--parallelism 1`
### What did you expect to see?
Expected data to be saved accoridng to schema
### What did you see instead?
Shows error, it looks like pulsar only supports basic avro schema deserilization for pulsar-admin sink but this is no where to be found in the documentation.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #19363: [Bug] pulsar-sink nested avro
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun closed issue #19363: [Bug] pulsar-sink nested avro
URL: https://github.com/apache/pulsar/issues/19363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on issue #19363: [Bug] pulsar-sink nested avro
Posted by "nicoloboschi (via GitHub)" <gi...@apache.org>.
nicoloboschi commented on issue #19363:
URL: https://github.com/apache/pulsar/issues/19363#issuecomment-1412410250
@RonnelOvalles could you provide the error and reformat the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #19363: [Bug] pulsar-sink nested avro
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #19363:
URL: https://github.com/apache/pulsar/issues/19363#issuecomment-1501489462
The repo link is broken and 2.9.x is end of life.
Closing as stale. But if it's still relevant to maintained version (ATM 2.10 and 2.11), please open a new issue with error stack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RonnelOvalles commented on issue #19363: [Bug] pulsar-sink nested avro
Posted by "RonnelOvalles (via GitHub)" <gi...@apache.org>.
RonnelOvalles commented on issue #19363:
URL: https://github.com/apache/pulsar/issues/19363#issuecomment-1422917153
Sure,
You can find it all here,
https://gitlab.com/rontech/pulsarsink-nestedavro.git
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #19363: [Bug] pulsar-sink nested avro
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19363:
URL: https://github.com/apache/pulsar/issues/19363#issuecomment-1464780313
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org