You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Charulata Shelar <ch...@gmail.com> on 2023/02/09 09:15:35 UTC
Help needed to join pubsub data with big query data using apache beam python sdk
Hi Team,
I have a problem statement to read the streaming data from pubsub topic
(PubSubTopic1) and join the data with Bigquery table (BQTable1) using
dataflow and then save the result into new Bigquery table (ResultBQTable)
*PubSubTopic1*: has ItemID, UnitPrice
*BQTable1*: has ItemID, ItemName, OfferPrice
*ResultBQTable*: need to have columns: ItemID, ItemName, UnitPrice,
OfferPrice, TotalCost
I am able to create Dataflow job using 'DataFlow SQL Workbench' but this is
one time, I can not automate this, hence I want to write python code using
apache beam ask and dataflow sdk to automate this so that it can be shared
with anyone to implement same thing.
I have been following to:
https://stackoverflow.com/questions/75364002/join-pubsub-data-with-bigquery-data-and-then-save-result-into-bigquery-using-dat
I can read the records from Bigquery: image_1.png
and when I uncomment line 4, I get the below error: image_2.png
Better and optimal approaches are all welcome.
Regards,
Charu Shelar