You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/30 15:27:31 UTC

[GitHub] [airflow] galzak opened a new issue #10648: dbapi_hook target_fields not working

galzak opened a new issue #10648:
URL: https://github.com/apache/airflow/issues/10648


   I have a table in Postgres with the below columns:
   	re_ads_snapshot_id serial NOT NULL,
   	ad_id int8 NOT NULL,
   	origin_table_id int4 NULL
   
   My ETL suppose to insert data into this table but only to ad_id, origin_table_id.
   the query is “select ad_id, origin_table_id from….”
   When I use target fields it’s still inserting the Data into the first two: re_ads_snapshot_id and ad_id instead of skipping the first serial column.
   
   Does anyone know how can i skip the first column? since it's serial number..
   
   That is my mySQL to Postgres operator code:
   
   ```
   class MySQLToPostgresOperator(BaseOperator):
       @apply_defaults
       def __init__(
               self,
               sql,
               pg_table,
               src_mysql_conn_id,
               dest_postgres_conn_id,
               target_fields=None,
               *args, **kwargs):
           super(MySQLToPostgresOperator, self).__init__(*args, **kwargs)
           self.sql = sql
           self.pg_table = pg_table
           self.src_mysql_conn_id = src_mysql_conn_id
           self.dest_postgres_conn_id = dest_postgres_conn_id
           self.target_fields = target_fields
       def execute(self, context):
           log.info("Executing: {}".format(str(self.sql)))
           src_mysql = MySqlHook(mysql_conn_id=self.src_mysql_conn_id)
           dest_pg = PostgresHook(postgres_conn_id=self.dest_postgres_conn_id)
           log.info("Transferring MySQL query results into Postgres database.")
           conn = src_mysql.get_conn()
           cursor = conn.cursor()
           cursor.execute(self.sql)
           log.info("Inserting rows into Postgres")
           dest_pg.insert_rows(table=self.pg_table, rows=cursor, target_fields=self.target_fields)
           log.info("Done.")
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] galzak closed issue #10648: dbapi_hook target_fields not working

Posted by GitBox <gi...@apache.org>.
galzak closed issue #10648:
URL: https://github.com/apache/airflow/issues/10648


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] galzak commented on issue #10648: dbapi_hook target_fields not working

Posted by GitBox <gi...@apache.org>.
galzak commented on issue #10648:
URL: https://github.com/apache/airflow/issues/10648#issuecomment-684517001


   Thank you.
   instead of using:
   ```
           conn = src_mysql.get_conn()
           cursor = conn.cursor()
           cursor.execute(self.sql)
   ```
   
   I had to change:
   `        records = src_mysql.get_records(self.sql)
   `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #10648: dbapi_hook target_fields not working

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10648:
URL: https://github.com/apache/airflow/issues/10648#issuecomment-683921278


   Commented about it on Slack.
   Your implementation seems to be a private case of [GenericTransfer](https://github.com/apache/airflow/blob/master/airflow/operators/generic_transfer.py) so you should be able to use that operator for your ETL.
   
   I think for now this issue can be closed.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #10648: dbapi_hook target_fields not working

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10648:
URL: https://github.com/apache/airflow/issues/10648#issuecomment-683433932


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org