[D] Help- please! (hop)

["jdoit18950 (via GitHub)" <gi...@apache.org>]

GitHub user jdoit18950 created a discussion: Help- please!

Hi everyone. I have been trying to figure it out for a week and I am still stuck. I need to create a dynamic approach to incrementally load about 1000 tables from a SQL Server database as the source and Postgres as the target. Hop is currently being run in a hosted environment due to strict firewall. I also have limited access to sql server. Any ideas or support would be greatly appreciated!

GitHub link: https://github.com/apache/hop/discussions/5092

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] Help- please! (hop)

["bamaer (via GitHub)" <gi...@apache.org>]

GitHub user bamaer added a comment to the discussion: Help- please!

If your database tables have the same structure in SQL Server and Postgres, you could just create a pipeline that accepts the table name or input query, target table name etc as a parameter, and use that in your table input and table output transform. In a second pipeline, add the list of 1000 table to process, and call the first pipeline through a pipeline executor, providing the parameters from the pipeline executor to your first pipeline. 
If your use case is more complex than that, you could take a look at [Metadata Injection](https://hop.apache.org/manual/latest/pipeline/metadata-injection.html). 

GitHub link: https://github.com/apache/hop/discussions/5092#discussioncomment-12660843

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] Help- please! (hop)

["dmainou (via GitHub)" <gi...@apache.org>]

GitHub user dmainou added a comment to the discussion: Help- please!

Assuming you know the following. 
- the list of source tables 
- the corresponding target tables
- the method of determining the delta.  E.g. the row id, the date or some other key. 

Use a table input step to get the data. 
Use either a table output step to load the data or,  if the volume is large,  use the postgresql bulk loader.

As Bart said,  if your process is more complex, you could query the list of tables, indices,  views,  etc.

Determine what needs created. 

Then get the columns and types for the tables, map those types to postgres types and build table ddls and create the tables. 

Etc. Follow the ping pong ball. 

GitHub link: https://github.com/apache/hop/discussions/5092#discussioncomment-12664883

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org