You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Alex Li (Jira)" <ji...@apache.org> on 2019/08/29 00:21:00 UTC

[jira] [Created] (GOBBLIN-865) Add feature that enables PK-chunking in partition

Alex Li created GOBBLIN-865:
-------------------------------

             Summary: Add feature that enables PK-chunking in partition 
                 Key: GOBBLIN-865
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-865
             Project: Apache Gobblin
          Issue Type: Task
            Reporter: Alex Li


In SFDC(salesforce) connector, we have partitioning mechanisms to split a giant query to multiple sub queries. There are 3 mechanisms:
 * simple partition (equally split by time)
 * dynamic pre-partition (generate histogram and split by row numbers)
 * user specified partition (set up time range in job file)

However there are tables like Task and Contract are failing time to time to fetch full data.

We may want to utilize PK-chunking to partition the query.

 

The pk-chunking doc from SFDC - [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)