You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/01/17 04:59:45 UTC

[I] [Feature][Connector] Add BigQuery Sink Connector [seatunnel]

davidzollo opened a new issue, #10355:
URL: https://github.com/apache/seatunnel/issues/10355

   ## Background
   
   BigQuery is Google Cloud's serverless, highly scalable, and cost-effective multi-cloud data warehouse. It is widely used by enterprises globally for data analytics, business intelligence, and machine learning workloads.
   
   Currently, SeaTunnel lacks native support for BigQuery as a sink, which limits its ability to integrate with the Google Cloud ecosystem efficiently.
   
   ## Motivation
   
   - **High Market Demand**: BigQuery is a core service in Google Cloud Platform (GCP) with a large enterprise customer base
   - **Cloud-Native Architecture**: While JDBC drivers exist for BigQuery, they provide poor performance and limited functionality compared to native SDK
   - **Advanced Features**: Native connector can support:
     - Streaming inserts for real-time data ingestion
     - Table partitioning and clustering
     - Nested and repeated fields (STRUCT, ARRAY)
     - Integration with Cloud Storage for efficient bulk loading
     - Schema auto-detection and evolution
   
   ## Proposed Solution
   
   Implement a dedicated BigQuery Sink connector using the Google Cloud Java SDK with the following capabilities:
   
   ### Core Features
   1. **Multiple Write Modes**
      - Batch loading via Cloud Storage (for high throughput)
      - Streaming inserts (for low latency)
      - Support for both append and overwrite modes
   
   2. **Schema Management**
      - Automatic schema creation and evolution
      - Support for complex data types (STRUCT, ARRAY, TIMESTAMP, GEOGRAPHY)
      - Schema validation and type mapping
   
   3. **Performance Optimization**
      - Configurable batch size and flush interval
      - Parallel writes with configurable parallelism
      - Retry mechanism with exponential backoff
   
   4. **Data Quality**
      - Row-level error handling
      - Dead letter queue for failed records
      - Data validation before insertion
   
   ### Configuration Example
   ```hocon
   sink {
     BigQuery {
       project = "my-gcp-project"
       dataset = "my_dataset"
       table = "my_table"
       
       # Authentication
       credentials_file = "/path/to/service-account.json"
       
       # Write configuration
       write_mode = "streaming" # or "batch"
       create_disposition = "CREATE_IF_NEEDED"
       write_disposition = "WRITE_APPEND" # or "WRITE_TRUNCATE"
       
       # Performance tuning
       max_batch_size = 1000
       max_batch_bytes = 10485760 # 10MB
       flush_interval_ms = 5000
       
       # Schema options
       auto_create_table = true
       schema_update_options = ["ALLOW_FIELD_ADDITION"]
     }
   }
   ```
   
   ## Expected Benefits
   
   1. **Better Performance**: Native SDK provides 10-100x better performance than JDBC for large-scale data ingestion
   2. **Cost Efficiency**: Optimized bulk loading via Cloud Storage reduces costs
   3. **Feature Completeness**: Access to BigQuery-specific features like streaming inserts and schema evolution
   4. **Enterprise Adoption**: Enables SeaTunnel to compete in GCP-based data integration scenarios
   
   ## Technical Considerations
   
   - **Dependencies**: Add `google-cloud-bigquery` SDK
   - **Authentication**: Support service account JSON, application default credentials, and workload identity
   - **Error Handling**: Implement robust retry logic and error reporting
   - **Testing**: Require integration tests with BigQuery emulator or test project
   
   ## References
   
   - [BigQuery Java Client Library](https://cloud.google.com/java/docs/reference/google-cloud-bigquery/latest/overview)
   - [BigQuery Storage Write API](https://cloud.google.com/bigquery/docs/write-api)
   - [Best Practices for Loading Data](https://cloud.google.com/bigquery/docs/best-practices-performance-input)
   
   ## Community Impact
   
   This connector will:
   - Expand SeaTunnel's cloud ecosystem support
   - Attract GCP users to the SeaTunnel community
   - Enable enterprises to build modern data pipelines on Google Cloud
   
   ---
   
   **Priority**: High  
   **Estimated Effort**: Medium  
   **Target Release**: 2.3.14 or 3.0.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add BigQuery Sink Connector [seatunnel]

Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on issue #10355:
URL: https://github.com/apache/seatunnel/issues/10355#issuecomment-3762950829

   Could you assign this issue to me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add BigQuery Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #10355: [Feature][Connector] Add BigQuery Sink Connector
URL: https://github.com/apache/seatunnel/issues/10355


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add BigQuery Sink Connector [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10355:
URL: https://github.com/apache/seatunnel/issues/10355#issuecomment-3763704089

   <!-- code-pr-reviewer -->
   Thanks for proposing this feature. SeaTunnel currently lacks a native BigQuery sink connector; users must use the JDBC connector with limited performance.
   
   A new `connector-bigquery` module would be valuable. We already have patterns to follow:
   - **Multi-table support**: `seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java` (implements `SupportMultiTableSinkWriter`)
   - **Google Cloud integration**: `seatunnel-connectors-v2/connector-google-firestore/pom.xml` (google-cloud SDK dependencies + Service Account auth)
   - **Per-table config pattern**: `seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/config/JdbcSourceTableConfig.java` (table-configs array structure)
   
   **Key questions**:
   1. Should this support CDC → BigQuery with dynamic multi-table routing?
   2. Streaming Insert API vs batch load via GCS — which takes priority?
   3. Is schema evolution with partition/clustering DDL required?
   
   Contributions are welcome. A starting point would be creating `seatunnel-connectors-v2/connector-bigquery` with `BigQuerySinkFactory`/`BigQuerySinkWriter`, leveraging the `google-cloud-bigquery` SDK and Firestore authentication patterns.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add BigQuery Sink Connector [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10355:
URL: https://github.com/apache/seatunnel/issues/10355#issuecomment-3764646832

   <!-- code-pr-reviewer -->
   Thanks for proposing this feature. Apache SeaTunnel does not currently have a BigQuery Sink connector.
   
   For implementation, you can reference:
   - `connector-google-sheets` and `connector-google-firestore` for Google Cloud integration patterns
   - Typical sink connectors under `connector-*` directories for structure and multi-table support patterns
   - `plugin-mapping.properties` and `connector-pom.xml` for connector registration and dependency management
   
   Before we proceed, could you clarify:
   1. Should this support batch mode, streaming mode, or both?
   2. What write patterns are needed? (append/overwrite/merge)
   3. Are there specific BigQuery APIs or client libraries you intend to use?
   
   This will help provide a more concrete starting point aligned with your requirements.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add BigQuery Sink Connector [seatunnel]

Posted by "ashutoshk314 (via GitHub)" <gi...@apache.org>.
ashutoshk314 commented on issue #10355:
URL: https://github.com/apache/seatunnel/issues/10355#issuecomment-3793724372

   Hello,
   
   I’m interested in contributing to this issue. Please let me know if issue is still open.
   
   Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org