You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by fo...@apache.org on 2025/09/09 18:16:11 UTC
(iceberg) branch main updated: Remove BladePipe due to broken links (#14033)
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new 720ef99720 Remove BladePipe due to broken links (#14033)
720ef99720 is described below
commit 720ef99720a1c59e4670db983c951243dffc4f3e
Author: Fokko Driesprong <fo...@apache.org>
AuthorDate: Tue Sep 9 20:16:06 2025 +0200
Remove BladePipe due to broken links (#14033)
It looks like the website of BladePipe is unreachable. Let's remove
it for now, and we can add it again when the website and images
are fixed.
---
site/docs/integrations/bladepipe.md | 119 ------------------------------------
site/mkdocs.yml | 1 -
site/nav.yml | 1 -
3 files changed, 121 deletions(-)
diff --git a/site/docs/integrations/bladepipe.md b/site/docs/integrations/bladepipe.md
deleted file mode 100644
index 73831ad3ca..0000000000
--- a/site/docs/integrations/bladepipe.md
+++ /dev/null
@@ -1,119 +0,0 @@
----
-title: "BladePipe"
----
-<!--
- - Licensed to the Apache Software Foundation (ASF) under one or more
- - contributor license agreements. See the NOTICE file distributed with
- - this work for additional information regarding copyright ownership.
- - The ASF licenses this file to You under the Apache License, Version 2.0
- - (the "License"); you may not use this file except in compliance with
- - the License. You may obtain a copy of the License at
- -
- - http://www.apache.org/licenses/LICENSE-2.0
- -
- - Unless required by applicable law or agreed to in writing, software
- - distributed under the License is distributed on an "AS IS" BASIS,
- - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- - See the License for the specific language governing permissions and
- - limitations under the License.
- -->
-
-# BladePipe
-
-[BladePipe](https://www.bladepipe.com/) is a real-time end-to-end data integration tool, offering 40+ out-of-the-box connectors for analytics or AI. It allows to move data faster and easier than ever, with ultra-low latency less than 3 seconds. It provides a one-stop data movement solution, including schema evolution, data migration and sync, verification and correction, monitoring and alerting.
-
-## Supported Sources
-Now BladePipe supports data integration to Iceberg from the following sources:
-
-- MySQL/MariaDB/AuroraMySQL
-- Oracle
-- PostgreSQL
-- SQL Server
-- Kafka
-
-More sources are to be supported.
-
-## Supported Catalogs and Storage
-BladePipe currently supports 3 catalogs and 2 object storage:
-
-- AWS Glue + AWS S3
-- Nessie + MinIO / AWS S3
-- REST Catalog + MinIO / AWS S3
-
-
-## Getting Started
-In this article, we will show how to load data from MySQL (self-hosted) to Iceberg (AWS Glue + S3).
-
-### 1. Download and Run BladePipe
-Follow the instructions in [Install Worker (Docker)](https://doc.bladepipe.com/productOP/byoc/installation/install_worker_docker) or [Install Worker (Binary)](https://doc.bladepipe.com/productOP/byoc/installation/install_worker_binary) to download and install a BladePipe Worker.
-
-**Note**: Alternatively, you can choose to deploy and run [BladePipe Enterprise](https://doc.bladepipe.com/productOP/onPremise/installation/install_all_in_one_binary).
-
-### 2. Add DataSources
-
-1. Log in to the [BladePipe Cloud](https://cloud.bladepipe.com).
-2. Click **DataSource** > **Add DataSource**.
-3. Add a MySQL instance and an Iceberg instance. For Iceberg, fill in the following content (replace `<...>` with your values):
- - **Address**: Fill in the AWS Glue endpoint.
-
- ```text
- glue.<aws_glue_region_code>.amazonaws.com
- ```
-
- - **Version**: Leave as default.
- - **Description**: Fill in meaningful words to help identify it.
- - **Extra Info**:
- - **httpsEnabled**: Enable it to set the value as true.
- - **catalogName**: Enter a meaningful name, such as glue_<biz_name>_catalog.
- - **catalogType**: Fill in `GLUE`.
- - **catalogWarehouse**: The place where metadata and files are stored, such as s3://<biz_name>_iceberg.
- - **catalogProps**:
-
- ```json
- {
- "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
- "s3.endpoint": "https://s3.<aws_s3_region_code>.amazonaws.com",
- "s3.access-key-id": "<aws_s3_iam_user_access_key>",
- "s3.secret-access-key": "<aws_s3_iam_user_secret_key>",
- "s3.path-style-access": "true",
- "client.region": "<aws_s3_region>",
- "client.credentials-provider.glue.access-key-id": "<aws_glue_iam_user_access_key>",
- "client.credentials-provider.glue.secret-access-key": "<aws_glue_iam_user_secret_key>",
- "client.credentials-provider": "com.amazonaws.glue.catalog.credentials.GlueAwsCredentialsProvider"
- }
- ```
-
- 
- See [Add an Iceberg DataSource](https://doc.bladepipe.com/dataMigrationAndSync/datasource_func/Iceberg/props_for_iceberg_ds) for more details.
-
-### 3. Create a DataJob
-1. Go to **DataJob** > [**Create DataJob**](https://doc.bladepipe.com/operation/job_manage/create_job/create_full_incre_task).
-2. Select the source and target DataSources, and click **Test Connection** for both. Here's the recommended Iceberg structure configuration:
- ```json
- {
- "format-version": "2",
- "parquet.compression": "snappy",
- "iceberg.write.format": "parquet",
- "write.metadata.delete-after-commit.enabled": "true",
- "write.metadata.previous-versions-max": "3",
- "write.update.mode": "merge-on-read",
- "write.delete.mode": "merge-on-read",
- "write.merge.mode": "merge-on-read",
- "write.distribution-mode": "hash",
- "write.object-storage.enabled": "true",
- "write.spark.accept-any-schema": "true"
- }
- ```
- 
-
-3. Select **Incremental** for DataJob Type, together with the **Full Data** option.
- 
-
-4. Select the tables to be replicated.
- 
-
-5. Select the columns to be replicated.
- 
-
-6. Confirm the DataJob creation, and start to run the DataJob.
- 
diff --git a/site/mkdocs.yml b/site/mkdocs.yml
index bce1feea87..f29cb67df6 100644
--- a/site/mkdocs.yml
+++ b/site/mkdocs.yml
@@ -56,7 +56,6 @@ plugins:
- redirects:
redirect_maps:
'docs/nightly/amoro.md': 'integrations/amoro.md'
- 'docs/nightly/bladepipe.md': 'integrations/bladepipe.md'
'docs/nightly/daft.md': 'integrations/daft.md'
'docs/nightly/risingwave.md': 'integrations/risingwave.md'
diff --git a/site/nav.yml b/site/nav.yml
index b479a6586b..855c4eb888 100644
--- a/site/nav.yml
+++ b/site/nav.yml
@@ -63,7 +63,6 @@ nav:
- Apache Amoro: integrations/amoro.md
- Apache Doris: https://doris.apache.org/docs/dev/lakehouse/catalogs/iceberg-catalog
- Apache Druid: https://druid.apache.org/docs/latest/development/extensions-contrib/iceberg/
- - BladePipe: integrations/bladepipe.md
- ClickHouse: https://clickhouse.com/docs/en/engines/table-engines/integrations/iceberg
- Daft: integrations/daft.md
- Databend: https://docs.databend.com/guides/access-data-lake/iceberg