You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "czy006 (via GitHub)" <gi...@apache.org> on 2025/01/14 12:31:18 UTC
[PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
czy006 opened a new pull request, #11966:
URL: https://github.com/apache/iceberg/pull/11966
add apache amoro(incubating) describe with iceberg (#11965)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1915106856
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
Review Comment:
```suggestion
**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1922212099
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro MetaStore)** is Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS(Amoro MetaStore). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
+
+Apache Amoro supports all catalog types supported by Iceberg, including but not limited to: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, and so on.
Review Comment:
since this mentions **all**, does that also include the REST catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1959028139
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
+
+# Auto Self-optimizing
Review Comment:
These headers need to be at second level.
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
+
+# Auto Self-optimizing
Review Comment:
These section headers need to be at second level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1915973689
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. The new table
+format is used for this purpose. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS. AMS is responsible for
Review Comment:
Is AMS an abbreviation of Amoro Management Service? It would be helpful to mention the full name at least once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2594965072
Thanks for bringing this up. @czy006
Apache Amoro(incubating) is a Lakehouse management system and entered the Apache incubator in 2024.
It indeed helps some [Iceberg users](https://github.com/apache/amoro/issues/1853) automatically optimize their Iceberg tables, making it easier for them to use Iceberg effectively.
I am glad to see that we are trying to add some information about Amoro to the Iceberg site so that more Iceberg users can see it.
However, from my perspective, is adding a link good enough? Amoro should provide information on how to deal with Iceberg on its own documentation site and continuously update and optimize it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1915111401
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. The new table
+format is used for this purpose. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
Review Comment:
what does `The new table format is used for this purpose` mean exactly? It suggests to the reader that Amoro introduces a separate table format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "XBaith (via GitHub)" <gi...@apache.org>.
XBaith commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1924861778
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides [REST catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) service for all compute engines, which can also be combined with existing metadata services like hms.
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by [AMS](https://amoro.apache.org/docs/latest/#architecture). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
+
+Apache Amoro supports all catalog types supported by Iceberg, including common catalog: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, [Rest Catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) and other third-party catalog.
Review Comment:
```suggestion
Apache Amoro supports all catalog types supported by Iceberg, including common catalog: [REST](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog), Hadoop, Hive, Glue, JDBC, Nessie and other third-party catalog.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1959027222
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
Review Comment:
It reads strange having "pluggable" side-by-side with "Table Maintenance". Why not using the words from [Amoro README](https://github.com/apache/amoro/blob/master/README.md)
> Amoro brings pluggable and self-managed features for Lakehouse to provide out-of-the-box data warehouse experience ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1922456307
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides [REST catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) service for all compute engines, which can also be combined with existing metadata services like hms.
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS(Amoro MetaService). AMS is responsible for
Review Comment:
this says AMS stands for Amore MetaService but a few lines above AMS stands for Amoro Management Service. Which is correct now?
Ideally I think this readme should really only have a few pointers about what Amoro is and point to the official docs in order to avoid frequent updates to this site whenever some functionality/features change btw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1922443672
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro MetaStore)** is Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
Review Comment:
Thank you for correcting this incorrect description. AMS is a management service (sometime confuse by myself...) fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1921916823
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. The new table
+format is used for this purpose. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS. AMS is responsible for
Review Comment:
I don't see a mention that AMS stands for Amoro Meta Store in the text. Could you please add that to the first time where AMS is mentioned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2664728995
> @czy006 IMHO, the doc looks too verbose, and it might be better linking to the detailed description on the Amoro website while only keeping a brief introduction here for each feature.
I will adjust some doc descriptions to make them look more concise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2664570228
@czy006 IMHO, the doc looks too verbose, and it might be better linking to the detailed description on the Amoro website while only keeping a brief introduction here for each feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1922210587
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro MetaStore)** is Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
Review Comment:
Should this rather say `AMS (Amore MetaStore) is a management service that provides Lakehouse management features...`? Otherwise the first part of the sentence is confusing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1946120994
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by [AMS](https://amoro.apache.org/docs/latest/#architecture). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
+
+Apache Amoro supports all catalog types supported by Iceberg, including common catalog: [REST](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog), Hadoop, Hive, Glue, JDBC, Nessie and other third-party catalog.
+Amoro supports all storage types supported by Iceberg, including common store: Hadoop, S3, GCS, ECS, OSS, and so on.
+
+At the same time, we also provide a unique form based on Apache Iceberg, including mixed-Iceberg Format and mixed-Hive Format, so that you can quickly upgrade to the iceberg+hive Mixed table while compatible with the original Hive data
+
+## Mixed-Iceberg Format
+
+[Mixed-Iceberg Format](https://amoro.apache.org/docs/latest/self-optimizing/) Compared with Iceberg format, Mixed-Iceberg format provides more features:
Review Comment:
The link seems wrong. I guess https://amoro.apache.org/docs/latest/mixed-iceberg-format/ is the correct link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "XBaith (via GitHub)" <gi...@apache.org>.
XBaith commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1924851909
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides [REST catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) service for all compute engines, which can also be combined with existing metadata services like hms.
Review Comment:
```suggestion
**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides [REST catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
```
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides [REST catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) service for all compute engines, which can also be combined with existing metadata services like hms.
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by [AMS](https://amoro.apache.org/docs/latest/#architecture). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
+
+Apache Amoro supports all catalog types supported by Iceberg, including common catalog: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, [Rest Catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) and other third-party catalog.
+Amoro supports all storage types supported by Iceberg, including common object store: Hadoop, S3, GCS, ECS, OSS, and so on.
Review Comment:
I believe Hadoop is not an object store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1926966473
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides [REST catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
Review Comment:
```suggestion
**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
```
I feel like mentioning REST and linking to Iceberg here is rather confusing and probably not what's being intended here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1959031682
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by [AMS](https://amoro.apache.org/docs/latest/#architecture). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
Review Comment:
* I'd suggest adding a second-level header `Table Formats` with a third-level header for each format
* The first two paragraphs below are not related to "Format"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2641949029
Hi @Fokko, can help look amoro project description with iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1960866695
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
Review Comment:
@czy006 please check this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2591713421
> add apache amoro(incubating) describe with iceberg (#11965)
You could use `Fixes #11965` comment in the PR description so we can automatically close the issue.
https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1922212686
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro MetaStore)** is Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS(Amoro MetaStore). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
+
+Apache Amoro supports all catalog types supported by Iceberg, including but not limited to: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, and so on.
+Amoro supports all storage types supported by Iceberg, including but not limited to: Hadoop, S3, GCS, ECS, OSS, and so on.
+
+At the same time, we also provide a unique form based on iceberg, including mixed-Iceberg Format and mixed-Hive Format, so that you can quickly upgrade to the iceberg+hive Mixed table while compatible with the original Hive data
Review Comment:
```suggestion
At the same time, we also provide a unique form based on Apache Iceberg, including mixed-Iceberg Format and mixed-Hive Format, so that you can quickly upgrade to the iceberg+hive Mixed table while compatible with the original Hive data
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1947555133
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
Review Comment:
Includes pluggable monitoring plugins and table maintenance merge scheduling strategy.Cann‘t list them all in doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1947502499
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
Review Comment:
Does "pluggable and" means pluggable and Table Maintenance/AMS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "XBaith (via GitHub)" <gi...@apache.org>.
XBaith commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1915882328
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
Review Comment:
```suggestion
**[Apache Amoro(incubating)](https://amoro.apache.org/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
```
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. The new table
+format is used for this purpose. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
Review Comment:
I believe it is a streaming-enhanced table format based on Iceberg. The table format essentially extends Iceberg by combining two Iceberg tables.
We can remove this statement as it lacks clarity in the current context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1959029226
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by [AMS](https://amoro.apache.org/docs/latest/#architecture). AMS is responsible for
Review Comment:
Why is AMS not linked when it's firstly introduced above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "XBaith (via GitHub)" <gi...@apache.org>.
XBaith commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1915887251
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. The new table
+format is used for this purpose. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
Review Comment:
I believe it is a streaming-enhanced table format based on Iceberg. The table format essentially extends Iceberg by combining two Iceberg tables.
We can remove this statement as it lacks clarity in the current context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2664953781
I have now corrected the format and removed some descriptions, and link some feature to amoro website cc @manuzhang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1960870944
##########
docs/mkdocs.yml:
##########
@@ -69,6 +70,7 @@ nav:
- Druid: https://druid.apache.org/docs/latest/development/extensions-contrib/iceberg/
- Kafka Connect: kafka-connect.md
- Integrations:
+ - amoro.md
Review Comment:
I don't think this is the right place, but it should replace the link above, like Daft and RisingWave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2667951803
Thanks for review. I will fix some image reference issues in the next [PR](https://github.com/apache/iceberg/pull/12332). cc @nastra @manuzhang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1961075017
##########
docs/mkdocs.yml:
##########
@@ -69,6 +70,7 @@ nav:
- Druid: https://druid.apache.org/docs/latest/development/extensions-contrib/iceberg/
- Kafka Connect: kafka-connect.md
- Integrations:
+ - amoro.md
Review Comment:
Thanks for correcting the document location
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #11966:
URL: https://github.com/apache/iceberg/pull/11966
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1922372019
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro MetaStore)** is Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS(Amoro MetaStore). AMS is responsible for
+detecting and planning Self-optimizing tasks for tables, and then scheduling them to Optimizers for distributed execution in real-time. Finally, AMS
+is responsible for submitting the optimizing results. Amoro achieves physical isolation of Optimizers through the Optimizer Group.
+
+The core features of [Amoro's Self-optimizing](https://amoro.apache.org/docs/latest/self-optimizing/) are:
+
+- Automated, Asynchronous and Transparent — Continuous background detecting of file changes, asynchronous distributed execution of optimizing tasks,
+ transparent and imperceptible to users
+- Resource Isolation and Sharing — Allow resources to be isolated and shared at the table level, as well as setting resource quotas
+- Flexible and Scalable Deployment — Optimizers support various deployment methods and convenient scaling
+
+
+# Iceberg Format
+
+Apache Amoro supports all catalog types supported by Iceberg, including but not limited to: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, and so on.
Review Comment:
sure,rest catalog include it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1921936611
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org/docs/latest/)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+# Auto Self-optimizing
+
+Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this
+architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time
+analysis scenarios, challenges arise. For example:
+
+- Streaming writes bring a massive amount of fragment files
+- CDC ingestion and streaming updates generate excessive redundant data
+- Using the new data lake format leads to orphan files and expired snapshots.
+
+These issues can significantly affect the performance and cost of data analysis. Therefore, Amoro has introduced a Self-optimizing mechanism to
+create an out-of-the-box Streaming Lakehouse management service that is as user-friendly as a traditional database or data warehouse. The new table
+format is used for this purpose. Self-optimizing involves various procedures such as file compaction, deduplication, and sorting.
+
+The architecture and working mechanism of Self-optimizing are shown in the figure below:
+
+
+
+The Optimizer is a component responsible for executing Self-optimizing tasks. It is a resident process managed by AMS. AMS is responsible for
Review Comment:
already add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #11966:
URL: https://github.com/apache/iceberg/pull/11966#discussion_r1959027363
##########
docs/docs/amoro.md:
##########
@@ -0,0 +1,89 @@
+---
+title: "Apache Amoro"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Apache Amoro With Iceberg
+
+**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and
+**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+**AMS(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore).
Review Comment:
nit: extra space before "provides"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org