You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "sagarlakshmipathy (via GitHub)" <gi...@apache.org> on 2024/08/21 21:44:51 UTC
[PR] [DOCS] add releases page and modify README.md [incubator-xtable]
sagarlakshmipathy opened a new pull request, #520:
URL: https://github.com/apache/incubator-xtable/pull/520
https://github.com/apache/incubator-xtable/issues/519
## What is the purpose of the pull request
1. Add Downloads navbar to the site
2. Add release file/links to the site under Downloads section
## Brief change log
1. Added new dir/files related to release
2. Changed `docusaurus.config.js` to support navigation
## Verify this pull request
Doc update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1726968011
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
Review Comment:
For the first release there's only features to list and the improvements we have made to the project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1727538353
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
+3. Metadata maintenance for the target table formats.
+ * For Hudi, unreferenced files will be marked as [cleaned](https://hudi.apache.org/docs/hoodie_cleaner/) to control the size of the metadata table.
+ * For Iceberg, snapshots will be [expired](https://iceberg.apache.org/docs/latest/maintenance/#expire-snapshots) after a configured amount of time.
+ * For Delta, the transaction log will be [retained](https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html) for a configured amount of time.
+
+
+## Improvements
+1. Added apache release guide and infra components to be compliant with ASF release process.
+2. Fix bugs related to dependency convicts, few edge cases related when parsing column stats.
Review Comment:
Yeah it was supposed to be `conflicts` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 merged PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1727543563
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
+3. Metadata maintenance for the target table formats.
+ * For Hudi, unreferenced files will be marked as [cleaned](https://hudi.apache.org/docs/hoodie_cleaner/) to control the size of the metadata table.
+ * For Iceberg, snapshots will be [expired](https://iceberg.apache.org/docs/latest/maintenance/#expire-snapshots) after a configured amount of time.
+ * For Delta, the transaction log will be [retained](https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html) for a configured amount of time.
+
+
+## Improvements
+1. Added apache release guide and infra components to be compliant with ASF release process.
+2. Fix bugs related to dependency convicts, few edge cases related when parsing column stats.
+3. Improved README, docker demo and website docs based on feedback provided by users.
+4. Refactored the codebase to follow apache naming practices.
+[apache-xtable-0.1.0-incubating.src (1).tgz](https://github.com/user-attachments/files/16709894/apache-xtable-0.1.0-incubating.src.1.tgz)
Review Comment:
This is a typo when copying the notes from GH release, will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "sagarlakshmipathy (via GitHub)" <gi...@apache.org>.
sagarlakshmipathy commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1727479832
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
Review Comment:
@vinishjail97 I think its better we have it in two places. My 2c, I'd think there would be people looking for features in one place, pointing them to release page would not be a good user exp. Then again, we can't add links to the features page from release notes as well.
I'm open to other ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1727557279
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
Review Comment:
okay let's keep it like it this and not overthink too much now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1726906793
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
Review Comment:
@sagarlakshmipathy Should we link the features page here to avoid duplication ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#discussion_r1727166359
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
Review Comment:
`apache` should be capitalized? and do we need `(Incubating)`
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
Review Comment:
```suggestion
Currently, Apache XTable™ (Incubating) supports the open-source table formats for Apache Hudi, Apache Iceberg, and Delta Lake.
```
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
Review Comment:
```suggestion
Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
```
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
+3. Metadata maintenance for the target table formats.
+ * For Hudi, unreferenced files will be marked as [cleaned](https://hudi.apache.org/docs/hoodie_cleaner/) to control the size of the metadata table.
+ * For Iceberg, snapshots will be [expired](https://iceberg.apache.org/docs/latest/maintenance/#expire-snapshots) after a configured amount of time.
+ * For Delta, the transaction log will be [retained](https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html) for a configured amount of time.
+
+
+## Improvements
+1. Added apache release guide and infra components to be compliant with ASF release process.
+2. Fix bugs related to dependency convicts, few edge cases related when parsing column stats.
Review Comment:
`convicts` is meant to be something else?
##########
website/releases/release-0.1.0-incubating.mdx:
##########
@@ -0,0 +1,34 @@
+---
+title: "Release 0.1.0-incubating"
+sidebar_position: 1
+---
+
+## [Release 0.1.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.1.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines.
+Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
+
+## Features
+Apache XTable™ (Incubating) provides users with the ability to translate metadata from one table format to another.
+
+Apache XTable™ (Incubating) provides two sync modes, "incremental" and "full." The incremental mode is more lightweight and has better performance, especially on large tables. If there is anything that prevents the incremental mode from working properly, the tool will fall back to the full sync mode.
+
+This sync provides users with the following:
+
+1. Syncing of data files along with their column level statistics and partition metadata
+2. Schema updates in the source are reflected in the target table metadata
+3. Metadata maintenance for the target table formats.
+ * For Hudi, unreferenced files will be marked as [cleaned](https://hudi.apache.org/docs/hoodie_cleaner/) to control the size of the metadata table.
+ * For Iceberg, snapshots will be [expired](https://iceberg.apache.org/docs/latest/maintenance/#expire-snapshots) after a configured amount of time.
+ * For Delta, the transaction log will be [retained](https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html) for a configured amount of time.
+
+
+## Improvements
+1. Added apache release guide and infra components to be compliant with ASF release process.
+2. Fix bugs related to dependency convicts, few edge cases related when parsing column stats.
+3. Improved README, docker demo and website docs based on feedback provided by users.
+4. Refactored the codebase to follow apache naming practices.
+[apache-xtable-0.1.0-incubating.src (1).tgz](https://github.com/user-attachments/files/16709894/apache-xtable-0.1.0-incubating.src.1.tgz)
Review Comment:
why is there a (1)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] add releases page and modify README.md [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on PR #520:
URL: https://github.com/apache/incubator-xtable/pull/520#issuecomment-2305307674
Squashed into a single commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org