You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Mansehaj Singh <ma...@snowflake.com.INVALID> on 2025/04/10 23:22:19 UTC
[DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Hi all! Nice to meet you.
I opened up https://github.com/apache/polaris-tools/pull/4 recently to add
a Polaris migration/synchronizer tool I've been working on to the
polaris-tools repo. By request, I'm sharing a design document here
detailing how the tool works and the roadmap for functionality that is in
development.
Here's the design doc giving a full overview:
https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
To summarize:
We can think of this tool as a configurable mirroring/migration tool to
migrate between two Polaris instances. I believe this would enable and
support many use cases that are quite cumbersome to carry out manually
today and break down barriers switching between open source and managed
offerings of Polaris. The tool has been designed with goals in mind that go
beyond supporting just the CLI implementation.
Please take a look at the design doc if you're interested!
Thank you!
- Sehaj
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Mansehaj Singh <ma...@snowflake.com.INVALID>.
Yes, the reason for not migrating/replicating from the metastore level was
that it wasn't necessarily true that you had access to the metastore layer,
but it was much more likely that there was some programmatic way to access
the Polaris APIs.
On Fri, Apr 11, 2025 at 6:17 PM Eric Maynard <er...@gmail.com>
wrote:
> For keeping two Polaris instances in sync, I agree that replicating at the
> persistence layer probably makes the most sense.
>
> However there are cases when you want to copy data from one Polaris
> instance to another but you may not have direct access to the metastore.
> For example, migrating from a self-hosted Polaris instance to a managed
> offering. To support these cases, I think a tool like this can be useful.
>
> On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
> > Hey, Thanks for the proposal and I agree with Yufei.
> >
> > We had a backend synchronization CLI for projectNessie[1]. Maybe we can
> > have something similar to that instead of taking a long path of the
> > register table for migration between polaris instances.
> >
> > [1] https://projectnessie.org/nessie-0-82-0/export_import/
> >
> > - Ajantha
> >
> > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks, Mansehaj, for the proposal! This tool has potential, but I
> think
> > we
> > > should clarify its capabilities a bit more explicitly. Given its
> current
> > > limitations, I'm not sure how broadly useful it would be. Have we
> > explored
> > > any alternative approaches—for example, performing synchronization on
> the
> > > backend(FoundationDB, Postgres)?
> > >
> > >
> > > Yufei
> > >
> > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
> > > <ma...@snowflake.com.invalid> wrote:
> > >
> > > > Hi all! Nice to meet you.
> > > >
> > > > I opened up https://github.com/apache/polaris-tools/pull/4 recently
> to
> > > add
> > > > a Polaris migration/synchronizer tool I've been working on to the
> > > > polaris-tools repo. By request, I'm sharing a design document here
> > > > detailing how the tool works and the roadmap for functionality that
> is
> > in
> > > > development.
> > > >
> > > > Here's the design doc giving a full overview:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
> > > >
> > > >
> > > > To summarize:
> > > >
> > > > We can think of this tool as a configurable mirroring/migration tool
> to
> > > > migrate between two Polaris instances. I believe this would enable
> and
> > > > support many use cases that are quite cumbersome to carry out
> manually
> > > > today and break down barriers switching between open source and
> managed
> > > > offerings of Polaris. The tool has been designed with goals in mind
> > that
> > > go
> > > > beyond supporting just the CLI implementation.
> > > >
> > > > Please take a look at the design doc if you're interested!
> > > >
> > > > Thank you!
> > > > - Sehaj
> > > >
> > >
> >
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Mansehaj Singh <ma...@snowflake.com.INVALID>.
Hi everyone,
I have merged in https://github.com/mansehajsingh/polaris-tools/pull/4 to
give us a better framework for adding file import/export to the tool in the
future.
This basically puts the Polaris read/write source behind a common interface
that we can implement with new variations for different schemes of
migration we want to perform later, including reading/writing from a file.
Right now the tool still only supports the API as a source and a target.
Hopefully this will be a good starting point so that we can reuse the core
logic of the tool in the future as well. Reviews of the main PR:
https://github.com/apache/polaris-tools/pull/4 are welcome!
Thanks everyone,
Sehaj
On Mon, Apr 14, 2025 at 3:45 PM Mansehaj Singh <ma...@snowflake.com>
wrote:
> Hey all!
>
> Totally agree going forward that metastore level access would be much more
> useful and correct for replicating than the Polaris APIs, and that some
> sort of import/export/polaris_dump functionality would open up a greater
> variety of use cases in the future. I don't think these have to be mutually
> exclusive with the tool in its current state, however. With the tool
> designed the way it is right now, we can essentially think of it as taking
> Polaris entities from a source, and creating those entities in a target
> destination. While it only supports Polaris API to API migrations right
> now, we can add support in the future for configuring this so that the
> source/target don't have to be Polaris APIs. For example:
>
> - Export: the "source" could be the Polaris APIs and the "target"
> could be a file that we are just writing the entities and their properties
> to
> - Import: the "source" could be a file from a previous export and the
> "target" could be the Polaris APIs
> - OSS to managed: the "source" could be a metastore level connection
> and the "target" a Polaris API
> - Managed to OSS: the "source" could be the Polaris APIs and the
> "target" a metastore connection
> - or really any combo of the above, we just need a common interface
> for them all to implement
>
> I'm working on a draft right now that can generify the interfaces enough
> semantically to where I think we can write implementations down the line to
> support the functionality above, but before I dive too deep into the
> implementation for that- would that plan sound okay for everyone?
>
> Based on some customers we've discussed with, the tool even in its current
> state seems to provide value, so how do people feel about moving forward
> with that initial structure knowing it can be easily extensible to add more
> thorough features and use cases?
>
> The customers I'm talking to are wanting something that's easier than
> having to fully write their own migration utility to be able to do more
> ad-hoc backups / migrations and this initial version has been sufficient to
> give that peace of mind that there are some utilities to ease a migration
> to / from a managed or OSS self hosted instance.
>
> I agree that there's definitely a lot of improvements that can be made on
> top of this to make an even smoother experience and widen the use cases the
> migrator can be used for though.
>
> Thanks all for the feedback!
> Sehaj
>
> On Mon, Apr 14, 2025 at 2:57 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Export / Import does not have to be at the Persistence layer. If the Admin
>> API is expressive enough, it should be possible use it instead, I guess.
>>
>> Cheers,
>> Dmitri.
>>
>> On Fri, Apr 11, 2025 at 9:17 PM Eric Maynard <er...@gmail.com>
>> wrote:
>>
>> > For keeping two Polaris instances in sync, I agree that replicating at
>> the
>> > persistence layer probably makes the most sense.
>> >
>> > However there are cases when you want to copy data from one Polaris
>> > instance to another but you may not have direct access to the metastore.
>> > For example, migrating from a self-hosted Polaris instance to a managed
>> > offering. To support these cases, I think a tool like this can be
>> useful.
>> >
>> > On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <aj...@gmail.com>
>> > wrote:
>> >
>> > > Hey, Thanks for the proposal and I agree with Yufei.
>> > >
>> > > We had a backend synchronization CLI for projectNessie[1]. Maybe we
>> can
>> > > have something similar to that instead of taking a long path of the
>> > > register table for migration between polaris instances.
>> > >
>> > > [1] https://projectnessie.org/nessie-0-82-0/export_import/
>> > >
>> > > - Ajantha
>> > >
>> > > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com>
>> wrote:
>> > >
>> > > > Thanks, Mansehaj, for the proposal! This tool has potential, but I
>> > think
>> > > we
>> > > > should clarify its capabilities a bit more explicitly. Given its
>> > current
>> > > > limitations, I'm not sure how broadly useful it would be. Have we
>> > > explored
>> > > > any alternative approaches—for example, performing synchronization
>> on
>> > the
>> > > > backend(FoundationDB, Postgres)?
>> > > >
>> > > >
>> > > > Yufei
>> > > >
>> > > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
>> > > > <ma...@snowflake.com.invalid> wrote:
>> > > >
>> > > > > Hi all! Nice to meet you.
>> > > > >
>> > > > > I opened up https://github.com/apache/polaris-tools/pull/4
>> recently
>> > to
>> > > > add
>> > > > > a Polaris migration/synchronizer tool I've been working on to the
>> > > > > polaris-tools repo. By request, I'm sharing a design document here
>> > > > > detailing how the tool works and the roadmap for functionality
>> that
>> > is
>> > > in
>> > > > > development.
>> > > > >
>> > > > > Here's the design doc giving a full overview:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
>> > > > >
>> > > > >
>> > > > > To summarize:
>> > > > >
>> > > > > We can think of this tool as a configurable mirroring/migration
>> tool
>> > to
>> > > > > migrate between two Polaris instances. I believe this would enable
>> > and
>> > > > > support many use cases that are quite cumbersome to carry out
>> > manually
>> > > > > today and break down barriers switching between open source and
>> > managed
>> > > > > offerings of Polaris. The tool has been designed with goals in
>> mind
>> > > that
>> > > > go
>> > > > > beyond supporting just the CLI implementation.
>> > > > >
>> > > > > Please take a look at the design doc if you're interested!
>> > > > >
>> > > > > Thank you!
>> > > > > - Sehaj
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Mansehaj Singh <ma...@snowflake.com.INVALID>.
Hey all!
Totally agree going forward that metastore level access would be much more
useful and correct for replicating than the Polaris APIs, and that some
sort of import/export/polaris_dump functionality would open up a greater
variety of use cases in the future. I don't think these have to be mutually
exclusive with the tool in its current state, however. With the tool
designed the way it is right now, we can essentially think of it as taking
Polaris entities from a source, and creating those entities in a target
destination. While it only supports Polaris API to API migrations right
now, we can add support in the future for configuring this so that the
source/target don't have to be Polaris APIs. For example:
- Export: the "source" could be the Polaris APIs and the "target" could
be a file that we are just writing the entities and their properties to
- Import: the "source" could be a file from a previous export and the
"target" could be the Polaris APIs
- OSS to managed: the "source" could be a metastore level connection and
the "target" a Polaris API
- Managed to OSS: the "source" could be the Polaris APIs and the
"target" a metastore connection
- or really any combo of the above, we just need a common interface for
them all to implement
I'm working on a draft right now that can generify the interfaces enough
semantically to where I think we can write implementations down the line to
support the functionality above, but before I dive too deep into the
implementation for that- would that plan sound okay for everyone?
Based on some customers we've discussed with, the tool even in its current
state seems to provide value, so how do people feel about moving forward
with that initial structure knowing it can be easily extensible to add more
thorough features and use cases?
The customers I'm talking to are wanting something that's easier than
having to fully write their own migration utility to be able to do more
ad-hoc backups / migrations and this initial version has been sufficient to
give that peace of mind that there are some utilities to ease a migration
to / from a managed or OSS self hosted instance.
I agree that there's definitely a lot of improvements that can be made on
top of this to make an even smoother experience and widen the use cases the
migrator can be used for though.
Thanks all for the feedback!
Sehaj
On Mon, Apr 14, 2025 at 2:57 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Export / Import does not have to be at the Persistence layer. If the Admin
> API is expressive enough, it should be possible use it instead, I guess.
>
> Cheers,
> Dmitri.
>
> On Fri, Apr 11, 2025 at 9:17 PM Eric Maynard <er...@gmail.com>
> wrote:
>
> > For keeping two Polaris instances in sync, I agree that replicating at
> the
> > persistence layer probably makes the most sense.
> >
> > However there are cases when you want to copy data from one Polaris
> > instance to another but you may not have direct access to the metastore.
> > For example, migrating from a self-hosted Polaris instance to a managed
> > offering. To support these cases, I think a tool like this can be useful.
> >
> > On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <aj...@gmail.com>
> > wrote:
> >
> > > Hey, Thanks for the proposal and I agree with Yufei.
> > >
> > > We had a backend synchronization CLI for projectNessie[1]. Maybe we can
> > > have something similar to that instead of taking a long path of the
> > > register table for migration between polaris instances.
> > >
> > > [1] https://projectnessie.org/nessie-0-82-0/export_import/
> > >
> > > - Ajantha
> > >
> > > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Thanks, Mansehaj, for the proposal! This tool has potential, but I
> > think
> > > we
> > > > should clarify its capabilities a bit more explicitly. Given its
> > current
> > > > limitations, I'm not sure how broadly useful it would be. Have we
> > > explored
> > > > any alternative approaches—for example, performing synchronization on
> > the
> > > > backend(FoundationDB, Postgres)?
> > > >
> > > >
> > > > Yufei
> > > >
> > > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
> > > > <ma...@snowflake.com.invalid> wrote:
> > > >
> > > > > Hi all! Nice to meet you.
> > > > >
> > > > > I opened up https://github.com/apache/polaris-tools/pull/4
> recently
> > to
> > > > add
> > > > > a Polaris migration/synchronizer tool I've been working on to the
> > > > > polaris-tools repo. By request, I'm sharing a design document here
> > > > > detailing how the tool works and the roadmap for functionality that
> > is
> > > in
> > > > > development.
> > > > >
> > > > > Here's the design doc giving a full overview:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
> > > > >
> > > > >
> > > > > To summarize:
> > > > >
> > > > > We can think of this tool as a configurable mirroring/migration
> tool
> > to
> > > > > migrate between two Polaris instances. I believe this would enable
> > and
> > > > > support many use cases that are quite cumbersome to carry out
> > manually
> > > > > today and break down barriers switching between open source and
> > managed
> > > > > offerings of Polaris. The tool has been designed with goals in mind
> > > that
> > > > go
> > > > > beyond supporting just the CLI implementation.
> > > > >
> > > > > Please take a look at the design doc if you're interested!
> > > > >
> > > > > Thank you!
> > > > > - Sehaj
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Export / Import does not have to be at the Persistence layer. If the Admin
API is expressive enough, it should be possible use it instead, I guess.
Cheers,
Dmitri.
On Fri, Apr 11, 2025 at 9:17 PM Eric Maynard <er...@gmail.com>
wrote:
> For keeping two Polaris instances in sync, I agree that replicating at the
> persistence layer probably makes the most sense.
>
> However there are cases when you want to copy data from one Polaris
> instance to another but you may not have direct access to the metastore.
> For example, migrating from a self-hosted Polaris instance to a managed
> offering. To support these cases, I think a tool like this can be useful.
>
> On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
> > Hey, Thanks for the proposal and I agree with Yufei.
> >
> > We had a backend synchronization CLI for projectNessie[1]. Maybe we can
> > have something similar to that instead of taking a long path of the
> > register table for migration between polaris instances.
> >
> > [1] https://projectnessie.org/nessie-0-82-0/export_import/
> >
> > - Ajantha
> >
> > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks, Mansehaj, for the proposal! This tool has potential, but I
> think
> > we
> > > should clarify its capabilities a bit more explicitly. Given its
> current
> > > limitations, I'm not sure how broadly useful it would be. Have we
> > explored
> > > any alternative approaches—for example, performing synchronization on
> the
> > > backend(FoundationDB, Postgres)?
> > >
> > >
> > > Yufei
> > >
> > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
> > > <ma...@snowflake.com.invalid> wrote:
> > >
> > > > Hi all! Nice to meet you.
> > > >
> > > > I opened up https://github.com/apache/polaris-tools/pull/4 recently
> to
> > > add
> > > > a Polaris migration/synchronizer tool I've been working on to the
> > > > polaris-tools repo. By request, I'm sharing a design document here
> > > > detailing how the tool works and the roadmap for functionality that
> is
> > in
> > > > development.
> > > >
> > > > Here's the design doc giving a full overview:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
> > > >
> > > >
> > > > To summarize:
> > > >
> > > > We can think of this tool as a configurable mirroring/migration tool
> to
> > > > migrate between two Polaris instances. I believe this would enable
> and
> > > > support many use cases that are quite cumbersome to carry out
> manually
> > > > today and break down barriers switching between open source and
> managed
> > > > offerings of Polaris. The tool has been designed with goals in mind
> > that
> > > go
> > > > beyond supporting just the CLI implementation.
> > > >
> > > > Please take a look at the design doc if you're interested!
> > > >
> > > > Thank you!
> > > > - Sehaj
> > > >
> > >
> >
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Eric Maynard <er...@gmail.com>.
For keeping two Polaris instances in sync, I agree that replicating at the
persistence layer probably makes the most sense.
However there are cases when you want to copy data from one Polaris
instance to another but you may not have direct access to the metastore.
For example, migrating from a self-hosted Polaris instance to a managed
offering. To support these cases, I think a tool like this can be useful.
On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <aj...@gmail.com> wrote:
> Hey, Thanks for the proposal and I agree with Yufei.
>
> We had a backend synchronization CLI for projectNessie[1]. Maybe we can
> have something similar to that instead of taking a long path of the
> register table for migration between polaris instances.
>
> [1] https://projectnessie.org/nessie-0-82-0/export_import/
>
> - Ajantha
>
> On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks, Mansehaj, for the proposal! This tool has potential, but I think
> we
> > should clarify its capabilities a bit more explicitly. Given its current
> > limitations, I'm not sure how broadly useful it would be. Have we
> explored
> > any alternative approaches—for example, performing synchronization on the
> > backend(FoundationDB, Postgres)?
> >
> >
> > Yufei
> >
> > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
> > <ma...@snowflake.com.invalid> wrote:
> >
> > > Hi all! Nice to meet you.
> > >
> > > I opened up https://github.com/apache/polaris-tools/pull/4 recently to
> > add
> > > a Polaris migration/synchronizer tool I've been working on to the
> > > polaris-tools repo. By request, I'm sharing a design document here
> > > detailing how the tool works and the roadmap for functionality that is
> in
> > > development.
> > >
> > > Here's the design doc giving a full overview:
> > >
> > >
> >
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
> > >
> > >
> > > To summarize:
> > >
> > > We can think of this tool as a configurable mirroring/migration tool to
> > > migrate between two Polaris instances. I believe this would enable and
> > > support many use cases that are quite cumbersome to carry out manually
> > > today and break down barriers switching between open source and managed
> > > offerings of Polaris. The tool has been designed with goals in mind
> that
> > go
> > > beyond supporting just the CLI implementation.
> > >
> > > Please take a look at the design doc if you're interested!
> > >
> > > Thank you!
> > > - Sehaj
> > >
> >
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Dmitri Bourlatchkov <di...@apache.org>.
I agree with Ajentha's point.
Treating this as export/import (via an intermediate format) will open more
use cases to the admin user.
The original migration from a catalog to another catalog is still possible
as a sub-case of export/import.
Note: I made a comment on the doc about NOT exporting secrets. I think it
still needs special attention in the export case.
We probably still want to support full Polaris recovery with this tool, but
the tool probably needs to expose special options for allowing the export
of principal secrets. Even if they are only password hashes, exposing them
in another environment will add security risks.
Cheers,
Dmitri.
On Fri, Apr 11, 2025 at 9:11 PM Ajantha Bhat <aj...@gmail.com> wrote:
> Hey, Thanks for the proposal and I agree with Yufei.
>
> We had a backend synchronization CLI for projectNessie[1]. Maybe we can
> have something similar to that instead of taking a long path of the
> register table for migration between polaris instances.
>
> [1] https://projectnessie.org/nessie-0-82-0/export_import/
>
> - Ajantha
>
> On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks, Mansehaj, for the proposal! This tool has potential, but I think
> we
> > should clarify its capabilities a bit more explicitly. Given its current
> > limitations, I'm not sure how broadly useful it would be. Have we
> explored
> > any alternative approaches—for example, performing synchronization on the
> > backend(FoundationDB, Postgres)?
> >
> >
> > Yufei
> >
> > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
> > <ma...@snowflake.com.invalid> wrote:
> >
> > > Hi all! Nice to meet you.
> > >
> > > I opened up https://github.com/apache/polaris-tools/pull/4 recently to
> > add
> > > a Polaris migration/synchronizer tool I've been working on to the
> > > polaris-tools repo. By request, I'm sharing a design document here
> > > detailing how the tool works and the roadmap for functionality that is
> in
> > > development.
> > >
> > > Here's the design doc giving a full overview:
> > >
> > >
> >
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
> > >
> > >
> > > To summarize:
> > >
> > > We can think of this tool as a configurable mirroring/migration tool to
> > > migrate between two Polaris instances. I believe this would enable and
> > > support many use cases that are quite cumbersome to carry out manually
> > > today and break down barriers switching between open source and managed
> > > offerings of Polaris. The tool has been designed with goals in mind
> that
> > go
> > > beyond supporting just the CLI implementation.
> > >
> > > Please take a look at the design doc if you're interested!
> > >
> > > Thank you!
> > > - Sehaj
> > >
> >
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Ajantha Bhat <aj...@gmail.com>.
Hey, Thanks for the proposal and I agree with Yufei.
We had a backend synchronization CLI for projectNessie[1]. Maybe we can
have something similar to that instead of taking a long path of the
register table for migration between polaris instances.
[1] https://projectnessie.org/nessie-0-82-0/export_import/
- Ajantha
On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <fl...@gmail.com> wrote:
> Thanks, Mansehaj, for the proposal! This tool has potential, but I think we
> should clarify its capabilities a bit more explicitly. Given its current
> limitations, I'm not sure how broadly useful it would be. Have we explored
> any alternative approaches—for example, performing synchronization on the
> backend(FoundationDB, Postgres)?
>
>
> Yufei
>
> On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
> <ma...@snowflake.com.invalid> wrote:
>
> > Hi all! Nice to meet you.
> >
> > I opened up https://github.com/apache/polaris-tools/pull/4 recently to
> add
> > a Polaris migration/synchronizer tool I've been working on to the
> > polaris-tools repo. By request, I'm sharing a design document here
> > detailing how the tool works and the roadmap for functionality that is in
> > development.
> >
> > Here's the design doc giving a full overview:
> >
> >
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
> >
> >
> > To summarize:
> >
> > We can think of this tool as a configurable mirroring/migration tool to
> > migrate between two Polaris instances. I believe this would enable and
> > support many use cases that are quite cumbersome to carry out manually
> > today and break down barriers switching between open source and managed
> > offerings of Polaris. The tool has been designed with goals in mind that
> go
> > beyond supporting just the CLI implementation.
> >
> > Please take a look at the design doc if you're interested!
> >
> > Thank you!
> > - Sehaj
> >
>
Re: [DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools
Posted by Yufei Gu <fl...@gmail.com>.
Thanks, Mansehaj, for the proposal! This tool has potential, but I think we
should clarify its capabilities a bit more explicitly. Given its current
limitations, I'm not sure how broadly useful it would be. Have we explored
any alternative approaches—for example, performing synchronization on the
backend(FoundationDB, Postgres)?
Yufei
On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh
<ma...@snowflake.com.invalid> wrote:
> Hi all! Nice to meet you.
>
> I opened up https://github.com/apache/polaris-tools/pull/4 recently to add
> a Polaris migration/synchronizer tool I've been working on to the
> polaris-tools repo. By request, I'm sharing a design document here
> detailing how the tool works and the roadmap for functionality that is in
> development.
>
> Here's the design doc giving a full overview:
>
> https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing
>
>
> To summarize:
>
> We can think of this tool as a configurable mirroring/migration tool to
> migrate between two Polaris instances. I believe this would enable and
> support many use cases that are quite cumbersome to carry out manually
> today and break down barriers switching between open source and managed
> offerings of Polaris. The tool has been designed with goals in mind that go
> beyond supporting just the CLI implementation.
>
> Please take a look at the design doc if you're interested!
>
> Thank you!
> - Sehaj
>