[DISCUSS] DP-8: Symlinked Log

[Sijie Guo <gu...@gmail.com>]

Hi all,

I created a proposal for supporting symlinks in Dlog, in order to migrate a
namespace from flat namespace to a hierarchical namespace.

https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log

Please take a look and let me know your thoughts.

- Sijie

Re: [DISCUSS] DP-8: Symlinked Log

[Leigh Stewart <ls...@twitter.com.INVALID>]

Another issue is that things like rename go far beyond the dl core library
- the eb/dl client refers to backend streams by name, so any config change
needs to take the entire stack into account.
I agree this begs for better modeling of things like stream name and other
configuration, but this work may end up being a fairly large project.

So the capabilities of metadata reconfiguration is kind of limited by the
> features offered by the metadata store. For example, zookeeper doesn't
> support rename operations.


There are things we can do though here, for example copies/renames in
transactions. Copying metadata to the same zk server actually might be
better than simlink in some ways, although less flexible.

Dan, do you have any ideas yet about how general purpose reconfiguration
might work?

On Wed, Apr 19, 2017 at 10:32 AM, Sijie Guo <gu...@gmail.com> wrote:

> Agreed that a general reconfiguration approach will be ideal for solving
> the problems
>
> However currently DL is using a metadata store (e.g. ZooKeeper) for
> storing and managing the metadata part. So the capabilities of metadata
> reconfiguration is kind of limited by the features offered by the metadata
> store. For example, zookeeper doesn't support rename operations. A symlink
> approach here is try to address the metadata reconfiguration problem when
> the underneath metadata store lacks such ability. Beside that, moving a
> flat namespace to a hierarchical namespace in general is a long-duration
> operation. So an approach should exist to allow both co-exist for a while
> and allow rolling back and forward.
>
> - Sijie
>
>
> On Wed, Apr 19, 2017 at 8:14 AM, Dan Benediktson <dbenediktson@twitter.com
> > wrote:
>
>> It seems to me that the fact that the namespace format needs to be
>> changed is the specific, immediate problem, but the bigger problem is that
>> there isn't a mechanism to perform an atomic reconfiguration step for log
>> metadata. Namespace format change is one example of a reconfiguration (if
>> we start viewing namespace format as a part of log metadata, which just
>> happens to currently be unspecified since there is no format other than the
>> flat one), but there are already other metadata reconfiguration changes
>> that are desired: https://issues.apache.org/jira/browse/DL-74 and
>> https://issues.apache.org/jira/browse/DL-77. I think we should try to
>> solve the general reconfiguration problem correctly first, and then that
>> should enable us to solve both this problem and those two more naturally. I
>> expect that a correct reconfiguration solution would obviate the need for
>> this sub-task: https://issues.apache.org/jira/browse/DL-201.
>>
>> On Tue, Apr 18, 2017 at 1:06 PM, Leigh Stewart <ls...@twitter.com>
>> wrote:
>>
>>> + Dan
>>>
>>> On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo <gu...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I created a proposal for supporting symlinks in Dlog, in order to
>>>> migrate a namespace from flat namespace to a hierarchical namespace.
>>>>
>>>> https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log
>>>>
>>>> Please take a look and let me know your thoughts.
>>>>
>>>> - Sijie
>>>>
>>>>
>>>
>>
>

Re: [DISCUSS] DP-8: Symlinked Log

[Sijie Guo <gu...@gmail.com>]

Agreed that a general reconfiguration approach will be ideal for solving
the problems

However currently DL is using a metadata store (e.g. ZooKeeper) for storing
and managing the metadata part. So the capabilities of metadata
reconfiguration is kind of limited by the features offered by the metadata
store. For example, zookeeper doesn't support rename operations. A symlink
approach here is try to address the metadata reconfiguration problem when
the underneath metadata store lacks such ability. Beside that, moving a
flat namespace to a hierarchical namespace in general is a long-duration
operation. So an approach should exist to allow both co-exist for a while
and allow rolling back and forward.

- Sijie

On Wed, Apr 19, 2017 at 8:14 AM, Dan Benediktson <db...@twitter.com>
wrote:

> It seems to me that the fact that the namespace format needs to be changed
> is the specific, immediate problem, but the bigger problem is that there
> isn't a mechanism to perform an atomic reconfiguration step for log
> metadata. Namespace format change is one example of a reconfiguration (if
> we start viewing namespace format as a part of log metadata, which just
> happens to currently be unspecified since there is no format other than the
> flat one), but there are already other metadata reconfiguration changes
> that are desired: https://issues.apache.org/jira/browse/DL-74 and
> https://issues.apache.org/jira/browse/DL-77. I think we should try to
> solve the general reconfiguration problem correctly first, and then that
> should enable us to solve both this problem and those two more naturally. I
> expect that a correct reconfiguration solution would obviate the need for
> this sub-task: https://issues.apache.org/jira/browse/DL-201.
>
> On Tue, Apr 18, 2017 at 1:06 PM, Leigh Stewart <ls...@twitter.com>
> wrote:
>
>> + Dan
>>
>> On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo <gu...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I created a proposal for supporting symlinks in Dlog, in order to
>>> migrate a namespace from flat namespace to a hierarchical namespace.
>>>
>>> https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log
>>>
>>> Please take a look and let me know your thoughts.
>>>
>>> - Sijie
>>>
>>>
>>
>

Re: [DISCUSS] DP-8: Symlinked Log

[Dan Benediktson <db...@twitter.com.INVALID>]

It seems to me that the fact that the namespace format needs to be changed
is the specific, immediate problem, but the bigger problem is that there
isn't a mechanism to perform an atomic reconfiguration step for log
metadata. Namespace format change is one example of a reconfiguration (if
we start viewing namespace format as a part of log metadata, which just
happens to currently be unspecified since there is no format other than the
flat one), but there are already other metadata reconfiguration changes
that are desired: https://issues.apache.org/jira/browse/DL-74 and
https://issues.apache.org/jira/browse/DL-77. I think we should try to solve
the general reconfiguration problem correctly first, and then that should
enable us to solve both this problem and those two more naturally. I expect
that a correct reconfiguration solution would obviate the need for this
sub-task: https://issues.apache.org/jira/browse/DL-201.

On Tue, Apr 18, 2017 at 1:06 PM, Leigh Stewart <ls...@twitter.com> wrote:

> + Dan
>
> On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo <gu...@gmail.com> wrote:
>
>> Hi all,
>>
>> I created a proposal for supporting symlinks in Dlog, in order to migrate
>> a namespace from flat namespace to a hierarchical namespace.
>>
>> https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log
>>
>> Please take a look and let me know your thoughts.
>>
>> - Sijie
>>
>>
>

Re: [DISCUSS] DP-8: Symlinked Log

[Leigh Stewart <ls...@twitter.com.INVALID>]

+ Dan

On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo <gu...@gmail.com> wrote:

> Hi all,
>
> I created a proposal for supporting symlinks in Dlog, in order to migrate
> a namespace from flat namespace to a hierarchical namespace.
>
> https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log
>
> Please take a look and let me know your thoughts.
>
> - Sijie
>
>