You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Jungtaek Lim <ka...@gmail.com> on 2016/03/15 14:36:48 UTC
[DISCUSS] Handling symbolic link to be compatible with Windows
Hi devs,
I'm experimenting current 1.x-branch on Windows 8 / Oracle Java 1.8.0_73
and there seems be some issues regarding BlobStore which makes *supervisor
not functioning*.
(I believe this relates to failures of BlobStore UT on Windows.)
I fixed one issue regarding moving non-empty directory (will submit PR) and
went ahead, but met symbolic link issue immediately.
Try some googling, but unfortunately making symlink seems to be complicated
in Windows.
- If account's group is administrator, account should make symlink with
running as Administrator.
- I ran supervisor by account which group is administrator, and it
throws FileSystemException: A required privilege is not held by the client.
- It runs fine when I run supervisor from *elevated command prompt*.
- If account's group is not administrator, user should add that account to
the list of accounts/groups of "Create symbolic links". Details:
http://superuser.com/a/105381
I just feel both of two options seems not ideal. According to options,
users should do something to make Storm run well with Windows.
What would be better way to handle symbolic link?
We can remove creating symlink if not mandatory, but I'm curious it's
convenient for *Nix users.
Please share your opinions.
Thanks!
Jungtaek Lim (HeartSaVioR)
Re: [DISCUSS] Handling symbolic link to be compatible with Windows
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Yes I have been trying to follow your work there. When I first saw the comment in the JIRA go by about the failure I didn't read it in depth, but now I remember a similar bug I reviewed when Hadoop was being ported to windows. I think that they were having issues with them switching to using the java built in tar that didn't support symlinks, vs using gnu-tar that did. For windows I would be fine with skipping that part that checks for symlinks in the test.
- Bobby
On Tuesday, March 15, 2016 9:22 PM, Jungtaek Lim <ka...@gmail.com> wrote:
Thanks Bobby for sharing opinion.
I was curious that what other projects handle it, and if Hadoop on Windows
also requires manual privilege, I'm also OK to just document how to set up.
Then release note for 1.0.0 should mention this change.
Maybe on-topic or off-topic, I'm also investigating BlobStore UTs failure
on Windows, and LocalizerTest contains tests which check symbolic links in
compressed file.
They just fail on Windows. In addition, I also can't extract these files
with WinZip regardless of running as administrator.
Would we like to assume that symbolic link in compressed file is not
supported? Or is it mandatory for BlobStore or some other features? If we
use this it still not work on Windows.
- Jungtaek Lim (HeartSaVioR)
2016년 3월 15일 (화) 오후 11:09, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
> We added in the symlinks copying what Hadoop does for their distributed
> cache, and they too require the same set of permissions to make symlinks
> and Hadoop work on windows. As such we didn't think it would be that big
> of an issue, but we don't use windows here at Yahoo and it is not that
> simple to get access to a windows box here so we didn't take the time to
> test out how painful it is.
> Currently it is not just a nice to have part of the blobstore. If a user
> wants to have access to a blob it shows up in the current working directory
> being pointed to by a symlink. This also allows us to both share the blobs
> among different workers running on the same box and atomically switch to an
> upgraded version of the file. Additionally the new worker-artifacts
> feature uses symlinks to keep files that a user wants to expose for
> download/debugging separate from temporary files that will go away when the
> worker is shot. Worker-artifacts we could rework to not use symlinks, but
> then we could not configure it to put the worker logs under the logs
> directory by default they would have to be hard coded to the worker
> directory and then moved when the worker finishes. At Yahoo we use the
> blob-store with dynamic update and compressed archives all the time and
> cannot remove it.
>
> Hard links are not an option because you cannot point to directories,
> which is a requirement for exposing uncompressed archives to end users.
> Perhaps what we could do is change worker-artifacts to not use symlinks,
> and then have an option to disable symlinks creation but at the same time
> also create a file that users/workers could read instead of the symlink to
> know where a blob is downloaded (a symlink without actually being a
> symlink). We would have to expose a library that would make it simple for
> end users to know how to get to their blob data, but that should not be too
> difficult.
> - Bobby
>
> On Tuesday, March 15, 2016 8:36 AM, Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>
> Hi devs,
>
> I'm experimenting current 1.x-branch on Windows 8 / Oracle Java 1.8.0_73
> and there seems be some issues regarding BlobStore which makes *supervisor
> not functioning*.
> (I believe this relates to failures of BlobStore UT on Windows.)
>
> I fixed one issue regarding moving non-empty directory (will submit PR) and
> went ahead, but met symbolic link issue immediately.
>
> Try some googling, but unfortunately making symlink seems to be complicated
> in Windows.
>
> - If account's group is administrator, account should make symlink with
> running as Administrator.
> - I ran supervisor by account which group is administrator, and it
> throws FileSystemException: A required privilege is not held by the client.
> - It runs fine when I run supervisor from *elevated command prompt*.
> - If account's group is not administrator, user should add that account to
> the list of accounts/groups of "Create symbolic links". Details:
> http://superuser.com/a/105381
>
> I just feel both of two options seems not ideal. According to options,
> users should do something to make Storm run well with Windows.
>
> What would be better way to handle symbolic link?
> We can remove creating symlink if not mandatory, but I'm curious it's
> convenient for *Nix users.
>
> Please share your opinions.
>
> Thanks!
> Jungtaek Lim (HeartSaVioR)
>
>
>
Re: [DISCUSS] Handling symbolic link to be compatible with Windows
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks Bobby for sharing opinion.
I was curious that what other projects handle it, and if Hadoop on Windows
also requires manual privilege, I'm also OK to just document how to set up.
Then release note for 1.0.0 should mention this change.
Maybe on-topic or off-topic, I'm also investigating BlobStore UTs failure
on Windows, and LocalizerTest contains tests which check symbolic links in
compressed file.
They just fail on Windows. In addition, I also can't extract these files
with WinZip regardless of running as administrator.
Would we like to assume that symbolic link in compressed file is not
supported? Or is it mandatory for BlobStore or some other features? If we
use this it still not work on Windows.
- Jungtaek Lim (HeartSaVioR)
2016년 3월 15일 (화) 오후 11:09, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
> We added in the symlinks copying what Hadoop does for their distributed
> cache, and they too require the same set of permissions to make symlinks
> and Hadoop work on windows. As such we didn't think it would be that big
> of an issue, but we don't use windows here at Yahoo and it is not that
> simple to get access to a windows box here so we didn't take the time to
> test out how painful it is.
> Currently it is not just a nice to have part of the blobstore. If a user
> wants to have access to a blob it shows up in the current working directory
> being pointed to by a symlink. This also allows us to both share the blobs
> among different workers running on the same box and atomically switch to an
> upgraded version of the file. Additionally the new worker-artifacts
> feature uses symlinks to keep files that a user wants to expose for
> download/debugging separate from temporary files that will go away when the
> worker is shot. Worker-artifacts we could rework to not use symlinks, but
> then we could not configure it to put the worker logs under the logs
> directory by default they would have to be hard coded to the worker
> directory and then moved when the worker finishes. At Yahoo we use the
> blob-store with dynamic update and compressed archives all the time and
> cannot remove it.
>
> Hard links are not an option because you cannot point to directories,
> which is a requirement for exposing uncompressed archives to end users.
> Perhaps what we could do is change worker-artifacts to not use symlinks,
> and then have an option to disable symlinks creation but at the same time
> also create a file that users/workers could read instead of the symlink to
> know where a blob is downloaded (a symlink without actually being a
> symlink). We would have to expose a library that would make it simple for
> end users to know how to get to their blob data, but that should not be too
> difficult.
> - Bobby
>
> On Tuesday, March 15, 2016 8:36 AM, Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>
> Hi devs,
>
> I'm experimenting current 1.x-branch on Windows 8 / Oracle Java 1.8.0_73
> and there seems be some issues regarding BlobStore which makes *supervisor
> not functioning*.
> (I believe this relates to failures of BlobStore UT on Windows.)
>
> I fixed one issue regarding moving non-empty directory (will submit PR) and
> went ahead, but met symbolic link issue immediately.
>
> Try some googling, but unfortunately making symlink seems to be complicated
> in Windows.
>
> - If account's group is administrator, account should make symlink with
> running as Administrator.
> - I ran supervisor by account which group is administrator, and it
> throws FileSystemException: A required privilege is not held by the client.
> - It runs fine when I run supervisor from *elevated command prompt*.
> - If account's group is not administrator, user should add that account to
> the list of accounts/groups of "Create symbolic links". Details:
> http://superuser.com/a/105381
>
> I just feel both of two options seems not ideal. According to options,
> users should do something to make Storm run well with Windows.
>
> What would be better way to handle symbolic link?
> We can remove creating symlink if not mandatory, but I'm curious it's
> convenient for *Nix users.
>
> Please share your opinions.
>
> Thanks!
> Jungtaek Lim (HeartSaVioR)
>
>
>
Re: [DISCUSS] Handling symbolic link to be compatible with Windows
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
We added in the symlinks copying what Hadoop does for their distributed cache, and they too require the same set of permissions to make symlinks and Hadoop work on windows. As such we didn't think it would be that big of an issue, but we don't use windows here at Yahoo and it is not that simple to get access to a windows box here so we didn't take the time to test out how painful it is.
Currently it is not just a nice to have part of the blobstore. If a user wants to have access to a blob it shows up in the current working directory being pointed to by a symlink. This also allows us to both share the blobs among different workers running on the same box and atomically switch to an upgraded version of the file. Additionally the new worker-artifacts feature uses symlinks to keep files that a user wants to expose for download/debugging separate from temporary files that will go away when the worker is shot. Worker-artifacts we could rework to not use symlinks, but then we could not configure it to put the worker logs under the logs directory by default they would have to be hard coded to the worker directory and then moved when the worker finishes. At Yahoo we use the blob-store with dynamic update and compressed archives all the time and cannot remove it.
Hard links are not an option because you cannot point to directories, which is a requirement for exposing uncompressed archives to end users. Perhaps what we could do is change worker-artifacts to not use symlinks, and then have an option to disable symlinks creation but at the same time also create a file that users/workers could read instead of the symlink to know where a blob is downloaded (a symlink without actually being a symlink). We would have to expose a library that would make it simple for end users to know how to get to their blob data, but that should not be too difficult.
- Bobby
On Tuesday, March 15, 2016 8:36 AM, Jungtaek Lim <ka...@gmail.com> wrote:
Hi devs,
I'm experimenting current 1.x-branch on Windows 8 / Oracle Java 1.8.0_73
and there seems be some issues regarding BlobStore which makes *supervisor
not functioning*.
(I believe this relates to failures of BlobStore UT on Windows.)
I fixed one issue regarding moving non-empty directory (will submit PR) and
went ahead, but met symbolic link issue immediately.
Try some googling, but unfortunately making symlink seems to be complicated
in Windows.
- If account's group is administrator, account should make symlink with
running as Administrator.
- I ran supervisor by account which group is administrator, and it
throws FileSystemException: A required privilege is not held by the client.
- It runs fine when I run supervisor from *elevated command prompt*.
- If account's group is not administrator, user should add that account to
the list of accounts/groups of "Create symbolic links". Details:
http://superuser.com/a/105381
I just feel both of two options seems not ideal. According to options,
users should do something to make Storm run well with Windows.
What would be better way to handle symbolic link?
We can remove creating symlink if not mandatory, but I'm curious it's
convenient for *Nix users.
Please share your opinions.
Thanks!
Jungtaek Lim (HeartSaVioR)
Re: [DISCUSS] Handling symbolic link to be compatible with Windows
Posted by Jungtaek Lim <ka...@gmail.com>.
FYI: I ran "mvn clean install" with elevated command prompt and no luck. UTs
in BlobStore still fail.
2016년 3월 15일 (화) 오후 10:36, Jungtaek Lim <ka...@gmail.com>님이 작성:
> Hi devs,
>
> I'm experimenting current 1.x-branch on Windows 8 / Oracle Java 1.8.0_73
> and there seems be some issues regarding BlobStore which makes *supervisor
> not functioning*.
> (I believe this relates to failures of BlobStore UT on Windows.)
>
> I fixed one issue regarding moving non-empty directory (will submit PR)
> and went ahead, but met symbolic link issue immediately.
>
> Try some googling, but unfortunately making symlink seems to be
> complicated in Windows.
>
> - If account's group is administrator, account should make symlink with
> running as Administrator.
> - I ran supervisor by account which group is administrator, and it
> throws FileSystemException: A required privilege is not held by the client.
> - It runs fine when I run supervisor from *elevated command prompt*.
> - If account's group is not administrator, user should add that account to
> the list of accounts/groups of "Create symbolic links". Details:
> http://superuser.com/a/105381
>
> I just feel both of two options seems not ideal. According to options,
> users should do something to make Storm run well with Windows.
>
> What would be better way to handle symbolic link?
> We can remove creating symlink if not mandatory, but I'm curious it's
> convenient for *Nix users.
>
> Please share your opinions.
>
> Thanks!
> Jungtaek Lim (HeartSaVioR)
>