You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Bernd Mathiske <be...@mesosphere.io> on 2015/01/13 14:24:29 UTC
Re: Review Request 29809: Fetcher program portion of MESOS-2069: basic
fetcher cache functionality.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/
-----------------------------------------------------------
(Updated Jan. 13, 2015, 5:24 a.m.)
Review request for mesos, Benjamin Hindman and Timothy Chen.
Changes
-------
Adjusted summary and descrition better to customary format and tense.
Summary (updated)
-----------------
Fetcher program portion of MESOS-2069: basic fetcher cache functionality.
Bugs: MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2069
Repository: mesos-git
Description (updated)
-------
Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
Diffs
-----
include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
Diff: https://reviews.apache.org/r/29809/diff/
Testing
-------
make check.
For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
The new functionality will be tested in the context of MESOS-2074.
Thanks,
Bernd Mathiske
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Bernd Mathiske <be...@mesosphere.io>.
> On Jan. 13, 2015, 11:25 a.m., Timothy Chen wrote:
> > src/launcher/fetcher.cpp, line 285
> > <https://reviews.apache.org/r/29809/diff/2/?file=818721#file818721line285>
> >
> > So I think the reason I think the method looks like it needed to be refactored, it's just that it seems very cluttered overall.
> >
> > I downloaded your patches and went through this file myself, and did most spacing cleanups:
> >
> > https://gist.github.com/tnachen/e97cb242d31fb2f281e4
> >
> > Mostly the following:
> >
> > - Leave spaces before returns and after some multiple line block
> > - Extra space in the switch statement, following the same style src/log/leveldb.cpp
> > - Remove extra LOG(ERROR), since we're already logging in LOG(ERROR) in the download code
> > - Reduce number of parameters passed to download
> > - Some small refactoring to reduce extra code
> >
> > Also I cleaned up some existing wierd formatting in the logging.
> >
> > If you like what I did then take the patch from the gist and apply a new review.
>
> Bernd Mathiske wrote:
> Good stuff! Thanks, Tim! I am going to incorporate your changes. On this trajectory, I found additional opportunities to furthr clean up and simplify the code. So I will work these out, too. Stay tuned!
Tim, after applying your patch, I found more opportunity to clean up the code and went for it. I think I have addressed all 5 points you made, but the affected method now looks quite different in its gestalt, having been split up into subroutines. Also there is a new way of dealing with URIs and actions that is way simpler, but changes the code structure in this place and others.
- Bernd
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/#review67915
-----------------------------------------------------------
On Jan. 13, 2015, 5:27 a.m., Bernd Mathiske wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29809/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2015, 5:27 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Timothy Chen.
>
>
> Bugs: MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2069
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
>
>
> Diffs
> -----
>
> include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
> src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
> src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
> src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
>
> Diff: https://reviews.apache.org/r/29809/diff/
>
>
> Testing
> -------
>
> make check.
>
> For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
>
> The new functionality will be tested in the context of MESOS-2074.
>
>
> Thanks,
>
> Bernd Mathiske
>
>
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Bernd Mathiske <be...@mesosphere.io>.
> On Jan. 13, 2015, 11:25 a.m., Timothy Chen wrote:
> > src/launcher/fetcher.cpp, line 285
> > <https://reviews.apache.org/r/29809/diff/2/?file=818721#file818721line285>
> >
> > So I think the reason I think the method looks like it needed to be refactored, it's just that it seems very cluttered overall.
> >
> > I downloaded your patches and went through this file myself, and did most spacing cleanups:
> >
> > https://gist.github.com/tnachen/e97cb242d31fb2f281e4
> >
> > Mostly the following:
> >
> > - Leave spaces before returns and after some multiple line block
> > - Extra space in the switch statement, following the same style src/log/leveldb.cpp
> > - Remove extra LOG(ERROR), since we're already logging in LOG(ERROR) in the download code
> > - Reduce number of parameters passed to download
> > - Some small refactoring to reduce extra code
> >
> > Also I cleaned up some existing wierd formatting in the logging.
> >
> > If you like what I did then take the patch from the gist and apply a new review.
Good stuff! Thanks, Tim! I am going to incorporate your changes. On this trajectory, I found additional opportunities to furthr clean up and simplify the code. So I will work these out, too. Stay tuned!
- Bernd
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/#review67915
-----------------------------------------------------------
On Jan. 13, 2015, 5:27 a.m., Bernd Mathiske wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29809/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2015, 5:27 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Timothy Chen.
>
>
> Bugs: MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2069
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
>
>
> Diffs
> -----
>
> include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
> src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
> src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
> src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
>
> Diff: https://reviews.apache.org/r/29809/diff/
>
>
> Testing
> -------
>
> make check.
>
> For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
>
> The new functionality will be tested in the context of MESOS-2074.
>
>
> Thanks,
>
> Bernd Mathiske
>
>
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/#review67915
-----------------------------------------------------------
src/launcher/fetcher.cpp
<https://reviews.apache.org/r/29809/#comment112001>
So I think the reason I think the method looks like it needed to be refactored, it's just that it seems very cluttered overall.
I downloaded your patches and went through this file myself, and did most spacing cleanups:
https://gist.github.com/tnachen/e97cb242d31fb2f281e4
Mostly the following:
- Leave spaces before returns and after some multiple line block
- Extra space in the switch statement, following the same style src/log/leveldb.cpp
- Remove extra LOG(ERROR), since we're already logging in LOG(ERROR) in the download code
- Reduce number of parameters passed to download
- Some small refactoring to reduce extra code
Also I cleaned up some existing wierd formatting in the logging.
If you like what I did then take the patch from the gist and apply a new review.
- Timothy Chen
On Jan. 13, 2015, 1:27 p.m., Bernd Mathiske wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29809/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2015, 1:27 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Timothy Chen.
>
>
> Bugs: MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2069
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
>
>
> Diffs
> -----
>
> include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
> src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
> src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
> src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
>
> Diff: https://reviews.apache.org/r/29809/diff/
>
>
> Testing
> -------
>
> make check.
>
> For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
>
> The new functionality will be tested in the context of MESOS-2074.
>
>
> Thanks,
>
> Bernd Mathiske
>
>
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Mesos ReviewBot <de...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/#review68548
-----------------------------------------------------------
Patch looks great!
Reviews applied: [29560, 29809]
All tests passed.
- Mesos ReviewBot
On Jan. 17, 2015, 11:34 a.m., Bernd Mathiske wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29809/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2015, 11:34 a.m.)
>
>
> Review request for mesos, Adam B, Benjamin Hindman, and Timothy Chen.
>
>
> Bugs: MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2069
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
>
>
> Diffs
> -----
>
> include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
> src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
> src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
> src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
>
> Diff: https://reviews.apache.org/r/29809/diff/
>
>
> Testing
> -------
>
> make check.
>
> For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
>
> The new functionality will be tested in the context of MESOS-2074.
>
>
> Thanks,
>
> Bernd Mathiske
>
>
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Bernd Mathiske <be...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/
-----------------------------------------------------------
(Updated Jan. 17, 2015, 3:34 a.m.)
Review request for mesos, Adam B, Benjamin Hindman, and Timothy Chen.
Changes
-------
+Adam
Bugs: MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2069
Repository: mesos-git
Description
-------
Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
Diffs
-----
include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
Diff: https://reviews.apache.org/r/29809/diff/
Testing
-------
make check.
For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
The new functionality will be tested in the context of MESOS-2074.
Thanks,
Bernd Mathiske
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Bernd Mathiske <be...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/
-----------------------------------------------------------
(Updated Jan. 17, 2015, 3:31 a.m.)
Review request for mesos, Benjamin Hindman and Timothy Chen.
Changes
-------
Better FetcherInfo message structure and naming. Instead of several repeatables (URIs, actions, cache file names) now using just one (items) with multiple fields (uri, action, cache file name). Making the cache file name field optional. This reduces overall complexity, also in FetcherProcess.
Cleaned up all of the launcher/fetcher.cpp code, with more breaking out of subroutines for better readability. Payed close attention to compatibility with legacy behavior (except for FetcherInfo structure). Also minimizing complexity in FetcherProcess.
Streamlined the main() function and provided an overview of what this program does in a comment.
Eliminated last remnants of the hadoop_home env var / fetchr parameter. I could not find any use of it when checking old states of master months back. Apparently fetching with hadoop has not been using this parameter for a long time.
Unified error logging in one central place, removing a lot of unnecessary code and improving readability. It is easy enough to trace back what error situation at what code line any log message refers to, by simple string matching part of the log message to the 'return Error("...")' statement in question. Therefore we do not really need immediate LOG statements at the place where the error originates.
Addressed the issues raised in Tim's latest comment, starting out with his patch and then carrying on with more cleanup.
Bugs: MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2069
Repository: mesos-git
Description
-------
Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
Diffs (updated)
-----
include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
Diff: https://reviews.apache.org/r/29809/diff/
Testing
-------
make check.
For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
The new functionality will be tested in the context of MESOS-2074.
Thanks,
Bernd Mathiske
Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Posted by Bernd Mathiske <be...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29809/
-----------------------------------------------------------
(Updated Jan. 13, 2015, 5:27 a.m.)
Review request for mesos, Benjamin Hindman and Timothy Chen.
Changes
-------
Even more compliant with expectations: no mention of the ticket number, past tense sentence.
Summary (updated)
-----------------
Enhanced Fetcher program with cache actions.
Bugs: MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2069
Repository: mesos-git
Description
-------
Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional parameter in FetcherInfo: the cache filenames to be used for these actions.
Diffs
-----
include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721
src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89
src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829
src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748
Diff: https://reviews.apache.org/r/29809/diff/
Testing
-------
make check.
For now it is OK if only the FETCH action works, which implements the legacy behavior without any caching.
The new functionality will be tested in the context of MESOS-2074.
Thanks,
Bernd Mathiske