You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by orhankislal <gi...@git.apache.org> on 2018/06/07 22:46:56 UTC
[GitHub] madlib pull request #276: Feature/dev check
GitHub user orhankislal opened a pull request:
https://github.com/apache/madlib/pull/276
Feature/dev check
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/madlib/madlib feature/dev-check
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/madlib/pull/276.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #276
----
commit 50a35b2e5799a5ddf6f4d23a7bca4566dbd1d05b
Author: Nandish Jayaram <nj...@...>
Date: 2018-06-06T23:42:56Z
Madpack: Add dev-check and a compact install-check.
- The current install check is expensive since it runs various hyper param
permutations for all MADlib modules. This commits moves all of those
tests to dev-check, which can be used by developers for iterating
faster. We have now created watered down install-check for each module,
which just runs one hyper-param combination for each MADlib function,
and does not do any asserts.
- This commit also includes changes in madpack to add a new madpack
option for dev-check.
TODO:
- complete trimming install check for all modules.
- update documentation for developer consumption.
Co-authored-by: Arvind Sridhar <as...@pivotal.io>
commit d6c7834f73d00d0bd3ddf84af524ba08725a6244
Author: Arvind Sridhar <as...@...>
Date: 2018-06-07T21:51:47Z
Install-Check: Add new IC files for the lightweight testing
Co-authored-by: Orhan Kislal <ok...@pivotal.io>
----
---
[GitHub] madlib pull request #276: Feature/dev check
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal closed the pull request at:
https://github.com/apache/madlib/pull/276
---
[GitHub] madlib issue #276: Feature/dev check
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/madlib/pull/276
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/520/
---
[GitHub] madlib pull request #276: Feature/dev check
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/madlib/pull/276#discussion_r195830080
--- Diff: src/madpack/madpack.py ---
@@ -898,13 +900,19 @@ def run_install_check(args, testcase):
% (test_user, test_schema, schema)
# Loop through all test SQL files for this module
- sql_files = maddir_mod_sql + '/' + module + '/test/*.sql_in'
+ if is_install_check:
+ sql_files = maddir_mod_sql + '/' + module + '/test/*.ic.sql_in'
+ else:
+ sql_files = maddir_mod_sql + '/' + module + '/test/*.sql_in'
for sqlfile in sorted(glob.glob(sql_files), reverse=True):
algoname = os.path.basename(sqlfile).split('.')[0]
# run only algo specified
if (module in modset and modset[module] and
algoname not in modset[module]):
continue
+ # Do not run test/*.ic.sql_in files for dev-check.
--- End diff --
That's great, will try it out.
---
[GitHub] madlib issue #276: Feature/dev check
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/madlib/pull/276
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/499/
---
[GitHub] madlib pull request #276: Feature/dev check
Posted by iyerr3 <gi...@git.apache.org>.
Github user iyerr3 commented on a diff in the pull request:
https://github.com/apache/madlib/pull/276#discussion_r195278229
--- Diff: src/madpack/madpack.py ---
@@ -898,13 +900,19 @@ def run_install_check(args, testcase):
% (test_user, test_schema, schema)
# Loop through all test SQL files for this module
- sql_files = maddir_mod_sql + '/' + module + '/test/*.sql_in'
+ if is_install_check:
+ sql_files = maddir_mod_sql + '/' + module + '/test/*.ic.sql_in'
+ else:
+ sql_files = maddir_mod_sql + '/' + module + '/test/*.sql_in'
for sqlfile in sorted(glob.glob(sql_files), reverse=True):
algoname = os.path.basename(sqlfile).split('.')[0]
# run only algo specified
if (module in modset and modset[module] and
algoname not in modset[module]):
continue
+ # Do not run test/*.ic.sql_in files for dev-check.
--- End diff --
I believe if you use `*[!ic].sql_in` in the dev-check glob expression, it will exclude the ic files. You won't need these lines if that works.
---
[GitHub] madlib issue #276: Feature/dev check
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on the issue:
https://github.com/apache/madlib/pull/276
Thank you for the comments @iyerr3 , will make the changes you have requested.
Having one IC file for each module makes sense, but on Greenplum, for some modules the IC run time is still quite high. For example, if a module had 5 IC files, where each ran for 10 seconds, the user would see IC progressing every 10 seconds. But if those were combined into one IC file, the progression would happen after 50 seconds. It seemed a little odd (longer IC run times for some modules in terms of user experience) when we were trying it out, so decided to keep multiple IC files for such modules.
---
[GitHub] madlib issue #276: Feature/dev check
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/madlib/pull/276
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/518/
---
[GitHub] madlib issue #276: Feature/dev check
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/madlib/pull/276
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/517/
---