You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2018/06/11 04:06:01 UTC
[GitHub] rabbah closed pull request #3407: WIP: No longer need to build
'whisk/*' runtimes for builtins.
rabbah closed pull request #3407: WIP: No longer need to build 'whisk/*' runtimes for builtins.
URL: https://github.com/apache/incubator-openwhisk/pull/3407
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/actionRuntimes/actionProxy/Dockerfile b/actionRuntimes/actionProxy/Dockerfile
deleted file mode 100644
index 9d28b0acce..0000000000
--- a/actionRuntimes/actionProxy/Dockerfile
+++ /dev/null
@@ -1,2 +0,0 @@
-# Dockerfile for docker skeleton (useful for running blackbox binaries, scripts, or Python 3 actions) .
-FROM openwhisk/dockerskeleton:1.1.0
diff --git a/actionRuntimes/actionProxy/README.md b/actionRuntimes/actionProxy/README.md
deleted file mode 100644
index ee5d5e91af..0000000000
--- a/actionRuntimes/actionProxy/README.md
+++ /dev/null
@@ -1,37 +0,0 @@
-Skeleton for "docker actions"
-================
-
-The `dockerskeleton` base image is useful for actions that run scripts (e.g., bash, perl, python)
-and compiled binaries or, more generally, any native executable. It provides a proxy service
-(using Flask, a Python web microframework) that implements the required `/init` and `/run` routes
-to interact with the OpenWhisk invoker service. The implementation of these routes is encapsulated
-in a class named `ActionRunner` which provides a basic framework for receiving code from an invoker,
-preparing it for execution, and then running the code when required.
-
-The initialization of the `ActionRunner` is done via `init()` which receives a JSON object containing
-a `code` property whose value is the source code to execute. It writes the source to a `source` file.
-This method also provides a hook to optionally augment the received code via an `epilogue()` method,
-and then performs a `build()` to generate an executable. The last step of the initialization applies
-`verify()` to confirm the executable has the proper permissions to run the code. The action runner
-is ready to run the action if `verify()` is true.
-
-The default implementations of `epilogue()` and `build()` are no-ops and should be overridden as needed.
-The base image contains a stub added which is already executable by construction via `docker build`.
-For language runtimes (e.g., C) that require compiling the source, the extending class should run the
-required source compiler during `build()`.
-
-The `run()` method runs the action via the executable generated during `init()`. This method is only called
-by the proxy service if `verify()` is true. `ActionRunner` subclasses are encouraged to override this method
-if they have additional logic that should cause `run()` to never execute. The `run()` method calls the executable
-via a process and sends the received input parameters (from the invoker) to the action via the command line
-(as a JSON string argument). Additional properties received from the invoker are passed on to the action via
-environment variables as well. To augment the action environment, override `env()`.
-
-By convention the action executable may log messages to `stdout` and `stderr`. The proxy requires that the last
-line of output to `stdout` is a valid JSON object serialized to string if the action returns a JSON result.
-A return value is optional but must be a JSON object (properly serialized) if present.
-
-For an example implementation of an `ActionRunner` that overrides `epilogue()` and `build()` see the
-[Swift 3.x](https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift3Action/swift3runner.py) action proxy. An implementation of the runner for Python actions
-is available [here](https://github.com/apache/incubator-openwhisk-runtime-python/blob/master/core/pythonAction/pythonrunner.py). Lastly, an example Docker action that uses `C` is
-available in this [example](https://github.com/apache/incubator-openwhisk-runtime-docker/blob/master/sdk/docker/Dockerfile).
diff --git a/actionRuntimes/actionProxy/build.gradle b/actionRuntimes/actionProxy/build.gradle
deleted file mode 100644
index 0fe825eb6a..0000000000
--- a/actionRuntimes/actionProxy/build.gradle
+++ /dev/null
@@ -1,2 +0,0 @@
-ext.dockerImageName = 'dockerskeleton'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/actionProxy/invoke.py b/actionRuntimes/actionProxy/invoke.py
deleted file mode 100755
index b4957f58c4..0000000000
--- a/actionRuntimes/actionProxy/invoke.py
+++ /dev/null
@@ -1,136 +0,0 @@
-#!/usr/bin/env python
-"""Executable Python script for testing the action proxy.
-
- This script is useful for testing the action proxy (or its derivatives)
- by simulating invoker interactions. Use it in combination with
- docker run <image> which starts up the action proxy.
- Example:
- docker run -i -t -p 8080:8080 dockerskeleton # locally built images may be referenced without a tag
- ./invoke.py init <action source file>
- ./invoke.py run '{"some":"json object as a string"}'
-
- For additional help, try ./invoke.py -h
-
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-"""
-
-import os
-import re
-import sys
-import json
-import base64
-import requests
-import codecs
-import traceback
-import argparse
-try:
- import argcomplete
-except ImportError:
- argcomplete = False
-
-def main():
- try:
- args = parseArgs()
- exitCode = {
- 'init' : init,
- 'run' : run
- }[args.cmd](args)
- except Exception as e:
- print(e)
- exitCode = 1
- sys.exit(exitCode)
-
-def dockerHost():
- dockerHost = 'localhost'
- if 'DOCKER_HOST' in os.environ:
- try:
- dockerHost = re.compile('tcp://(.*):[\d]+').findall(os.environ['DOCKER_HOST'])[0]
- except Exception:
- print('cannot determine docker host from %s' % os.environ['DOCKER_HOST'])
- sys.exit(-1)
- return dockerHost
-
-def containerRoute(args, path):
- return 'http://%s:%s/%s' % (args.host, args.port, path)
-
-def parseArgs():
- parser = argparse.ArgumentParser(description='initialize and run an OpenWhisk action container')
- parser.add_argument('-v', '--verbose', help='verbose output', action='store_true')
- parser.add_argument('--host', help='action container host', default=dockerHost())
- parser.add_argument('-p', '--port', help='action container port number', default=8080, type=int)
-
- subparsers = parser.add_subparsers(title='available commands', dest='cmd')
-
- initmenu = subparsers.add_parser('init', help='initialize container with src or zip/tgz file')
- initmenu.add_argument('main', nargs='?', default='main', help='name of the "main" entry method for the action')
- initmenu.add_argument('artifact', help='a source file or zip/tgz archive')
-
- runmenu = subparsers.add_parser('run', help='send arguments to container to run action')
- runmenu.add_argument('payload', nargs='?', help='the arguments to send to the action, either a reference to a file or an inline JSON object', default=None)
-
- if argcomplete:
- argcomplete.autocomplete(parser)
- return parser.parse_args()
-
-def init(args):
- main = args.main
- artifact = args.artifact
-
- if artifact and (artifact.endswith('.zip') or artifact.endswith('tgz') or artifact.endswith('jar')):
- with open(artifact, 'rb') as fp:
- contents = fp.read()
- contents = base64.b64encode(contents)
- binary = True
- elif artifact is not '':
- with(codecs.open(artifact, 'r', 'utf-8')) as fp:
- contents = fp.read()
- binary = False
- else:
- contents = None
- binary = False
-
- r = requests.post(
- containerRoute(args, 'init'),
- json = {"value": {"code": contents,
- "binary": binary,
- "main": main}})
- print(r.text)
-
-def run(args):
- value = processPayload(args.payload)
- if args.verbose:
- print('Sending value: %s...' % json.dumps(value)[0:40])
- r = requests.post(containerRoute(args, 'run'), json = {"value": value})
- print(r.text)
-
-def processPayload(payload):
- if payload and os.path.exists(payload):
- with open(payload) as fp:
- return json.load(fp)
- try:
- d = json.loads(payload if payload else '{}')
- if isinstance(d, dict):
- return d
- else:
- raise
- except:
- print('payload must be a JSON object.')
- sys.exit(-1)
-
-if __name__ == '__main__':
- main()
diff --git a/actionRuntimes/javaAction/Dockerfile b/actionRuntimes/javaAction/Dockerfile
deleted file mode 100644
index 076f2c5daa..0000000000
--- a/actionRuntimes/javaAction/Dockerfile
+++ /dev/null
@@ -1 +0,0 @@
-FROM openwhisk/java8action:1.0.1
diff --git a/actionRuntimes/javaAction/build.gradle b/actionRuntimes/javaAction/build.gradle
deleted file mode 100644
index 8688fd7ff8..0000000000
--- a/actionRuntimes/javaAction/build.gradle
+++ /dev/null
@@ -1,2 +0,0 @@
-ext.dockerImageName = 'java8action'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/javaAction/delete-build-run.sh b/actionRuntimes/javaAction/delete-build-run.sh
deleted file mode 100755
index 665800f706..0000000000
--- a/actionRuntimes/javaAction/delete-build-run.sh
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/usr/bin/env bash
-
-# Useful for local testing.
-# USE WITH CAUTION !!
-
-# Removes all previously built instances.
-docker rm $(docker ps -a -q)
-
-docker build -t javabox .
-
-echo ""
-echo " ---- RUNNING ---- "
-echo ""
-
-docker run -i -t -p 8080:8080 javabox
diff --git a/actionRuntimes/nodejs6Action/Dockerfile b/actionRuntimes/nodejs6Action/Dockerfile
deleted file mode 100644
index 52ce9ddb26..0000000000
--- a/actionRuntimes/nodejs6Action/Dockerfile
+++ /dev/null
@@ -1,2 +0,0 @@
-FROM openwhisk/nodejs6action:1.5.0
-
diff --git a/actionRuntimes/nodejs6Action/build.gradle b/actionRuntimes/nodejs6Action/build.gradle
deleted file mode 100644
index 8e9275fb45..0000000000
--- a/actionRuntimes/nodejs6Action/build.gradle
+++ /dev/null
@@ -1,10 +0,0 @@
-apply plugin: 'eclipse'
-eclipse {
- project {
- natures 'org.eclipse.wst.jsdt.core.jsNature'
- buildCommand 'org.eclipse.wst.jsdt.core.javascriptValidator'
- }
-}
-
-ext.dockerImageName = 'nodejs6action'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/nodejs8Action/Dockerfile b/actionRuntimes/nodejs8Action/Dockerfile
deleted file mode 100644
index 0fcb9b0298..0000000000
--- a/actionRuntimes/nodejs8Action/Dockerfile
+++ /dev/null
@@ -1,2 +0,0 @@
-FROM openwhisk/action-nodejs-v8:1.2.0
-
diff --git a/actionRuntimes/nodejs8Action/build.gradle b/actionRuntimes/nodejs8Action/build.gradle
deleted file mode 100644
index aa07d2b5ef..0000000000
--- a/actionRuntimes/nodejs8Action/build.gradle
+++ /dev/null
@@ -1,10 +0,0 @@
-apply plugin: 'eclipse'
-eclipse {
- project {
- natures 'org.eclipse.wst.jsdt.core.jsNature'
- buildCommand 'org.eclipse.wst.jsdt.core.javascriptValidator'
- }
-}
-
-ext.dockerImageName = 'action-nodejs-v8'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/php7.1Action/Dockerfile b/actionRuntimes/php7.1Action/Dockerfile
deleted file mode 100644
index 0b6941474f..0000000000
--- a/actionRuntimes/php7.1Action/Dockerfile
+++ /dev/null
@@ -1 +0,0 @@
-FROM openwhisk/action-php-v7.1:1.0.0
diff --git a/actionRuntimes/php7.1Action/build.gradle b/actionRuntimes/php7.1Action/build.gradle
deleted file mode 100644
index 1e4c161911..0000000000
--- a/actionRuntimes/php7.1Action/build.gradle
+++ /dev/null
@@ -1,2 +0,0 @@
-ext.dockerImageName = 'action-php-v7.1'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/python2Action/Dockerfile b/actionRuntimes/python2Action/Dockerfile
deleted file mode 100644
index a3ff63ebde..0000000000
--- a/actionRuntimes/python2Action/Dockerfile
+++ /dev/null
@@ -1 +0,0 @@
-FROM openwhisk/python2action:1.0.0
diff --git a/actionRuntimes/python2Action/build.gradle b/actionRuntimes/python2Action/build.gradle
deleted file mode 100644
index d9823197ac..0000000000
--- a/actionRuntimes/python2Action/build.gradle
+++ /dev/null
@@ -1,2 +0,0 @@
-ext.dockerImageName = 'python2action'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/pythonAction/Dockerfile b/actionRuntimes/pythonAction/Dockerfile
deleted file mode 100644
index 1f67c38b23..0000000000
--- a/actionRuntimes/pythonAction/Dockerfile
+++ /dev/null
@@ -1 +0,0 @@
-FROM openwhisk/python3action:1.0.0
diff --git a/actionRuntimes/pythonAction/build.gradle b/actionRuntimes/pythonAction/build.gradle
deleted file mode 100644
index f3a57241d2..0000000000
--- a/actionRuntimes/pythonAction/build.gradle
+++ /dev/null
@@ -1,2 +0,0 @@
-ext.dockerImageName = 'python3action'
-apply from: '../../gradle/docker.gradle'
diff --git a/actionRuntimes/swift3.1.1Action/Dockerfile b/actionRuntimes/swift3.1.1Action/Dockerfile
deleted file mode 100755
index 1f1de159c7..0000000000
--- a/actionRuntimes/swift3.1.1Action/Dockerfile
+++ /dev/null
@@ -1 +0,0 @@
-FROM openwhisk/action-swift-v3.1.1:1.0.0
diff --git a/actionRuntimes/swift3.1.1Action/build.gradle b/actionRuntimes/swift3.1.1Action/build.gradle
deleted file mode 100755
index 59235b6eb1..0000000000
--- a/actionRuntimes/swift3.1.1Action/build.gradle
+++ /dev/null
@@ -1,3 +0,0 @@
-ext.dockerImageName = 'action-swift-v3.1.1'
-apply from: '../../gradle/docker.gradle'
-
diff --git a/ansible/files/namespace_throttlings_design_document_for_subjects_db.json b/ansible/files/namespace_throttlings_design_document_for_subjects_db.json
new file mode 100644
index 0000000000..23463819c4
--- /dev/null
+++ b/ansible/files/namespace_throttlings_design_document_for_subjects_db.json
@@ -0,0 +1,10 @@
+{
+ "_id": "_design/namespaceThrottlings",
+ "views": {
+ "blockedNamespaces": {
+ "map": "function (doc) {\n if (doc._id.indexOf(\"/limits\") >= 0) {\n if (doc.concurrentInvocations === 0 || doc.invocationsPerMinute === 0) {\n var namespace = doc._id.replace(\"/limits\", \"\");\n emit(namespace, 1);\n }\n } else if (doc.subject && doc.namespaces && doc.blocked) {\n doc.namespaces.forEach(function(namespace) {\n emit(namespace.name, 1);\n });\n }\n}",
+ "reduce": "_sum"
+ }
+ },
+ "language": "javascript"
+}
diff --git a/ansible/group_vars/all b/ansible/group_vars/all
index 3105c5bf1e..423a2c9ae0 100644
--- a/ansible/group_vars/all
+++ b/ansible/group_vars/all
@@ -207,6 +207,9 @@ docker:
image:
prefix: "{{ docker_image_prefix | default('whisk') }}"
tag: "{{ docker_image_tag | default('latest') }}"
+ runtimes:
+ prefix: "{{ runtime_image_prefix | default('openwhisk') }}"
+ tag: "{{ runtime_image_tag | default('latest') }}"
version: 1.12.0-0~trusty
storagedriver: overlay
port: 4243
diff --git a/ansible/roles/invoker/tasks/deploy.yml b/ansible/roles/invoker/tasks/deploy.yml
index bf91a50b71..47b18d9d8f 100644
--- a/ansible/roles/invoker/tasks/deploy.yml
+++ b/ansible/roles/invoker/tasks/deploy.yml
@@ -12,17 +12,17 @@
delay: "{{ docker.pull.delay }}"
- name: "pull runtime action images with tag {{docker.image.tag}}"
- shell: "docker pull {{docker_registry}}{{docker.image.prefix}}/{{item}}:{{docker.image.tag}}"
+ shell: "docker pull {{docker_registry}}{{docker.runtimes.prefix}}/{{item}}:{{docker.runtimes.tag}}"
with_items: "{{ runtimesManifest.runtimes.values() | sum(start=[]) | selectattr('deprecated', 'equalto',false) | map(attribute='image.name') | list | unique }}"
- when: docker_registry != ""
+ #when: docker_registry != ""
retries: "{{ docker.pull.retries }}"
delay: "{{ docker.pull.delay }}"
- name: "pull blackboxes action images with tag {{docker.image.tag}}"
- shell: "docker pull {{docker_registry}}{{docker.image.prefix}}/{{item.name}}:{{docker.image.tag}}"
+ shell: "docker pull {{docker_registry}}{{docker.runtimes.prefix}}/{{item.name}}:{{docker.runtimes.tag}}"
with_items:
- "{{ runtimesManifest.blackboxes }}"
- when: docker_registry != ""
+ #when: docker_registry != ""
retries: "{{ docker.pull.retries }}"
delay: "{{ docker.pull.delay }}"
@@ -169,6 +169,7 @@
-e DB_PASSWORD='{{ db_password }}'
-e DB_WHISK_ACTIONS='{{ db.whisk.actions }}'
-e DB_WHISK_ACTIVATIONS='{{ db.whisk.activations }}'
+ -e DB_WHISK_AUTHS='{{ db.whisk.auth }}'
-e CONFIG_whisk_db_actionsDdoc='{{ db_whisk_actions_ddoc | default() }}'
-e CONFIG_whisk_db_activationsDdoc='{{ db_whisk_activations_ddoc | default() }}'
-e CONFIG_whisk_db_activationsFilterDdoc='{{ db_whisk_activations_filter_ddoc | default() }}'
@@ -176,8 +177,8 @@
-e WHISK_API_HOST_PORT='{{ whisk_api_host_port | default('443') }}'
-e WHISK_API_HOST_NAME='{{ whisk_api_host_name | default(groups['edge'] | first) }}'
-e RUNTIMES_MANIFEST='{{ runtimesManifest | to_json }}'
- -e CONFIG_whisk_runtimes_defaultImagePrefix='{{ runtimes_default_image_prefix | default() }}'
- -e CONFIG_whisk_runtimes_defaultImageTag='{{ runtimes_default_image_tag | default() }}'
+ -e CONFIG_whisk_runtimes_defaultImagePrefix='{{ docker.runtimes.prefix }}'
+ -e CONFIG_whisk_runtimes_defaultImageTag='{{ docker.runtimes.tag }}'
-e CONFIG_whisk_runtimes_bypassPullForLocalImages='{{ runtimes_bypass_pull_for_local_images | default() }}'
-e CONFIG_whisk_runtimes_localImagePrefix='{{ runtimes_local_image_prefix | default() }}'
-e DOCKER_REGISTRY='{{ docker_registry }}'
diff --git a/ansible/tasks/initdb.yml b/ansible/tasks/initdb.yml
index 46db9f8900..5a8665a187 100644
--- a/ansible/tasks/initdb.yml
+++ b/ansible/tasks/initdb.yml
@@ -14,6 +14,7 @@
with_items:
- "{{ openwhisk_home }}/ansible/files/auth_index.json"
- "{{ openwhisk_home }}/ansible/files/filter_design_document.json"
+ - "{{ openwhisk_home }}/ansible/files/namespace_throttlings_design_document_for_subjects_db.json"
- name: create necessary "auth" keys
include: db/recreateDoc.yml
diff --git a/common/scala/Dockerfile b/common/scala/Dockerfile
index 17849dc9fd..3fffa3b89f 100644
--- a/common/scala/Dockerfile
+++ b/common/scala/Dockerfile
@@ -15,19 +15,23 @@ ENV JRE_HOME ${JAVA_HOME}/jre
ENV PATH $JAVA_HOME/bin:$PATH
-RUN apk upgrade --update && \
- apk add --update libstdc++ curl ca-certificates bash sed && \
+RUN apk add --update --no-cache libstdc++ curl ca-certificates bash sed && \
update-ca-certificates && \
- for pkg in glibc-${GLIBC_VERSION} glibc-bin-${GLIBC_VERSION} glibc-i18n-${GLIBC_VERSION}; do curl -sSL https://github.com/andyshinn/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/${pkg}.apk -o /tmp/${pkg}.apk; done && \
+ for pkg in glibc-${GLIBC_VERSION} glibc-bin-${GLIBC_VERSION} glibc-i18n-${GLIBC_VERSION}; do \
+ curl -sSL https://github.com/andyshinn/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/${pkg}.apk -o /tmp/${pkg}.apk \
+ ; done && \
apk add --allow-untrusted /tmp/*.apk && \
rm -v /tmp/*.apk && \
( /usr/glibc-compat/bin/localedef --force --inputfile POSIX --charmap UTF-8 C.UTF-8 || true ) && \
echo "export LANG=${LANG}" > /etc/profile.d/locale.sh && \
/usr/glibc-compat/sbin/ldconfig /lib /usr/glibc-compat/lib && \
- curl --silent --location --retry 3 --header "Cookie: oraclelicense=accept-securebackup-cookie" \
- http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz \
- | tar xz -C /tmp && \
- mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" && \
+ ( cd /tmp && \
+ curl -O --silent --location --retry 3 --header "Cookie: oraclelicense=accept-securebackup-cookie" \
+ http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz && \
+ tar xzf "/tmp/jdk-${VERSION}u${UPDATE}-linux-x64.tar.gz" && \
+ rm "/tmp/jdk-${VERSION}u${UPDATE}-linux-x64.tar.gz" \
+ ) && \
+ mkdir -p /usr/lib/jvm && mv "/tmp/jdk1.${VERSION}.0_${UPDATE}" "${JAVA_HOME}" && \
rm -rf "$JAVA_HOME"/*src.zip \
"$JAVA_HOME"/lib/missioncontrol \
"$JAVA_HOME"/lib/visualvm \
@@ -75,4 +79,4 @@ COPY transformEnvironment.sh /
RUN chmod +x transformEnvironment.sh
COPY copyJMXFiles.sh /
-RUN chmod +x copyJMXFiles.sh
\ No newline at end of file
+RUN chmod +x copyJMXFiles.sh
diff --git a/common/scala/src/main/scala/whisk/core/WhiskConfig.scala b/common/scala/src/main/scala/whisk/core/WhiskConfig.scala
index 1ff53fb6c9..a5e1fe1f19 100644
--- a/common/scala/src/main/scala/whisk/core/WhiskConfig.scala
+++ b/common/scala/src/main/scala/whisk/core/WhiskConfig.scala
@@ -256,6 +256,7 @@ object ConfigKeys {
val runcTimeouts = s"$runc.timeouts"
val containerFactory = "whisk.container-factory"
val containerArgs = s"$containerFactory.container-args"
+ val blacklist = "whisk.blacklist"
val kubernetes = "whisk.kubernetes"
val kubernetesTimeouts = s"$kubernetes.timeouts"
diff --git a/common/scala/src/main/scala/whisk/core/entity/WhiskStore.scala b/common/scala/src/main/scala/whisk/core/entity/WhiskStore.scala
index 0a1417e842..016363a23f 100644
--- a/common/scala/src/main/scala/whisk/core/entity/WhiskStore.scala
+++ b/common/scala/src/main/scala/whisk/core/entity/WhiskStore.scala
@@ -155,7 +155,7 @@ object WhiskActivationStore {
* @param ddoc the design document
* @param view the view name within the design doc
*/
-protected[core] class View(ddoc: String, view: String) {
+protected[core] case class View(ddoc: String, view: String) {
/** The name of the table to query. */
val name = s"$ddoc/$view"
diff --git a/core/invoker/src/main/resources/application.conf b/core/invoker/src/main/resources/application.conf
index c63e1b7992..7f85eb571c 100644
--- a/core/invoker/src/main/resources/application.conf
+++ b/core/invoker/src/main/resources/application.conf
@@ -3,6 +3,10 @@ include "logging"
include "akka-http-version"
whisk {
+ blacklist {
+ poll-interval: 5 minutes
+ }
+
# Timeouts for docker commands. Set to "Inf" to disable timeout.
docker.timeouts {
run: 1 minute
@@ -40,6 +44,5 @@ whisk {
network: bridge
dns-servers: []
extra-args: {}
-
}
}
diff --git a/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainerFactory.scala b/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainerFactory.scala
index ff0dfb5f78..9bc39c7d04 100644
--- a/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainerFactory.scala
+++ b/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainerFactory.scala
@@ -56,11 +56,11 @@ class DockerContainerFactory(config: WhiskConfig,
actionImage: ExecManifest.ImageName,
userProvidedImage: Boolean,
memory: ByteSize)(implicit config: WhiskConfig, logging: Logging): Future[Container] = {
- val image = if (userProvidedImage) {
+ val image = //if (userProvidedImage) {
actionImage.publicImageName
- } else {
- actionImage.localImageName(config.dockerRegistry, config.dockerImagePrefix, Some(config.dockerImageTag))
- }
+ //} else {
+ // actionImage.localImageName(config.dockerRegistry, config.dockerImagePrefix, Some(config.dockerImageTag))
+ //}
DockerContainer.create(
tid,
diff --git a/core/invoker/src/main/scala/whisk/core/invoker/Invoker.scala b/core/invoker/src/main/scala/whisk/core/invoker/Invoker.scala
index 4a9f0715f1..d6d273547f 100644
--- a/core/invoker/src/main/scala/whisk/core/invoker/Invoker.scala
+++ b/core/invoker/src/main/scala/whisk/core/invoker/Invoker.scala
@@ -22,13 +22,10 @@ import scala.concurrent.duration._
import scala.concurrent.Future
import scala.util.Failure
import scala.util.Try
-
import kamon.Kamon
-
import org.apache.curator.retry.RetryUntilElapsed
import org.apache.curator.framework.CuratorFrameworkFactory
import org.apache.curator.framework.recipes.shared.SharedCount
-
import akka.Done
import akka.actor.ActorSystem
import akka.actor.CoordinatedShutdown
@@ -39,10 +36,7 @@ import whisk.core.WhiskConfig
import whisk.core.WhiskConfig._
import whisk.core.connector.MessagingProvider
import whisk.core.connector.PingMessage
-import whisk.core.entity.ExecManifest
-import whisk.core.entity.InstanceId
-import whisk.core.entity.WhiskActivationStore
-import whisk.core.entity.WhiskEntityStore
+import whisk.core.entity._
import whisk.http.BasicHttpService
import whisk.spi.SpiLoader
import whisk.utils.ExecutionContextFactory
@@ -60,6 +54,7 @@ object Invoker {
ExecManifest.requiredProperties ++
WhiskEntityStore.requiredProperties ++
WhiskActivationStore.requiredProperties ++
+ WhiskAuthStore.requiredProperties ++
kafkaHosts ++
zookeeperHosts ++
wskApiHost ++ Map(
diff --git a/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala b/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
index 0c9cdb4504..77c63fa704 100644
--- a/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
+++ b/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
@@ -19,37 +19,27 @@ package whisk.core.invoker
import java.nio.charset.StandardCharsets
import java.time.Instant
-import scala.concurrent.Future
-import scala.concurrent.duration._
-import scala.util.Failure
-import scala.util.Success
-import org.apache.kafka.common.errors.RecordTooLargeException
-import akka.actor.ActorRefFactory
-import akka.actor.ActorSystem
-import akka.actor.Props
+
+import akka.actor.{ActorRefFactory, ActorSystem, Props}
+import akka.event.Logging.InfoLevel
import akka.stream.ActorMaterializer
+import org.apache.kafka.common.errors.RecordTooLargeException
+import pureconfig._
import spray.json._
-import whisk.common.Logging
-import whisk.common.LoggingMarkers
-import whisk.common.TransactionId
-import whisk.core.WhiskConfig
-import whisk.core.connector.ActivationMessage
-import whisk.core.connector.CompletionMessage
-import whisk.core.connector.MessageFeed
-import whisk.core.connector.MessageProducer
-import whisk.core.connector.MessagingProvider
-import whisk.core.containerpool.ContainerFactoryProvider
-import whisk.core.containerpool.ContainerPool
-import whisk.core.containerpool.ContainerProxy
-import whisk.core.containerpool.PrewarmingConfig
-import whisk.core.containerpool.Run
+import whisk.common.{Logging, LoggingMarkers, Scheduler, TransactionId}
+import whisk.core.{ConfigKeys, WhiskConfig}
+import whisk.core.connector._
+import whisk.core.containerpool._
import whisk.core.containerpool.logging.LogStoreProvider
import whisk.core.database._
import whisk.core.entity._
import whisk.core.entity.size._
import whisk.http.Messages
import whisk.spi.SpiLoader
-import akka.event.Logging.InfoLevel
+
+import scala.concurrent.Future
+import scala.concurrent.duration._
+import scala.util.{Failure, Success}
class InvokerReactive(config: WhiskConfig, instance: InstanceId, producer: MessageProducer)(
implicit actorSystem: ActorSystem,
@@ -87,6 +77,17 @@ class InvokerReactive(config: WhiskConfig, instance: InstanceId, producer: Messa
/** Initialize needed databases */
private val entityStore = WhiskEntityStore.datastore(config)
private val activationStore = WhiskActivationStore.datastore(config)
+ private val authStore = WhiskAuthStore.datastore(config)
+
+ private val namespaceBlacklist = new NamespaceBlacklist(authStore)
+
+ Scheduler.scheduleWaitAtMost(loadConfigOrThrow[NamespaceBlacklistConfig](ConfigKeys.blacklist).pollInterval) { () =>
+ logging.debug(this, "running background job to update blacklist")
+ namespaceBlacklist.refreshBlacklist()(ec, TransactionId.invoker).andThen {
+ case Success(set) => logging.info(this, s"updated blacklist to ${set.size} entries")
+ case Failure(t) => logging.error(this, s"error on updating the blacklist: ${t.getMessage}")
+ }
+ }
/** Initialize message consumers */
val topic = s"invoker${instance.toInt}"
@@ -160,7 +161,14 @@ class InvokerReactive(config: WhiskConfig, instance: InstanceId, producer: Messa
/** Is called when an ActivationMessage is read from Kafka */
def processActivationMessage(bytes: Array[Byte]): Future[Unit] = {
Future(ActivationMessage.parse(new String(bytes, StandardCharsets.UTF_8)))
- .flatMap(Future.fromTry(_))
+ .flatMap(Future.fromTry)
+ .flatMap { msg =>
+ if (!namespaceBlacklist.isBlacklisted(msg.user)) {
+ Future.successful(msg)
+ } else {
+ Future.failed(NamespaceBlacklistedException(msg.user.namespace.name))
+ }
+ }
.filter(_.action.version.isDefined)
.flatMap { msg =>
implicit val transid = msg.transid
@@ -235,7 +243,10 @@ class InvokerReactive(config: WhiskConfig, instance: InstanceId, producer: Messa
// Iff everything above failed, we have a terminal error at hand. Either the message failed
// to deserialize, or something threw an error where it is not expected to throw.
activationFeed ! MessageFeed.Processed
- logging.error(this, s"terminal failure while processing message: $t")
+ t match {
+ case nse: NamespaceBlacklistedException => logging.warn(this, nse.getMessage)
+ case _ => logging.error(this, s"terminal failure while processing message: $t")
+ }
Future.successful(())
}
}
diff --git a/core/invoker/src/main/scala/whisk/core/invoker/NamespaceBlacklist.scala b/core/invoker/src/main/scala/whisk/core/invoker/NamespaceBlacklist.scala
new file mode 100644
index 0000000000..9909c82dfd
--- /dev/null
+++ b/core/invoker/src/main/scala/whisk/core/invoker/NamespaceBlacklist.scala
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package whisk.core.invoker
+
+import whisk.common.TransactionId
+import whisk.core.database.StaleParameter
+import whisk.core.entity.{Identity, View}
+import whisk.core.entity.types.AuthStore
+
+import scala.concurrent.{ExecutionContext, Future}
+import spray.json.DefaultJsonProtocol._
+
+import scala.concurrent.duration.FiniteDuration
+
+/**
+ * The namespace blacklist gets all namespaces that are throttled to 0 or blocked from the database.
+ *
+ * The caller is responsible for periodically updating the blacklist with `refreshBlacklist`.
+ *
+ * @param authStore Subjects database with the limit-documents.
+ */
+class NamespaceBlacklist(authStore: AuthStore) {
+
+ private var blacklist: Set[String] = Set.empty
+
+ /**
+ * Check if the identity, who invoked the activation is in the blacklist.
+ *
+ * @param identity which invoked the action.
+ * @return whether or not the current identity is considered blacklisted
+ */
+ def isBlacklisted(identity: Identity): Boolean = blacklist.contains(identity.namespace.name)
+
+ /** Refreshes the current blacklist from the database. */
+ def refreshBlacklist()(implicit ec: ExecutionContext, tid: TransactionId): Future[Set[String]] = {
+ authStore
+ .query(
+ table = NamespaceBlacklist.view.name,
+ startKey = List.empty,
+ endKey = List.empty,
+ skip = 0,
+ limit = Int.MaxValue,
+ includeDocs = false,
+ descending = true,
+ reduce = false,
+ stale = StaleParameter.UpdateAfter)
+ .map(_.map(_.fields("key").convertTo[String]).toSet)
+ .map { newBlacklist =>
+ blacklist = newBlacklist
+ newBlacklist
+ }
+ }
+}
+
+object NamespaceBlacklist {
+ val view = View("namespaceThrottlings", "blockedNamespaces")
+}
+
+/** Configuration relevant to the namespace blacklist */
+case class NamespaceBlacklistConfig(pollInterval: FiniteDuration)
+
+/** Indicates the activation was stopped due to a blacklisted identity */
+case class NamespaceBlacklistedException(ns: String) extends Exception(s"Namespace $ns was blocked in invoker.")
diff --git a/tests/dat/blackbox/badaction/Dockerfile b/tests/dat/blackbox/badaction/Dockerfile
index f342e3d182..136e200fea 100644
--- a/tests/dat/blackbox/badaction/Dockerfile
+++ b/tests/dat/blackbox/badaction/Dockerfile
@@ -1,6 +1,6 @@
# Dockerfile for example whisk docker action
-FROM dockerskeleton
-
+FROM openwhisk/dockerskeleton
+
ENV FLASK_PROXY_PORT 8080
ADD runner.py /actionProxy/
diff --git a/tests/dat/blackbox/badaction/build.gradle b/tests/dat/blackbox/badaction/build.gradle
index b5921ba9aa..728eaf2d87 100644
--- a/tests/dat/blackbox/badaction/build.gradle
+++ b/tests/dat/blackbox/badaction/build.gradle
@@ -1,4 +1,4 @@
ext.dockerImageName = 'badaction'
apply from: '../../../../gradle/docker.gradle'
-distDocker.dependsOn ':actionRuntimes:actionProxy:distDocker'
+//distDocker.dependsOn ':actionRuntimes:actionProxy:distDocker'
diff --git a/tests/dat/blackbox/badproxy/Dockerfile b/tests/dat/blackbox/badproxy/Dockerfile
index f483a2c385..4b6e91006f 100644
--- a/tests/dat/blackbox/badproxy/Dockerfile
+++ b/tests/dat/blackbox/badproxy/Dockerfile
@@ -1,6 +1,6 @@
# Dockerfile for example whisk docker action
-FROM dockerskeleton
-
+FROM openwhisk/dockerskeleton
+
ENV FLASK_PROXY_PORT 8080
CMD ["/bin/bash", "-c", "tail -f /dev/null"]
diff --git a/tests/dat/blackbox/badproxy/build.gradle b/tests/dat/blackbox/badproxy/build.gradle
index 6de5f5e307..6e2e61dfb1 100644
--- a/tests/dat/blackbox/badproxy/build.gradle
+++ b/tests/dat/blackbox/badproxy/build.gradle
@@ -1,4 +1,4 @@
ext.dockerImageName = 'badproxy'
apply from: '../../../../gradle/docker.gradle'
-distDocker.dependsOn ':actionRuntimes:actionProxy:distDocker'
+//distDocker.dependsOn ':actionRuntimes:actionProxy:distDocker'
diff --git a/tests/src/test/scala/actionContainers/ActionContainer.scala b/tests/src/test/scala/actionContainers/ActionContainer.scala
index 4f6ef24347..ccd6748be6 100644
--- a/tests/src/test/scala/actionContainers/ActionContainer.scala
+++ b/tests/src/test/scala/actionContainers/ActionContainer.scala
@@ -144,17 +144,21 @@ object ActionContainer {
}
// ...find out its IP address...
- val (ip, port) = if (WhiskProperties.getProperty("whisk.version.name") == "local") {
- val p = 8988 // port must be available or docker run will fail
- createContainer(Some(p))
- Thread.sleep(1500) // let container/server come up cleanly
- ("localhost", p)
- } else { // "mac"
- createContainer()
- val ipOut = awaitDocker(s"""inspect --format '{{.NetworkSettings.IPAddress}}' $name""", 10 seconds)
- assert(ipOut._1 == 0, "'docker inspect did not exit with 0")
- (ipOut._2.replaceAll("""[^0-9.]""", ""), 8080)
- }
+ val (ip, port) =
+ if (WhiskProperties.getProperty("whisk.version.name") == "local" &&
+ WhiskProperties.onMacOSX()) {
+ // on MacOSX, where docker for mac does not permit communicating with container directly
+ val p = 8988 // port must be available or docker run will fail
+ createContainer(Some(p))
+ Thread.sleep(1500) // let container/server come up cleanly
+ ("localhost", p)
+ } else {
+ // not "mac" i.e., docker-for-mac, use direct container IP directly (this is OK for Ubuntu, and docker-machine)
+ createContainer()
+ val ipOut = awaitDocker(s"""inspect --format '{{.NetworkSettings.IPAddress}}' $name""", 10 seconds)
+ assert(ipOut._1 == 0, "'docker inspect did not exit with 0")
+ (ipOut._2.replaceAll("""[^0-9.]""", ""), 8080)
+ }
// ...we create an instance of the mock container interface...
val mock = new ActionContainer {
diff --git a/tests/src/test/scala/whisk/core/invoker/test/NamespaceBlacklistTests.scala b/tests/src/test/scala/whisk/core/invoker/test/NamespaceBlacklistTests.scala
new file mode 100644
index 0000000000..89670f1f9c
--- /dev/null
+++ b/tests/src/test/scala/whisk/core/invoker/test/NamespaceBlacklistTests.scala
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package whisk.core.invoker.test
+
+import akka.stream.ActorMaterializer
+import common.{StreamLogging, WhiskProperties, WskActorSystem}
+import org.junit.runner.RunWith
+import org.scalatest.concurrent.{IntegrationPatience, ScalaFutures}
+import org.scalatest.junit.JUnitRunner
+import org.scalatest.{FlatSpec, Matchers}
+import spray.json.DefaultJsonProtocol._
+import spray.json._
+import whisk.common.TransactionId
+import whisk.core.WhiskConfig
+import whisk.core.database.test.{DbUtils, ExtendedCouchDbRestClient}
+import whisk.core.entity._
+import whisk.core.invoker.NamespaceBlacklist
+
+import scala.concurrent.duration._
+
+@RunWith(classOf[JUnitRunner])
+class NamespaceBlacklistTests
+ extends FlatSpec
+ with Matchers
+ with DbUtils
+ with ScalaFutures
+ with IntegrationPatience
+ with WskActorSystem
+ with StreamLogging {
+
+ behavior of "NamespaceBlacklist"
+
+ val config = new WhiskConfig(WhiskAuthStore.requiredProperties)
+
+ implicit val materializer = ActorMaterializer()
+ implicit val tid = TransactionId.testing
+
+ val authStore = WhiskAuthStore.datastore(config)
+ val subjectsDb = new ExtendedCouchDbRestClient(
+ WhiskProperties.getProperty(WhiskConfig.dbProtocol),
+ WhiskProperties.getProperty(WhiskConfig.dbHost),
+ WhiskProperties.getProperty(WhiskConfig.dbPort).toInt,
+ WhiskProperties.getProperty(WhiskConfig.dbUsername),
+ WhiskProperties.getProperty(WhiskConfig.dbPassword),
+ WhiskProperties.getProperty(WhiskConfig.dbAuths))
+
+ /* Identities needed for the first test */
+ val identities = Seq(
+ Identity(Subject(), EntityName("testnamespace1"), AuthKey(), Set.empty, UserLimits(invocationsPerMinute = Some(0))),
+ Identity(
+ Subject(),
+ EntityName("testnamespace2"),
+ AuthKey(),
+ Set.empty,
+ UserLimits(concurrentInvocations = Some(0))),
+ Identity(
+ Subject(),
+ EntityName("testnamespace3"),
+ AuthKey(),
+ Set.empty,
+ UserLimits(invocationsPerMinute = Some(1), concurrentInvocations = Some(1))))
+
+ /* Subject document needed for the second test */
+ val subject = WhiskAuth(
+ Subject(),
+ Set(WhiskNamespace(EntityName("different1"), AuthKey()), WhiskNamespace(EntityName("different2"), AuthKey())))
+ val blockedSubject = JsObject(subject.toJson.fields + ("blocked" -> true.toJson))
+
+ val blockedNamespacesCount = 2 + subject.namespaces.size
+
+ def authToIdentities(auth: WhiskAuth): Set[Identity] = {
+ auth.namespaces.map { ns =>
+ Identity(auth.subject, ns.name, ns.authkey, Set(), UserLimits())
+ }
+ }
+
+ override def beforeAll() = {
+ val documents = identities.map { i =>
+ (i.namespace.name + "/limits", i.limits.toJson.asJsObject)
+ } :+ (subject.subject.asString, blockedSubject)
+
+ // Add all documents to the database
+ documents.foreach { case (id, doc) => subjectsDb.putDoc(id, doc).futureValue }
+
+ // Waits for the 2 blocked identities + the namespaces of the blocked subject
+ waitOnView(subjectsDb, NamespaceBlacklist.view.ddoc, NamespaceBlacklist.view.view, blockedNamespacesCount)(
+ executionContext,
+ 1.minute)
+ }
+
+ override def afterAll() = {
+ val ids = identities.map(_.namespace.name + "/limits") :+ subject.subject.asString
+
+ // Force remove all documents with those ids by first getting and then deleting the documents
+ ids.foreach { id =>
+ val docE = subjectsDb.getDoc(id).futureValue
+ docE shouldBe 'right
+ val doc = docE.right.get
+ subjectsDb
+ .deleteDoc(doc.fields("_id").convertTo[String], doc.fields("_rev").convertTo[String])
+ .futureValue
+ }
+
+ super.afterAll()
+ }
+
+ it should "mark a namespace as blocked if limit is 0 in database or if one of its subjects is blocked" in {
+ val blacklist = new NamespaceBlacklist(authStore)
+
+ blacklist.refreshBlacklist().futureValue should have size blockedNamespacesCount
+
+ identities.map(blacklist.isBlacklisted) shouldBe Seq(true, true, false)
+ authToIdentities(subject).toSeq.map(blacklist.isBlacklisted) shouldBe Seq(true, true)
+ }
+}
diff --git a/tools/travis/build.sh b/tools/travis/build.sh
index c98285f1d4..a962cb2601 100755
--- a/tools/travis/build.sh
+++ b/tools/travis/build.sh
@@ -30,7 +30,6 @@ fi
cd $ROOTDIR/ansible
ANSIBLE_CMD="ansible-playbook -i environments/local -e docker_image_prefix=testing"
-GRADLE_PROJS_SKIP="-x :actionRuntimes:pythonAction:distDocker -x :actionRuntimes:python2Action:distDocker -x actionRuntimes:swift3.1.1Action:distDocker -x :actionRuntimes:javaAction:distDocker"
$ANSIBLE_CMD setup.yml -e mode=HA
$ANSIBLE_CMD prereq.yml
@@ -40,7 +39,7 @@ $ANSIBLE_CMD apigateway.yml
cd $ROOTDIR
-TERM=dumb ./gradlew distDocker -PdockerImagePrefix=testing $GRADLE_PROJS_SKIP
+TERM=dumb ./gradlew distDocker -PdockerImagePrefix=testing
cd $ROOTDIR/ansible
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services