You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by OpenWhisk Team Slack <ra...@apache.org> on 2019/03/01 07:19:16 UTC
[slack-digest] [2019-02-28] #general
2019-02-28 01:36:18 UTC - Dominic Kim: Hello guys
wave : Carlos Santana, James Thomas
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551317778180100
----
2019-02-28 02:24:02 UTC - Perry Dykes: Folks, I need to process an incoming xml stream. The legacy clients that uploads a Post XML file can't be changed, and attempting OpenWhisk to process using the raw web action function. I have created my tests using in the docs at <https://console.bluemix.net/docs/openwhisk/openwhisk_webactions.html#openwhisk_webactions>. I did have to change the Javascript (using Node) due new Buffer() deprecation:
```
function main(args) {
var decoded = Buffer.from(args.__ow_body, 'base64');
decoded = decoded.toString('utf-8');
console.log("incoming text", decoded);
return { "body": decoded };
}
```
I set the endpoint options to Enable as Web Action and Raw HTTP Handling. Then used the curl invocation (and several variants ;-)) to get a feel for this the raw handling. In net, trying to upload an XML file and processes, but stuck at this step processing a simple string, let alone an xml snippet.
curl -k -H "content-type: application" -X POST -d "some test string to dump out" <url to action>
However, this documentation snippet example seems to simply fail with much error that makes sense, e.g. "The request's Content-Type is not supported. Expected:\napplication/json"}". I had thought the reason for raw was to be able to take a text string. Anyway, missing some newbie thing I guess, tips or links to a xml processing sample and/or ideas on the above appreciated.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551320642185600
----
2019-02-28 02:26:46 UTC - Rodric Rabbah: What url are you using
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551320806186200
----
2019-02-28 03:23:27 UTC - Perry Dykes: @Rodric Rabbah -u <api-key> <https://eu-de.functions.cloud.ibm.com/api/v1/namespaces/IBMSystemsSupport_dev/actions/tss_mvs/CiscoAlert> (basically the url provided in the Endpoints CURL example)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551324207187300
----
2019-02-28 03:38:11 UTC - Rodric Rabbah: A web action has a different url - if the action is a web action, `wsk action get <name> --url` will show you the url to use
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325091188800
----
2019-02-28 03:43:34 UTC - Rodric Rabbah: It has a “web/” in the path.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325414189300
----
2019-02-28 03:45:37 UTC - Perry Dykes: Hmm, ok, anoher public method unprotected is here: <https://eu-de.functions.cloud.ibm.com/api/v1/web/IBMSystemsSupport_dev/tss_mvs/CiscoAlert.json>
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325537189900
----
2019-02-28 03:46:57 UTC - Perry Dykes: Ok, that does work vs. the url I was trying.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325617190400
----
2019-02-28 03:47:27 UTC - Perry Dykes: Guess I have to read up on the differences.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325647190800
----
2019-02-28 03:47:48 UTC - Perry Dykes: seems odd to require I end a uri in .json doesn't it?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325668191200?thread_ts=1551325668.191200&cid=C3TPCAQG1
----
2019-02-28 03:49:50 UTC - Perry Dykes: Thanks for the pointer, that does allow me to process the xml as I had hoped.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325790191600
----
2019-02-28 05:50:58 UTC - Dominic Kim: Is anyone facing this issue?
-1 : Rodric Rabbah
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551333058000200
----
2019-02-28 08:03:41 UTC - chetanm: Currently if we reset any namespace key the old keys are still valid for some time due to cache effect.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341021001100
----
2019-02-28 08:04:29 UTC - chetanm: Should the cache entry for Identity be also have invalidation support such that upon expiry/invalidation of any namespace key system admin service can trigger a cache invalidation message?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341069002100
----
2019-02-28 08:07:17 UTC - Markus Thömmes: @chetanm it could, however the controller does not know about that invalidation, does it?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341237002700
----
2019-02-28 08:08:14 UTC - chetanm: yup … so whatever admin service is being used by system which does the namespace invalidation would need to push an explicit message to `cacheInvalidation` topic
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341294003600
----
2019-02-28 08:08:44 UTC - chetanm: This would though not work where we are directly using wskadmin python script
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341324004000
----
2019-02-28 08:08:52 UTC - chetanm: may be for wskadmin-next we can add
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341332004300
----
2019-02-28 08:09:25 UTC - Markus Thömmes: ooooor use this as a segway for your "cache invalidation through database changes feed" proposal? :slightly_smiling_face:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341365004700
----
2019-02-28 08:10:15 UTC - chetanm: That was a by product I was looking for :slightly_smiling_face:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341415005300
----
2019-02-28 08:10:47 UTC - chetanm: In testing we observed that invalidation is not effective immediately hence was thinking that would also get solved once we have such invalidation support
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341447006000
----
2019-02-28 08:11:06 UTC - chetanm: but realized that current logic only invalidates `whisks` related caches
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341466006500
----
2019-02-28 08:11:43 UTC - Markus Thömmes: I'd rather invest in that vs. investing in adding kafka stuff to wskadmin
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341503007200
----
2019-02-28 08:12:32 UTC - chetanm: sure … so add support in `RemoteCacheInvalidation` to look for message to invalidate entries from Identity cache also
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341552008000
----
2019-02-28 08:12:46 UTC - chetanm: and then have a feed support to push them
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341566008500
----
2019-02-28 08:12:50 UTC - Markus Thömmes: now, invalidation will never be immediately consistent unless you wait for all controllers to have picked it up
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341570008600
----
2019-02-28 08:13:42 UTC - chetanm: yeah … just that with namespace and security … we may like to make it earlier …. anyways it would expire after 5 mins. So current situation is somewhat ok
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341622009600
----
2019-02-28 08:13:57 UTC - Carlos Santana: Don’t use an extension it would be the same a .http
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341637010200?thread_ts=1551325668.191200&cid=C3TPCAQG1
----
2019-02-28 08:15:01 UTC - Markus Thömmes: makes sense
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341701010900
----
2019-02-28 09:20:58 UTC - Rodric Rabbah: Is setting up a feed more work vs just posting an explicit message to Kafka - I mean that we already have code for doing that which wskadmin-next can use don’t we?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551345658013700
----
2019-02-28 09:29:35 UTC - Markus Thömmes: It's more of a generalization concern on my end. The feed work is valuable in a more general way. With Kafka messages, you need to make sure the admin tool actually runs in-cluster etc etc
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346175014400
----
2019-02-28 09:31:42 UTC - chetanm: feed is lot more generic. Further I am not sure how system operators are handling the user part (I presume wskadmin is more for dev purpose). For our case we have a separate service which manipulates the user db
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346302015600
----
2019-02-28 09:33:56 UTC - Markus Thömmes: so a changes feed answers the more general question of "inform me whenever something happens in the database", which can be useful for all kinds of database manipulations (although that should be discouraged of course)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346436016500
----
2019-02-28 09:43:18 UTC - Rodric Rabbah: what about just an http request - I’m just thinking - if this is a low probability event - is a much simpler solution is adequate vs a more general but potentially laborious solution
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346998018700
----
2019-02-28 09:45:06 UTC - Rodric Rabbah: (i will read the proposal in the wiki - I haven’t yet)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551347106021700
----
2019-02-28 09:45:55 UTC - chetanm: @Rodric Rabbah Feed approach here is in context of a broader usecase around multi region support. <https://github.com/apache/incubator-openwhisk/issues/4277#cache-invalidation-service>
While doing that I thought to use same approach to invalidate namespace keys upon invalidation … whihc started this thread
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551347155022800
----
2019-02-28 09:48:47 UTC - Rodric Rabbah: :+1: clearly I should have read the wiki first
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551347327023400
----
2019-02-28 10:04:41 UTC - Dominic Kim: Is it possible for other databases which do not support something like `_changes` API in CouchDB to relay on the "feed"?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551348281025400
----
2019-02-28 10:05:29 UTC - Dominic Kim: I hope this is handled in a database-independent way.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551348329026100
----
2019-02-28 10:27:36 UTC - Markus Thömmes: @Dominic Kim which database would come to your mind? Relational databases usually have a "trigger" concept for that kind of thing
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349656027000
----
2019-02-28 10:28:25 UTC - Dominic Kim: Actually, we here are using ElasticSearch as a database.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349705027500
----
2019-02-28 10:30:03 UTC - Markus Thömmes: I'd guess we'd have an interface to implement per database to obtain that kind of information. We could leave Kafka based invalidation in-place as well
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349803028200
----
2019-02-28 10:32:38 UTC - Dominic Kim: That means, in case the database does not support any event notification feature, it can still rely on Kafka based invalidation?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349958030800
----
2019-02-28 10:33:06 UTC - chetanm: To clarify some aspect - Proposal here was more for invalidating cache in multi cluster setup which share same DB. So idea here was (in case of CosmosDB) we can make use of change feed to detect change in db and convert them as events to each clusters `cacheInvalidation` topic. So kafka based invalidation is still used
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349986031300
----
2019-02-28 10:33:26 UTC - Dominic Kim: Yes.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350006031800
----
2019-02-28 10:34:12 UTC - chetanm: As those multi clusters do not share kafka we need a way to ensure that changes made in entities are flushed out. Which work fine in single cluster setup as Controller which handled the change pushes the event to `cacheInvalidation` topic.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350052033200
----
2019-02-28 10:34:42 UTC - chetanm: In MR setups this needs to be done by a separate service which uses some sort of change feed if possible. So solution would be db specific
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350082033900
----
2019-02-28 10:36:28 UTC - Dominic Kim: @chetanm So you keep the independent service which listens on the changes event from the database in your mind?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350188034800
----
2019-02-28 10:36:42 UTC - chetanm: yup
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350202035000
----
2019-02-28 10:36:46 UTC - Dominic Kim: got it.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350206035200
----
2019-02-28 10:36:59 UTC - Dominic Kim: I thought controllers are receiving that kinds of events.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350219035600
----
2019-02-28 10:37:53 UTC - Dominic Kim: In that case, we can just implement the separate service to achieve the same thing though the database does not support such "feed" feature. Maybe relying on long-polling or something.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350273037300
----
2019-02-28 10:38:12 UTC - chetanm: So controller which say recived the change request for action A1 would make change to db and would also push an event to cacheInvalidation topic. This would then be picked by other Controllers and they would evict that entry from there respective caches
white_check_mark : Dominic Kim
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350292037700
----
2019-02-28 10:39:03 UTC - chetanm: Now when we have multiple such clusters then we need a way to project such events in other cluster (separate from one where the actual change happened)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350343038600
----
2019-02-28 10:39:30 UTC - chetanm: Thats where this service would come in and do this db change feed to cache invalidation event conversion
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350370039200
----
2019-02-28 10:39:55 UTC - Dominic Kim: Yes because each cluster does not share the same Kafka.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350395039800
----
2019-02-28 10:40:30 UTC - Markus Thömmes: For full disclosure: My proposal towards that was to ditch the Kafka topic entirely and only rely on database based notification for cache invalidation period
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350430041000
----
2019-02-28 10:40:33 UTC - chetanm: Yes. Other approach can be to replicate the kafka cluster topic … which in our case we are avoiding as Cosmosdb serve similar purpose easily
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350433041200
----
2019-02-28 10:41:09 UTC - Markus Thömmes: I think it's much more elegant than the current Kafka design
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350469041500
----
2019-02-28 10:41:55 UTC - Dominic Kim: @Markus Thömmes Then you meant, we can relay on Kafka iff the database does not support such a feature?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350515042200
----
2019-02-28 10:42:13 UTC - Markus Thömmes: if we make it pluggable in the right spots, sure
white_check_mark : Dominic Kim
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350533042600
----
2019-02-28 10:42:38 UTC - Markus Thömmes: I'd prefer doing that over creating a seperate service that transforms db events to kafka events
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350558043100
----
2019-02-28 10:44:12 UTC - chetanm: hmm … if each controller pulls in the change feed then it would put load on db which incurs cost in ComsosDB. So using kafka for local cache sync and feed for cross cluster may work better
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350652044300
----
2019-02-28 10:48:33 UTC - Markus Thömmes: The changes feed is expensive?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350913044500
----
2019-02-28 11:41:17 UTC - chetanm: It has some cost. Each document read adds to the read units consumed. Further each feed consumer needs to manage it's offset ... Kind of similar to how Kafka consumer works. The offset are again stored in a separate collection in CosmosDB
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551354077046600
----
2019-02-28 15:00:59 UTC - Olivier Tardieu: openwhisk-composer 0.10.0 has been released and is available from npm and github:
Changelog:
- Add new parallel and map combinators to run compositions in parallel using a Redis instance to store intermediate results.
- Add dynamic combinator to invoke an action with a name chosen at run time.
- Add option to bypass TLS certificate validation failures (off by default).
- Add API to generate the conductor action code from a composition.
- Add control over needle options and logging.
Download links:
- <https://www.npmjs.com/package/openwhisk-composer>
- <https://github.com/apache/incubator-openwhisk-composer/releases>
- <http://openwhisk.incubator.apache.org/downloads.html>
tada : Carlos Santana, Matt Sicker, James Thomas, chetanm, mmarth, seonghyun, dubee
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551366059047800
----