You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by "RKuttruff (via GitHub)" <gi...@apache.org> on 2023/04/11 20:53:40 UTC
[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #241: SDAP-457 - Script to purge DOMS execution data
RKuttruff opened a new pull request, #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241
Usage:
`python purge.py -u USERNAME -p PASSWORD [CASSANDRA ARGS...] (--before DATETIME | --before-months MONTHS | --keep-completed | --all) [--keep-failed] [--dry-run]`
Options:
- `-u & -p`: Cassandra username and password
- One of:
- `--before`: Datetime (ie, '2023-04-11T19:50:00') before which all data will be purged.
- `--before-months`: Number of months prior to current date before which all data will be purged.
- `--keep-completed`: Keep all completed executions. (only purge uncompleted executions)
- `--all`: Purge ALL data. (drops and re-creates keyspace)
- Cassandra args (optional)
- `--cassandra`: Cassandra hostname(s) or IP(s). (Default: localhost)
- `-k / --cassandraKeyspace`: Cassandra keyspace for storing DOMS data. (Default: doms)
- `--cassandraPort`: Port used to connect to Cassandra. (Default: 9042)
- `--cassandraProtocolVersion`: The version of the Cassandra protocol the driver should use. (Default: 3)
- Additional args
- `--keep-failed`: Do not purge uncompleted executions (by default all are purged). Incompatible with `--keep-completed`
- `--dry-run`: Only print the execution ids to be deleted / DB operations to be performed and exit. Do not actually alter the DB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#issuecomment-1505786768
> This is great! I would add a README here: `tools/domspurge/` and add the body from your PR which is really helpful
Added the readme
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "skorper (via GitHub)" <gi...@apache.org>.
skorper commented on code in PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#discussion_r1163443883
##########
tools/domspurge/purge.py:
##########
@@ -0,0 +1,344 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import argparse
+import json
+import logging
+from datetime import datetime
+from typing import Tuple, List
+
+from cassandra.auth import PlainTextAuthProvider
+from cassandra.cluster import Cluster, NoHostAvailable, ExecutionProfile, EXEC_PROFILE_DEFAULT
+from cassandra.policies import RoundRobinPolicy, TokenAwarePolicy
+from dateutil import parser as du_parser
+from dateutil.relativedelta import relativedelta
+from six.moves import input
+from tqdm import tqdm
+
+logging.getLogger('webservice.NexusHandler').setLevel(logging.CRITICAL)
+
+from webservice.algorithms.doms.DomsInitialization import DomsInitializer
+
+
+logging.basicConfig(level=logging.INFO)
Review Comment:
Maybe log to stdout as well since this is a script?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on code in PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#discussion_r1164535822
##########
tools/domspurge/purge.py:
##########
@@ -0,0 +1,344 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import argparse
+import json
+import logging
+from datetime import datetime
+from typing import Tuple, List
+
+from cassandra.auth import PlainTextAuthProvider
+from cassandra.cluster import Cluster, NoHostAvailable, ExecutionProfile, EXEC_PROFILE_DEFAULT
+from cassandra.policies import RoundRobinPolicy, TokenAwarePolicy
+from dateutil import parser as du_parser
+from dateutil.relativedelta import relativedelta
+from six.moves import input
+from tqdm import tqdm
+
+logging.getLogger('webservice.NexusHandler').setLevel(logging.CRITICAL)
+
+from webservice.algorithms.doms.DomsInitialization import DomsInitializer
+
+
+logging.basicConfig(level=logging.INFO)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#issuecomment-1505887732
I also added a means to deploy the script as a `CronJob` in k8s
- Helm template + associated entries in `values.yml`
- `Dockerfile` and `entrypoint.sh` script to run the tool (k8s `CronJob`s are run as `Pod`s thus requiring an image to run)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on code in PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#discussion_r1302215867
##########
tools/domspurge/README.md:
##########
@@ -0,0 +1,37 @@
+# Purge DOMS/CDMS Execution Data
+
+## Prerequisites
+
+_If you are running the `deletebyquery.py` script from within the nexus-webapp-driver Docker image, the following prerequisites are not necessary._
Review Comment:
@ngachung
Yes. I copy/pasted that text from deletebyquery's README and forgot to change it.
Corrected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "skorper (via GitHub)" <gi...@apache.org>.
skorper commented on PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#issuecomment-1504312754
This is great! I would add a README here: `tools/domspurge/` and add the body from your PR which is really helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung commented on a diff in pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "ngachung (via GitHub)" <gi...@apache.org>.
ngachung commented on code in PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#discussion_r1302212992
##########
tools/domspurge/README.md:
##########
@@ -0,0 +1,37 @@
+# Purge DOMS/CDMS Execution Data
+
+## Prerequisites
+
+_If you are running the `deletebyquery.py` script from within the nexus-webapp-driver Docker image, the following prerequisites are not necessary._
Review Comment:
Do you mean running the purge.py script here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241#issuecomment-1505936522
Tested k8s `CronJob` deployment locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff merged pull request #241: SDAP-457 - Script to purge DOMS execution data
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff merged PR #241:
URL: https://github.com/apache/incubator-sdap-nexus/pull/241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org