You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by "Kevin Su (Jira)" <ji...@apache.org> on 2022/01/16 09:18:00 UTC
[jira] [Resolved] (SUBMARINE-1124) The automatic deletion of notebook does not delete the related PVC and IngressRoute
[ https://issues.apache.org/jira/browse/SUBMARINE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Su resolved SUBMARINE-1124.
---------------------------------
Resolution: Fixed
Issue resolved by pull request 855
[https://github.com/apache/submarine/pull/855]
> The automatic deletion of notebook does not delete the related PVC and IngressRoute
> -----------------------------------------------------------------------------------
>
> Key: SUBMARINE-1124
> URL: https://issues.apache.org/jira/browse/SUBMARINE-1124
> Project: Apache Submarine
> Issue Type: Bug
> Components: K8s Submitter
> Reporter: cdmikechen
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.7.0
>
> Attachments: image-2021-12-10-09-56-32-491.png, screenshot-1.png
>
>
> I use newest submarine server to create a notebook called *test*. Notebook pod can not start by this error with image *apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT*
> !image-2021-12-10-09-56-32-491.png!
> Submarine will consistently display the status of the notebook in waiting.
> !screenshot-1.png!
> After a long time, submarine will automatically delete the notebook row in the database. I see the corresponding log information in the background.
> {code}
> [INFO ] 2021-12-10 09:23:25,015 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> Checked out connection 1073981532 from pool.
> Testing connection 1073981532 ...
> Connection 1073981532 is GOOD!
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> Testing connection 1073981532 ...
> Connection 1073981532 is GOOD!
> Returned connection 1073981532 to pool.
> [INFO ] 2021-12-10 09:23:25,593 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:25,602 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [INFO ] 2021-12-10 09:23:25,904 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:25,918 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [WARN ] 2021-12-10 09:23:27,257 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> [INFO ] 2021-12-10 09:23:27,279 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> Checked out connection 1073981532 from pool.
> Testing connection 1073981532 ...
> [INFO ] 2021-12-10 09:23:27,283 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> Connection 1073981532 is GOOD!
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> [WARN ] 2021-12-10 09:23:28,972 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> [INFO ] 2021-12-10 09:23:29,221 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> [INFO ] 2021-12-10 09:23:30,257 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> [INFO ] 2021-12-10 09:23:31,273 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> [INFO ] 2021-12-10 09:23:32,683 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> Created connection 1441218336.
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> Created connection 875758614.
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> Created connection 616832400.
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> Created connection 670559153.
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> Created connection 2081921914.
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
> Testing connection 1073981532 ...
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> Connection 1073981532 is GOOD!
> Returned connection 1073981532 to pool.
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
> Testing connection 875758614 ...
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
> Connection 875758614 is GOOD!
> Returned connection 875758614 to pool.
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
> Testing connection 616832400 ...
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
> <== Columns: id, notebook_spec, create_by, create_time, update_by, update_time
> <== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
> Connection 616832400 is GOOD!
> Returned connection 616832400 to pool.
> Testing connection 2081921914 ...
> <== Total: 1
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
> Connection 2081921914 is GOOD!
> Returned connection 2081921914 to pool.
> Testing connection 670559153 ...
> Connection 670559153 is GOOD!
> Returned connection 670559153 to pool.
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
> Testing connection 1441218336 ...
> <== Updates: 1
> Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
> <== Updates: 0
> Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
> Connection 1441218336 is GOOD!
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
> Closed connection 1441218336.
> Testing connection 1103188293 ...
> Connection 1103188293 is GOOD!
> Closed connection 1103188293.
> Testing connection 307179679 ...
> [WARN ] 2021-12-10 09:23:35,350 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> Connection 307179679 is GOOD!
> Closed connection 307179679.
> [WARN ] 2021-12-10 09:23:35,352 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> [INFO ] 2021-12-10 09:23:38,326 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:38,334 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [INFO ] 2021-12-10 09:23:38,338 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:38,342 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [INFO ] 2021-12-10 09:23:40,057 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:40,062 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [INFO ] 2021-12-10 09:23:42,452 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:42,458 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [INFO ] 2021-12-10 09:23:42,705 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
> Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
> [INFO ] 2021-12-10 09:23:42,711 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
> notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
> [WARN ] 2021-12-10 09:23:43,055 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> [WARN ] 2021-12-10 09:23:43,065 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> [INFO ] 2021-12-10 09:23:43,078 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> Checked out connection 1073981532 from pool.
> Testing connection 1073981532 ...
> Connection 1073981532 is GOOD!
> Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> ==> Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook
> ==> Parameters:
> <== Total: 0
> Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
> Testing connection 1073981532 ...
> Connection 1073981532 is GOOD!
> Returned connection 1073981532 to pool.
> [WARN ] 2021-12-10 09:23:43,083 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
> Unsupported parse class java.util.ArrayList Dict Annotation!
> [INFO ] 2021-12-10 09:23:43,085 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
> Notebook selectAll
> Opening JDBC Connection
> Checked out connection 875758614 from pool.
> {code}
> I checked codes and found that submarine just delete database row but it does not delete the resource like CRD, PVC and IngressRoute. So that if I recreate with a duplicate name, submarine tell me I can not create this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org