You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/21 00:32:02 UTC
(texera) branch gh-readonly-queue/main/pr-7789-4842e93f2d274399eb9baa9594d3f231b00ec519 created (now bc94796189)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-7789-4842e93f2d274399eb9baa9594d3f231b00ec519
in repository https://gitbox.apache.org/repos/asf/texera.git
at bc94796189 feat(storage): singularize the resource-type prefix on logical paths (#7789)
This branch includes the following new commits:
new bc94796189 feat(storage): singularize the resource-type prefix on logical paths (#7789)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(texera) 01/01: feat(storage): singularize the resource-type prefix on logical paths (#7789)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7789-4842e93f2d274399eb9baa9594d3f231b00ec519
in repository https://gitbox.apache.org/repos/asf/texera.git
commit bc9479618953a24b057b4e7ce4d13e9af71775a2
Author: Tanishq Gandhi <56...@users.noreply.github.com>
AuthorDate: Fri Aug 21 00:31:39 2026 +0000
feat(storage): singularize the resource-type prefix on logical paths (#7789)
### What changes were proposed in this PR?
The leading segment of a versioned-resource logical path was plural
while every name it refers to is singular — the backing tables
(`dataset`, `model`), the dashboard search's resource type, and the
hub's entity type. This makes the
prefix match them.
```
Before: /datasets/bob@x.com/twitter/v1/f.csv table: dataset
/models/bob@x.com/resnet/v1/w.pt table: model
After: /dataset/bob@x.com/twitter/v1/f.csv table: dataset
/model/bob@x.com/resnet/v1/w.pt table: model
```
| Layer | Change |
| --- | --- |
| Enums | `ResourceType` in Scala (`Datasets`/`Models` →
`Dataset`/`Model`), TypeScript (`Datasets` → `Dataset`), Python
(`DATASETS` → `DATASET`) |
| Call sites | `FileResolver`, `FileListerSourceOpExec`,
`DocumentFactory`, `LakeFSFileDocument`, `DatasetResource`,
`DatasetFileNode`, `dataset-selection-modal`, `dataset_file_document` |
| Migration | `36.sql` restored as it shipped; normalization added in a
new `39.sql` |
| Examples | the two shipped `bin/single-node/examples/workflows/*.json`
carried the plural prefix |
Two hardcoded `"dataset"` string literals on the DB side now read from
the enum, since the prefix and the type name are the same word again:
- `SearchQueryBuilder.DATASET_RESOURCE_TYPE`, inlined into the unified
search SQL
- `EntityType.Dataset.value`, written to `user_action.resource_type`
**On the migration.** `36.sql` introduced the prefix in its plural form
and is restored here byte-for-byte, so its checksum stays intact; the
normalization lives in a new `39.sql` (38 is taken). Correcting 36 in
place could not reach the databases that need it: liquibase selects
changeSets by id — `bin/local-dev/main.sh:1824` skips on a match in
`databasechangelog` without reading the file — so the databases holding
plural paths are exactly the ones that recorded id 36 and will never run
it again, and `MD5SUM` is left NULL by design, so nothing reports a
mismatch and the stale paths surface only when someone opens one of
those workflows.
`39.sql` normalizes both shapes, so it converges whether or not 36 ran:
| Stored value | Action |
| --- | --- |
| `/bob@x.com/ds/v1/f.csv` (unprefixed legacy) | prepend `dataset` |
| `/datasets/bob@x.com/ds/v1/f.csv` (plural prefix) | rewrite the
leading segment |
| `/dataset/...` | already current — untouched |
Both cases apply only when the path's owner and name segments match a
real `(user.email, dataset.name)` pair — read at parts 1 and 2
unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off
the plain filesystem paths and URLs this column also holds: `/datasets`
is an ordinary directory name, and `FileResolver` tries
`localResolveFunc` first, so a `fileName` of `/datasets/imdb/movies.csv`
can be a working local mount and is left alone. A path naming a dataset
that no longer exists therefore keeps its old form; it cannot resolve
either way. `/models/` is out of scope, since `37.sql` creates that
table and runs after this changeset.
### Any related issues, documentation, discussions?
Additional work on #6495, which introduced the resource-type prefix.
### How was this PR tested?
Existing suites, updated where they pin the path form.
```bash
sbt "WorkflowCore/testOnly org.apache.texera.amber.storage.FileResolverSpec org.apache.texera.amber.core.storage.DocumentFactorySpec org.apache.texera.amber.core.storage.model.LakeFSFileDocumentSpec" \
"WorkflowOperator/testOnly org.apache.texera.amber.operator.source.dataset.FileListerSourceOpExecSpec" \
"FileService/testOnly org.apache.texera.service.type.DatasetFileNodeSpec" \
"WorkflowExecutionService/testOnly org.apache.texera.web.resource.dashboard.DatasetSearchQueryBuilderSpec org.apache.texera.web.resource.dashboard.DashboardResourceSpec org.apache.texera.web.resource.dashboard.UnifiedResourceSchemaSpec org.apache.texera.web.resource.dashboard.hub.HubResourceSpec org.apache.texera.web.resource.dashboard.user.workflow.WorkflowExecutionsResourceSpec org.apache.texera.web.resource.dashboard.file.DatasetResourceSpec"
```
```bash
pytest amber/src/test/python/pytexera/storage/test_dataset_file_document.py
```
```bash
cd frontend && yarn ng test --watch=false --include='**/datasetVersionFileTree.spec.ts' --include='**/dataset-selection-modal.component.spec.ts' --include='**/dataset-version-selector.component.spec.ts' --include='**/user-dataset-version-filetree.component.spec.ts' --include='**/dataset-detail.component.spec.ts'
```
`sbt Test/compile` and `tsc` on both app and spec sources are clean, so
the enum rename has no missed call sites.
`39.sql` was run against a seeded Postgres over 13 fixtures, one per
case: legacy unprefixed, plural prefix, `/datasets`-rooted local mounts
(3-segment, 4-segment, and as `datasetVersionPath`), plural naming a
deleted dataset, `/models/`, already-singular, a URL, `/datasets/`
appearing mid-path, non-array `operators`, absent property, and a row
checking other operator properties, links and operator order survive.
Correct in every case, on Postgres 18 and 15.11, and idempotent — a
second run reports 0 rows.
Tested through liquibase's id tracking rather than only by piping the
SQL in, emulating `main.sh` against a real `databasechangelog` table:
```
fresh DB: changeSet 36 ran, 39 rewrote -> /dataset/bob@x.com/ds/v1/a.csv
recorded id 36: changeSet 36 skipped, 39 rewrote -> /dataset/bob@x.com/ds/v1/a.csv
```
Also covered: `39` alone with no `36`, `39` twice, and `39` before `36`
out of order — all four converge on the same state, with the local mount
untouched throughout.
The same statements were also run inside a Kubernetes deployment:
`bin/k8s` rendered with `values-development.yaml`, the postgresql
resources applied to minikube, and the SQL piped into the pod against
the real 39-table schema the chart bootstraps. Row-for-row identical to
local, including the local-mount cases staying untouched. Note that the
chart's postgres init runs only `texera_ddl.sql` and the three catalog
scripts, and `texera_db` has no `databasechangelog` table — no liquibase
changeSet executes on a Kubernetes deployment.
`sbt scalafmtCheckAll`, `ruff check`, `ruff format --check`, and
`prettier --check`
are clean.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
---------
Co-authored-by: Claude Opus 5 <no...@anthropic.com>
Co-authored-by: ali <al...@gmail.com>
---
.../pytexera/storage/dataset_file_document.py | 10 +-
.../main/python/pytexera/storage/resource_type.py | 2 +-
.../resource/dashboard/SearchQueryBuilder.scala | 11 +-
.../web/resource/dashboard/hub/EntityType.scala | 3 +-
.../pytexera/storage/test_dataset_file_document.py | 34 ++-
.../workflow/WorkflowExecutionsResourceSpec.scala | 6 +-
...xample] Data Exploration on Movies Dataset.json | 2 +-
...[Example] Machine Learning on Iris Dataset.json | 2 +-
.../amber/core/storage/DocumentFactory.scala | 8 +-
.../texera/amber/core/storage/FileResolver.scala | 16 +-
.../texera/amber/core/storage/ResourceType.scala | 4 +-
.../core/storage/model/LakeFSFileDocument.scala | 4 +-
.../amber/core/storage/DocumentFactorySpec.scala | 8 +-
.../storage/model/LakeFSFileDocumentSpec.scala | 10 +-
.../texera/amber/storage/FileResolverSpec.scala | 24 +-
.../source/dataset/FileListerSourceOpExec.scala | 6 +-
.../dataset/FileListerSourceOpExecSpec.scala | 26 +--
.../texera/service/resource/DatasetResource.scala | 2 +-
.../service/type/dataset/DatasetFileNode.scala | 16 +-
.../service/type/dataset/DatasetFileNodeSpec.scala | 12 +-
.../app/common/type/datasetVersionFileTree.spec.ts | 8 +-
.../src/app/common/type/datasetVersionFileTree.ts | 2 +-
frontend/src/app/common/type/resource-type.ts | 4 +-
.../dataset-detail.component.spec.ts | 4 +-
...user-dataset-version-filetree.component.spec.ts | 6 +-
.../dataset-selection-modal.component.spec.ts | 8 +-
.../dataset-selection-modal.component.ts | 4 +-
.../dataset-version-selector.component.spec.ts | 8 +-
sql/changelog.xml | 5 +
sql/updates/39.sql | 254 +++++++++++++++++++++
30 files changed, 388 insertions(+), 121 deletions(-)
diff --git a/amber/src/main/python/pytexera/storage/dataset_file_document.py b/amber/src/main/python/pytexera/storage/dataset_file_document.py
index b8d778d342..2d14f858e4 100644
--- a/amber/src/main/python/pytexera/storage/dataset_file_document.py
+++ b/amber/src/main/python/pytexera/storage/dataset_file_document.py
@@ -59,18 +59,18 @@ class DatasetFileDocument:
:param file_path:
Expected format -
- "/datasets/ownerEmail/datasetName/versionName/fileRelativePath"
+ "/dataset/ownerEmail/datasetName/versionName/fileRelativePath"
Example:
- "/datasets/bob@texera.com/twitterDataset/v1/california/tw1.csv"
+ "/dataset/bob@texera.com/twitterDataset/v1/california/tw1.csv"
"""
parts = file_path.strip("/").split("/")
invalid_format = ValueError(
"Invalid file path format. Expected: "
- "/datasets/ownerEmail/datasetName/versionName/fileRelativePath"
+ "/dataset/ownerEmail/datasetName/versionName/fileRelativePath"
)
- # TODO(datasets-prefix): require the prefix once all stored paths are migrated (36.sql) and ml model support work is completed.
+ # TODO(dataset-prefix): require the prefix once all stored paths are migrated (36.sql) and ml model support work is completed.
if parts and parts[0] in {t.value for t in ResourceType}:
if len(parts) < 5:
raise invalid_format
@@ -80,7 +80,7 @@ class DatasetFileDocument:
self.version_name = parts[3]
self.file_relative_path = "/".join(parts[4:])
elif len(parts) >= 4:
- self.resource_type = ResourceType.DATASETS
+ self.resource_type = ResourceType.DATASET
self.owner_email = parts[0]
self.dataset_name = parts[1]
self.version_name = parts[2]
diff --git a/amber/src/main/python/pytexera/storage/resource_type.py b/amber/src/main/python/pytexera/storage/resource_type.py
index 596fb5d745..e77ec9dd49 100644
--- a/amber/src/main/python/pytexera/storage/resource_type.py
+++ b/amber/src/main/python/pytexera/storage/resource_type.py
@@ -21,4 +21,4 @@ from enum import Enum
class ResourceType(str, Enum):
"""The leading segment of a logical file path, identifying the resource kind"""
- DATASETS = "datasets"
+ DATASET = "dataset"
diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
index e19755e611..e8a651cb82 100644
--- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
+++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
@@ -19,6 +19,7 @@
package org.apache.texera.web.resource.dashboard
+import org.apache.texera.amber.core.storage.ResourceType
import org.apache.texera.dao.SqlServer
import org.apache.texera.web.resource.dashboard.DashboardResource.{
DashboardClickableFileEntry,
@@ -33,10 +34,18 @@ object SearchQueryBuilder {
SqlServer
.getInstance()
.createDSLContext()
+ // Resource-type discriminators for dashboard search. They are inlined into the search SQL,
+ // pattern-matched in DashboardResource, and duplicated as bare literals in the frontend.
+ // DATASET_RESOURCE_TYPE aliases ResourceType.Dataset, whose value doubles as the leading
+ // segment of a storage logical path, so renaming that prefix silently changes search results
+ // with no compile error here.
+ // TODO: give every resource type (file, workflow, project, dataset, model) a single shared
+ // constant/enum instead of these raw strings, so the search contract has one source of
+ // truth and no longer borrows an unrelated storage constant.
val FILE_RESOURCE_TYPE = "file"
val WORKFLOW_RESOURCE_TYPE = "workflow"
val PROJECT_RESOURCE_TYPE = "project"
- val DATASET_RESOURCE_TYPE = "dataset"
+ val DATASET_RESOURCE_TYPE = ResourceType.Dataset.toString
val ALL_RESOURCE_TYPE = ""
}
diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
index 0cf1bd4cc8..bd83b7837e 100644
--- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
+++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
@@ -20,6 +20,7 @@
package org.apache.texera.web.resource.dashboard.hub
import com.fasterxml.jackson.annotation.{JsonCreator, JsonValue}
+import org.apache.texera.amber.core.storage.ResourceType
/**
* Defines all supported entity types for Hub resources.
@@ -34,7 +35,7 @@ sealed trait EntityType {
object EntityType {
case object Workflow extends EntityType { val value = "workflow" }
- case object Dataset extends EntityType { val value = "dataset" }
+ case object Dataset extends EntityType { val value: String = ResourceType.Dataset.toString }
private val values = Seq(Workflow, Dataset)
diff --git a/amber/src/test/python/pytexera/storage/test_dataset_file_document.py b/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
index bef134bc74..b90dcce81b 100644
--- a/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
+++ b/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
@@ -47,18 +47,18 @@ def make_response(status_code: int, body=None, content: bytes = b""):
class TestDatasetFileDocumentInit:
def test_parses_prefixed_path(self, auth_env):
- doc = DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ doc = DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
assert doc.owner_email == "bob@x.com"
assert doc.dataset_name == "ds"
assert doc.version_name == "v1"
assert doc.file_relative_path == "file.csv"
def test_joins_nested_relative_path_back_with_slashes(self, auth_env):
- doc = DatasetFileDocument("/datasets/bob@x.com/ds/v1/a/b/c/file.csv")
+ doc = DatasetFileDocument("/dataset/bob@x.com/ds/v1/a/b/c/file.csv")
assert doc.file_relative_path == "a/b/c/file.csv"
def test_strips_leading_and_trailing_slashes_before_parsing(self, auth_env):
- doc = DatasetFileDocument("///datasets/bob@x.com/ds/v1/file.csv///")
+ doc = DatasetFileDocument("///dataset/bob@x.com/ds/v1/file.csv///")
assert doc.owner_email == "bob@x.com"
assert doc.file_relative_path == "file.csv"
@@ -80,7 +80,7 @@ class TestDatasetFileDocumentInit:
def test_rejects_prefixed_path_with_too_few_segments(self, auth_env):
with pytest.raises(ValueError, match="Invalid file path format"):
- DatasetFileDocument("/datasets/bob@x.com/ds/v1")
+ DatasetFileDocument("/dataset/bob@x.com/ds/v1")
def test_rejects_legacy_path_with_too_few_segments(self, auth_env):
with pytest.raises(ValueError, match="Invalid file path format"):
@@ -92,29 +92,29 @@ class TestDatasetFileDocumentInit:
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
)
with pytest.raises(ValueError, match="JWT token is required"):
- DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
def test_treats_empty_jwt_as_missing(self, monkeypatch):
# An empty string is falsy and should be rejected just like an unset var.
monkeypatch.setenv("USER_JWT_TOKEN", "")
with pytest.raises(ValueError, match="JWT token is required"):
- DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
def test_falls_back_to_default_endpoint_when_env_missing(self, monkeypatch):
monkeypatch.setenv("USER_JWT_TOKEN", "tok")
monkeypatch.delenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", raising=False
)
- doc = DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ doc = DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
assert doc.presign_endpoint == DEFAULT_ENDPOINT
def test_uses_explicit_endpoint_from_environment(self, auth_env):
- doc = DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ doc = DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
assert doc.presign_endpoint == CUSTOM_ENDPOINT
class TestGetPresignedUrl:
- def _make_doc(self, monkeypatch, path="/datasets/bob@x.com/ds/v1/file.csv"):
+ def _make_doc(self, monkeypatch, path="/dataset/bob@x.com/ds/v1/file.csv"):
monkeypatch.setenv("USER_JWT_TOKEN", "test-jwt-token")
monkeypatch.setenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
@@ -144,9 +144,7 @@ class TestGetPresignedUrl:
def test_url_encodes_filepath_query_parameter(self, monkeypatch):
# urllib.parse.quote keeps "/" as safe by default, but encodes "@"
# and " " — pin both pieces so the contract is explicit.
- doc = self._make_doc(
- monkeypatch, path="/datasets/bob@x.com/ds/v1/data file.csv"
- )
+ doc = self._make_doc(monkeypatch, path="/dataset/bob@x.com/ds/v1/data file.csv")
with patch(
"pytexera.storage.dataset_file_document.requests.Session.get"
) as mock_get:
@@ -158,16 +156,16 @@ class TestGetPresignedUrl:
assert "bob%40x.com" in file_path
assert file_path.startswith("/")
- def test_sends_datasets_prefixed_filepath(self, monkeypatch):
- # The reconstructed filePath sent to the file-service carries the "datasets" prefix.
- doc = self._make_doc(monkeypatch, path="/datasets/bob@x.com/ds/v1/file.csv")
+ def test_sends_dataset_prefixed_filepath(self, monkeypatch):
+ # The reconstructed filePath sent to the file-service carries the "dataset" prefix.
+ doc = self._make_doc(monkeypatch, path="/dataset/bob@x.com/ds/v1/file.csv")
with patch(
"pytexera.storage.dataset_file_document.requests.Session.get"
) as mock_get:
mock_get.return_value = make_response(200, body={"presignedUrl": "u"})
doc.get_presigned_url()
_, kwargs = mock_get.call_args
- assert kwargs["params"]["filePath"].startswith("/datasets/")
+ assert kwargs["params"]["filePath"].startswith("/dataset/")
def test_calls_configured_endpoint(self, monkeypatch):
doc = self._make_doc(monkeypatch)
@@ -235,7 +233,7 @@ class TestReadFile:
monkeypatch.setenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
)
- return DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ return DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
def test_returns_bytesio_with_downloaded_content(self, monkeypatch):
doc = self._make_doc(monkeypatch)
@@ -291,7 +289,7 @@ class TestTimeoutsAndRetries:
monkeypatch.setenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
)
- return DatasetFileDocument("/datasets/bob@x.com/ds/v1/file.csv")
+ return DatasetFileDocument("/dataset/bob@x.com/ds/v1/file.csv")
def test_presigned_url_request_passes_request_timeout(self, monkeypatch):
doc = self._make_doc(monkeypatch)
diff --git a/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala b/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
index 7dc5742a18..7cc549004d 100644
--- a/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
+++ b/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
@@ -910,7 +910,7 @@ class WorkflowExecutionsResourceSpec
val content =
"""{
| "operators": [
- | {"operatorID": "scanA", "operatorProperties": {"fileName": "/datasets/owner@example.com/LockedDS/v1/data.csv"}},
+ | {"operatorID": "scanA", "operatorProperties": {"fileName": "/dataset/owner@example.com/LockedDS/v1/data.csv"}},
| {"operatorID": "downstreamB", "operatorProperties": {}}
| ],
| "links": [
@@ -969,7 +969,7 @@ class WorkflowExecutionsResourceSpec
val content =
"""{
| "operators": [
- | {"operatorID": "scan", "operatorProperties": {"fileName": "/datasets/test@example.com/MyDS/v1/data.csv"}}
+ | {"operatorID": "scan", "operatorProperties": {"fileName": "/dataset/test@example.com/MyDS/v1/data.csv"}}
| ],
| "links": []
|}""".stripMargin
@@ -1408,7 +1408,7 @@ class WorkflowExecutionsResourceSpec
private def scanOperator(operatorId: String, datasetName: String): String =
s"""{"operatorID": "$operatorId", "operatorProperties": """ +
- s"""{"fileName": "/datasets/$foreignOwnerEmail/$datasetName/v1/data.csv"}}"""
+ s"""{"fileName": "/dataset/$foreignOwnerEmail/$datasetName/v1/data.csv"}}"""
// Seeds three datasets owned by somebody other than testUser and wires the workflow as
//
diff --git a/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json b/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json
index cdc9a0e870..5df44f5382 100644
--- a/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json
+++ b/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json
@@ -8,7 +8,7 @@
"fileEncoding": "UTF_8",
"customDelimiter": ",",
"hasHeader": true,
- "fileName": "/datasets/texera/popular-movies-of-imdb/v1/TMDb_updated.csv",
+ "fileName": "/dataset/texera/popular-movies-of-imdb/v1/TMDb_updated.csv",
"offset": 0,
"limit": 1000
},
diff --git a/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json b/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json
index 7e9a5d11fd..e4cdc3a8af 100644
--- a/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json
+++ b/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json
@@ -8,7 +8,7 @@
"fileEncoding": "UTF_8",
"customDelimiter": ",",
"hasHeader": true,
- "fileName": "/datasets/texera/iris-species/v1/Iris.csv"
+ "fileName": "/dataset/texera/iris-species/v1/Iris.csv"
},
"inputPorts": [],
"outputPorts": [
diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
index 8481906c9c..4915d57897 100644
--- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
+++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
@@ -60,8 +60,8 @@ object DocumentFactory {
*/
def openReadonlyDocument(fileUri: URI): ReadonlyVirtualDocument[_] = {
fileUri.getScheme match {
- case DATASET_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Datasets)
- case MODEL_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Models)
+ case DATASET_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Dataset)
+ case MODEL_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Model)
case "file" => new ReadonlyLocalFileDocument(fileUri)
case unsupportedScheme =>
throw new UnsupportedOperationException(
@@ -184,8 +184,8 @@ object DocumentFactory {
*/
def openDocument(uri: URI): (VirtualDocument[_], Option[Schema]) = {
uri.getScheme match {
- case DATASET_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Datasets), None)
- case MODEL_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Models), None)
+ case DATASET_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Dataset), None)
+ case MODEL_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Model), None)
case VFS_FILE_URI_SCHEME =>
val IcebergLocation(warehouse, namespace, storageKey) = resolveIcebergLocation(uri)
diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
index 1a2e4fcf2a..95127e234d 100644
--- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
+++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
@@ -93,7 +93,7 @@ object FileResolver {
* caller can dispatch to the right backing table.
* - Legacy unprefixed (datasets only, backward compat): /ownerEmail/datasetName/versionName/
* fileRelativePath (>= 4 segments), resolved as a dataset. Models are new and always require
- * the /models/ prefix.
+ * the /model/ prefix.
*
* @param fileName the file path to parse
* @return Some((resourceType, ownerEmail, resourceName, versionName, fileRelativePath)) if valid,
@@ -115,11 +115,11 @@ object FileResolver {
)
case None =>
// Legacy unprefixed dataset path (backward compat): /ownerEmail/datasetName/versionName/<file>.
- // TODO(datasets-prefix): require the prefix once all stored paths are migrated (36.sql).
+ // TODO(dataset-prefix): require the prefix once all stored paths are migrated (36.sql).
if (pathSegments.length >= 4)
Some(
(
- ResourceType.Datasets,
+ ResourceType.Dataset,
pathSegments(0),
pathSegments(1),
pathSegments(2),
@@ -139,8 +139,8 @@ object FileResolver {
*
* Input: /<prefix>/ownerEmail/resourceName/versionName/fileRelativePath (or legacy unprefixed)
* Output: {scheme}:///{repositoryName}/{versionHash}/fileRelativePath
- * e.g. /datasets/bob@x.com/twitter/v1/dir/f.csv -> dataset:///dataset-15/adeq233td/dir/f.csv
- * /models/bob@x.com/resnet/v1/weights/m.pt -> model:///model-15/adeq233td/weights/m.pt
+ * e.g. /dataset/bob@x.com/twitter/v1/dir/f.csv -> dataset:///dataset-15/adeq233td/dir/f.csv
+ * /model/bob@x.com/resnet/v1/weights/m.pt -> model:///model-15/adeq233td/weights/m.pt
*
* @throws java.io.FileNotFoundException if the path is not a valid versioned-resource path, the
* resource/version does not exist, or the URI is malformed
@@ -152,10 +152,10 @@ object FileResolver {
)
val (scheme, repositoryName, versionHash) = resourceType match {
- case ResourceType.Datasets =>
+ case ResourceType.Dataset =>
val (repo, hash) = lookupDataset(ownerEmail, resourceName, versionName, fileName)
(DATASET_FILE_URI_SCHEME, repo, hash)
- case ResourceType.Models =>
+ case ResourceType.Model =>
val (repo, hash) = lookupModel(ownerEmail, resourceName, versionName, fileName)
(MODEL_FILE_URI_SCHEME, repo, hash)
case other =>
@@ -306,7 +306,7 @@ object FileResolver {
return None
}
parsePrefixedPath(path).collect {
- case (ResourceType.Datasets, ownerEmail, datasetName, _, _) => (ownerEmail, datasetName)
+ case (ResourceType.Dataset, ownerEmail, datasetName, _, _) => (ownerEmail, datasetName)
}
}
}
diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
index 087dc6ef87..47e90a1818 100644
--- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
+++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
@@ -26,8 +26,8 @@ package org.apache.texera.amber.core.storage
* Path shape: /<prefix>/ownerEmail/resourceName/versionName/fileRelativePath
*/
object ResourceType extends Enumeration {
- val Datasets: Value = Value("datasets")
- val Models: Value = Value("models")
+ val Dataset: Value = Value("dataset")
+ val Model: Value = Value("model")
/**
* Returns the resource type named by the given path segment, or None if it is not a known
diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
index b821ff72f5..55fab33ce2 100644
--- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
+++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
@@ -60,8 +60,8 @@ object LakeFSFileDocument {
*/
def presignEndpointOf(resourceType: ResourceType.Value): String =
resourceType match {
- case ResourceType.Datasets => datasetPresignEndpoint
- case ResourceType.Models => modelPresignEndpoint
+ case ResourceType.Dataset => datasetPresignEndpoint
+ case ResourceType.Model => modelPresignEndpoint
}
}
diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
index a957819b31..f219383f25 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
@@ -137,7 +137,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA
val datasetUri = new URI(s"dataset:///repo/$versionHash/file.txt")
val doc = DocumentFactory.openReadonlyDocument(datasetUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Datasets
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Dataset
doc.getURI shouldBe datasetUri
}
@@ -145,7 +145,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA
val modelUri = new URI(s"model:///model-1/$versionHash/weights/model.pt")
val doc = DocumentFactory.openReadonlyDocument(modelUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Models
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Model
doc.getURI shouldBe modelUri
val resource = doc.asInstanceOf[OnVersionedFileResource]
@@ -181,7 +181,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA
val datasetUri = new URI(s"dataset:///repo/$versionHash/file.txt")
val (doc, schemaOpt) = DocumentFactory.openDocument(datasetUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Datasets
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Dataset
schemaOpt shouldBe None
}
@@ -189,7 +189,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA
val modelUri = new URI(s"model:///model-1/$versionHash/weights/model.pt")
val (doc, schemaOpt) = DocumentFactory.openDocument(modelUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Models
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Model
schemaOpt shouldBe None
}
diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
index 56d5cd36ce..fa569f97ca 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
@@ -35,7 +35,7 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
private val versionHash = "97fd4c2a755b69b7c66d322eab40b7e5c2ad5d10"
// URI parsing is shared by every resource type; exercise it through the dataset type.
- private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri, ResourceType.Datasets)
+ private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri, ResourceType.Dataset)
"LakeFSFileDocument" should "parse a valid 3-segment dataset URI into its components" in {
val uri = new URI(s"dataset:///test_dataset/$versionHash/1.txt")
@@ -160,7 +160,7 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
"http://localhost:9092/api/dataset/presign-download"
)
.trim
- LakeFSFileDocument.presignEndpointOf(ResourceType.Datasets) shouldBe expected
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Dataset) shouldBe expected
}
// Each resource type resolves its own endpoint: a dataset grant must not authorize a model
@@ -173,9 +173,9 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
"http://localhost:9092/api/model/presign-download"
)
.trim
- LakeFSFileDocument.presignEndpointOf(ResourceType.Models) shouldBe expected
- LakeFSFileDocument.presignEndpointOf(ResourceType.Models) should not be
- LakeFSFileDocument.presignEndpointOf(ResourceType.Datasets)
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Model) shouldBe expected
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Model) should not be
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Dataset)
}
// The user JWT token is shared by every LakeFS-backed document, whatever its resource type.
diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
index e414371222..2649a3d860 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
@@ -122,13 +122,13 @@ class FileResolverSpec
private val localCsvFilePath = "common/workflow-core/src/test/resources/country_sales_small.csv"
- private val datasetACsvFilePath = "/datasets/test_user@test.com/test_dataset/v2/directory/a.csv"
+ private val datasetACsvFilePath = "/dataset/test_user@test.com/test_dataset/v2/directory/a.csv"
- private val dataset1TxtFilePath = "/datasets/test_user@test.com/test_dataset/v1/1.txt"
+ private val dataset1TxtFilePath = "/dataset/test_user@test.com/test_dataset/v1/1.txt"
- private val modelWeightsFilePath = "/models/test_user@test.com/test_model/v2/weights/model.pt"
+ private val modelWeightsFilePath = "/model/test_user@test.com/test_model/v2/weights/model.pt"
- private val modelReadmeFilePath = "/models/test_user@test.com/test_model/v1/README.md"
+ private val modelReadmeFilePath = "/model/test_user@test.com/test_model/v1/README.md"
// Legacy unprefixed form — still resolvable as a dataset (backward compat).
private val unprefixedDataset1TxtFilePath = "/test_user@test.com/test_dataset/v1/1.txt"
@@ -137,11 +137,11 @@ class FileResolverSpec
private val unknownResourceTypeFilePath =
"/notAResourceType/test_user@test.com/test_dataset/v1/1.txt"
- // A model name presented under the datasets prefix must not resolve as a dataset.
- private val modelNameUnderDatasetPrefix = "/datasets/test_user@test.com/test_model/v1/README.md"
+ // A model name presented under the dataset prefix must not resolve as a dataset.
+ private val modelNameUnderDatasetPrefix = "/dataset/test_user@test.com/test_model/v1/README.md"
- // A dataset name presented under the models prefix must not resolve as a model.
- private val datasetNameUnderModelPrefix = "/models/test_user@test.com/test_dataset/v1/1.txt"
+ // A dataset name presented under the model prefix must not resolve as a model.
+ private val datasetNameUnderModelPrefix = "/model/test_user@test.com/test_dataset/v1/1.txt"
override protected def beforeAll(): Unit = {
initializeDBAndReplaceDSLContext()
@@ -225,7 +225,7 @@ class FileResolverSpec
"FileResolver" should "throw not found exception when a prefixed path has too few segments" in {
assertThrows[FileNotFoundException] {
- FileResolver.resolve("/datasets/test_user@test.com/test_dataset")
+ FileResolver.resolve("/dataset/test_user@test.com/test_dataset")
}
}
@@ -254,12 +254,12 @@ class FileResolverSpec
"parseDatasetOwnerAndName" should "extract owner email and dataset name from a valid path" in {
assert(
- FileResolver.parseDatasetOwnerAndName("/datasets/test_user@test.com/test_dataset/v1/1.txt")
+ FileResolver.parseDatasetOwnerAndName("/dataset/test_user@test.com/test_dataset/v1/1.txt")
== Some(("test_user@test.com", "test_dataset"))
)
// extra segments beyond the file-relative path are ignored
assert(
- FileResolver.parseDatasetOwnerAndName("/datasets/owner@x.com/ds/v2/directory/nested/a.csv")
+ FileResolver.parseDatasetOwnerAndName("/dataset/owner@x.com/ds/v2/directory/nested/a.csv")
== Some(("owner@x.com", "ds"))
)
}
@@ -272,7 +272,7 @@ class FileResolverSpec
}
it should "return None when the prefixed path has too few segments" in {
- assert(FileResolver.parseDatasetOwnerAndName("/datasets/owner@x.com/ds").isEmpty)
+ assert(FileResolver.parseDatasetOwnerAndName("/dataset/owner@x.com/ds").isEmpty)
assert(FileResolver.parseDatasetOwnerAndName("owner/dataset").isEmpty)
}
diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
index fb27257865..26a6496a24 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
@@ -32,7 +32,7 @@ import org.apache.texera.dao.jooq.generated.tables.User.USER
object FileListerSourceOpExec {
/**
- * Parses a dataset version path (/datasets/ownerEmail/datasetName/versionName) into its
+ * Parses a dataset version path (/dataset/ownerEmail/datasetName/versionName) into its
* (resourceTypePrefix, ownerEmail, datasetName, versionName) components.
*
* @throws IllegalArgumentException if the path is not a well-formed dataset version path
@@ -42,14 +42,14 @@ object FileListerSourceOpExec {
): (String, String, String, String) = {
val segments = datasetVersionPath.split("/").filter(_.nonEmpty)
val invalidPath = s"Invalid dataset version path '$datasetVersionPath'; " +
- "expected /datasets/ownerEmail/datasetName/versionName"
+ "expected /dataset/ownerEmail/datasetName/versionName"
if (segments.headOption.exists(ResourceType.isValidPrefix)) {
require(segments.length >= 4, invalidPath)
(segments(0), segments(1), segments(2), segments(3))
} else {
require(segments.length >= 3, invalidPath)
- (ResourceType.Datasets.toString, segments(0), segments(1), segments(2))
+ (ResourceType.Dataset.toString, segments(0), segments(1), segments(2))
}
}
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
index 19fb7513d5..ec0ffd9a29 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
@@ -23,10 +23,10 @@ import org.scalatest.flatspec.AnyFlatSpec
class FileListerSourceOpExecSpec extends AnyFlatSpec {
- "parseDatasetVersionPath" should "extract components from a datasets-prefixed path" in {
+ "parseDatasetVersionPath" should "extract components from a dataset-prefixed path" in {
val (prefix, owner, name, version) =
- FileListerSourceOpExec.parseDatasetVersionPath("/datasets/bob@texera.com/twitterDataset/v1")
- assert(prefix == "datasets")
+ FileListerSourceOpExec.parseDatasetVersionPath("/dataset/bob@texera.com/twitterDataset/v1")
+ assert(prefix == "dataset")
assert(owner == "bob@texera.com")
assert(name == "twitterDataset")
assert(version == "v1")
@@ -34,7 +34,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
it should "work when the owner segment is a username without an '@'" in {
val (prefix, owner, name, version) =
- FileListerSourceOpExec.parseDatasetVersionPath("/datasets/texera/test-ds/v1")
+ FileListerSourceOpExec.parseDatasetVersionPath("/dataset/texera/test-ds/v1")
assert(owner == "texera")
assert(name == "test-ds")
assert(version == "v1")
@@ -42,8 +42,8 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
it should "ignore trailing slashes and extra segments" in {
val (prefix, owner, name, version) =
- FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds/v2/extra/")
- assert(prefix == "datasets")
+ FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds/v2/extra/")
+ assert(prefix == "dataset")
assert(owner == "alice")
assert(name == "ds")
assert(version == "v2")
@@ -52,11 +52,11 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
"canonicalVersionPath" should "rebuild the prefixed version path from its components" in {
assert(
FileListerSourceOpExec.canonicalVersionPath(
- "datasets",
+ "dataset",
"bob@texera.com",
"twitterDataset",
"v1"
- ) == "/datasets/bob@texera.com/twitterDataset/v1"
+ ) == "/dataset/bob@texera.com/twitterDataset/v1"
)
}
@@ -64,17 +64,17 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
// Emitted paths must be rooted at the parsed components, not the raw configured path:
// a stray "extra" segment would otherwise leak into every emitted file path.
val (prefix, owner, name, version) =
- FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds/v2/extra/")
+ FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds/v2/extra/")
assert(
FileListerSourceOpExec.canonicalVersionPath(prefix, owner, name, version)
- == "/datasets/alice/ds/v2"
+ == "/dataset/alice/ds/v2"
)
}
it should "still accept a legacy unprefixed path" in {
val (prefix, owner, name, version) =
FileListerSourceOpExec.parseDatasetVersionPath("/alice/ds/v1")
- assert(prefix == "datasets")
+ assert(prefix == "dataset")
assert(owner == "alice")
assert(name == "ds")
assert(version == "v1")
@@ -85,7 +85,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
FileListerSourceOpExec.parseDatasetVersionPath("/alice/ds/v1")
assert(
FileListerSourceOpExec.canonicalVersionPath(prefix, owner, name, version)
- == "/datasets/alice/ds/v1"
+ == "/dataset/alice/ds/v1"
)
}
@@ -99,7 +99,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
it should "reject a prefixed path with too few segments" in {
assertThrows[IllegalArgumentException] {
- FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds")
+ FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds")
}
}
diff --git a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
index 8bb1a70661..4c8243b850 100644
--- a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
+++ b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
@@ -353,7 +353,7 @@ class DatasetResource extends LazyLogging {
private val COVER_IMAGE_SIZE_LIMIT_BYTES: Long = 10 * 1024 * 1024 // 10 MB
private val ALLOWED_IMAGE_EXTENSIONS: Set[String] = Set(".jpg", ".jpeg", ".png", ".gif", ".webp")
- private val resourceType = ResourceType.Datasets
+ private val resourceType = ResourceType.Dataset
/**
* Helper function to get the dataset from DB with additional information including user access privilege and owner email
diff --git a/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala b/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
index 821d02045c..85709b8e5b 100644
--- a/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
+++ b/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
@@ -25,8 +25,8 @@ import org.apache.texera.amber.core.storage.ResourceType
import scala.collection.mutable
// DatasetFileNode represents a unique file in dataset, its full path is in the format of:
-// /datasets/ownerEmail/datasetName/versionName/fileRelativePath
-// e.g. /datasets/bob@texera.com/twitterDataset/v1/california/irvine/tw1.csv
+// /dataset/ownerEmail/datasetName/versionName/fileRelativePath
+// e.g. /dataset/bob@texera.com/twitterDataset/v1/california/irvine/tw1.csv
class DatasetFileNode(
val name: String, // direct name of this node
val nodeType: String, // "file" or "directory"
@@ -79,10 +79,10 @@ object DatasetFileNode {
): List[DatasetFileNode] = {
val rootNode = new DatasetFileNode("/", "directory", null, "")
- // Root the tree at the datasets prefix node (a directory node named "datasets").
- val datasetsNode =
- new DatasetFileNode(ResourceType.Datasets.toString, "directory", rootNode, "")
- rootNode.children = Some(List(datasetsNode))
+ // Root the tree at the dataset prefix node (a directory node named "dataset").
+ val prefixNode =
+ new DatasetFileNode(ResourceType.Dataset.toString, "directory", rootNode, "")
+ rootNode.children = Some(List(prefixNode))
// Owner level nodes map
val ownerNodes = mutable.Map[String, DatasetFileNode]()
@@ -91,8 +91,8 @@ object DatasetFileNode {
case ((ownerEmail, datasetName, versionName), objects) =>
val ownerNode = ownerNodes.getOrElseUpdate(
ownerEmail, {
- val newNode = new DatasetFileNode(ownerEmail, "directory", datasetsNode, ownerEmail)
- datasetsNode.children = Some(datasetsNode.getChildren :+ newNode)
+ val newNode = new DatasetFileNode(ownerEmail, "directory", prefixNode, ownerEmail)
+ prefixNode.children = Some(prefixNode.getChildren :+ newNode)
newNode
}
)
diff --git a/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala b/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
index 160412f8b9..12321d4180 100644
--- a/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
+++ b/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
@@ -99,13 +99,13 @@ class DatasetFileNodeSpec extends AnyFlatSpec with Matchers {
Map(("bob@texera.com", "twitter", "v1") -> objects)
)
- // The tree is rooted at a single "datasets" prefix node; owners nest under it.
+ // The tree is rooted at a single "dataset" prefix node; owners nest under it.
roots should have size 1
- val datasetsNode = roots.head
- datasetsNode.getName shouldBe "datasets"
- datasetsNode.getNodeType shouldBe "directory"
+ val prefixNode = roots.head
+ prefixNode.getName shouldBe "dataset"
+ prefixNode.getNodeType shouldBe "directory"
- val ownerNode = datasetsNode.getChildren.find(_.getName == "bob@texera.com").get
+ val ownerNode = prefixNode.getChildren.find(_.getName == "bob@texera.com").get
ownerNode.getNodeType shouldBe "directory"
val datasetNode = ownerNode.getChildren.find(_.getName == "twitter").get
@@ -123,7 +123,7 @@ class DatasetFileNodeSpec extends AnyFlatSpec with Matchers {
val file1 = bDir.getChildren.find(_.getName == "1.csv").get
file1.getNodeType shouldBe "file"
file1.getSize shouldBe Some(2L)
- file1.getFilePath shouldBe "/datasets/bob@texera.com/twitter/v1/b/1.csv"
+ file1.getFilePath shouldBe "/dataset/bob@texera.com/twitter/v1/b/1.csv"
// Total size equals the sum of the three files.
DatasetFileNode.calculateTotalSize(roots) shouldBe 6L
diff --git a/frontend/src/app/common/type/datasetVersionFileTree.spec.ts b/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
index 64231dd156..207182e5fd 100644
--- a/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
+++ b/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
@@ -41,7 +41,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "tw1.csv",
type: "file",
- parentDir: "/datasets/bob@texera.com/twitterDataset/v1/california/irvine",
+ parentDir: "/dataset/bob@texera.com/twitterDataset/v1/california/irvine",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("california/irvine/tw1.csv");
});
@@ -50,7 +50,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "readme.txt",
type: "file",
- parentDir: "/datasets/bob@texera.com/twitterDataset/v1",
+ parentDir: "/dataset/bob@texera.com/twitterDataset/v1",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("readme.txt");
});
@@ -59,7 +59,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "v1",
type: "directory",
- parentDir: "/datasets/bob@texera.com/twitterDataset",
+ parentDir: "/dataset/bob@texera.com/twitterDataset",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("");
});
@@ -68,7 +68,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "f.csv",
type: "file",
- parentDir: "/datasets/bob@texera.com/twitterDataset//v1/sub",
+ parentDir: "/dataset/bob@texera.com/twitterDataset//v1/sub",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("sub/f.csv");
});
diff --git a/frontend/src/app/common/type/datasetVersionFileTree.ts b/frontend/src/app/common/type/datasetVersionFileTree.ts
index 98f898d443..efbb875128 100644
--- a/frontend/src/app/common/type/datasetVersionFileTree.ts
+++ b/frontend/src/app/common/type/datasetVersionFileTree.ts
@@ -32,7 +32,7 @@ export function getFullPathFromDatasetFileNode(node: DatasetFileNode): string {
/**
* Returns the relative path of a DatasetFileNode by stripping the first four segments
- * (datasets/ownerEmail/datasetName/versionName).
+ * (dataset/ownerEmail/datasetName/versionName).
* @param node The DatasetFileNode whose relative path is needed.
* @returns The relative path (without the first four segments and without a leading slash).
*/
diff --git a/frontend/src/app/common/type/resource-type.ts b/frontend/src/app/common/type/resource-type.ts
index a129c917e2..22c4a8592a 100644
--- a/frontend/src/app/common/type/resource-type.ts
+++ b/frontend/src/app/common/type/resource-type.ts
@@ -17,8 +17,8 @@
*/
/**
- * The leading segment of a logical file path, identifying the resource kind (e.g. /datasets/...).
+ * The leading segment of a logical file path, identifying the resource kind (e.g. /dataset/...).
*/
export enum ResourceType {
- Datasets = "datasets",
+ Dataset = "dataset",
}
diff --git a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
index 78c35761a4..7bb6af6e0d 100644
--- a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
+++ b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
@@ -2694,7 +2694,7 @@ describe("DatasetDetailComponent rendered template", () => {
const leaf = (name: string): DatasetFileNode => ({
name,
type: "file",
- parentDir: `/datasets/${OWNER}/ds/v1/nested`,
+ parentDir: `/dataset/${OWNER}/ds/v1/nested`,
size: 2048,
});
@@ -2719,7 +2719,7 @@ describe("DatasetDetailComponent rendered template", () => {
// The heading is the full path — the copy-path button beside it copies
// exactly this string — not the bare file name or the relative path.
expect(text(q<HTMLElement>(fixture.nativeElement, ".file-title-main"))).toBe(
- `/datasets/${OWNER}/ds/v1/nested/b.csv`
+ `/dataset/${OWNER}/ds/v1/nested/b.csv`
);
// 2048 bytes reaches the reader as a human-readable size, not as a raw count.
expect(text(q<HTMLElement>(fixture.nativeElement, ".file-size"))).toBe("2.00 KB");
diff --git a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
index 806d2a6f25..ff9e82609a 100644
--- a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
+++ b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
@@ -183,7 +183,7 @@ describe("UserDatasetVersionFiletreeComponent", () => {
const file = (name: string): DatasetFileNode => ({
name,
type: "file",
- parentDir: "/datasets/owner/dataset/v1",
+ parentDir: "/dataset/owner/dataset/v1",
});
// Deliberately synchronous: the tree keeps a pending timer, so awaiting whenStable()
@@ -262,9 +262,9 @@ describe("UserDatasetVersionFiletreeComponent", () => {
const emitted: string[] = [];
component.setCoverImage.subscribe((path: string) => emitted.push(path));
- // parentDir has exactly the four stripped segments (datasets/owner/dataset/version),
+ // parentDir has exactly the four stripped segments (dataset/owner/dataset/version),
// so the relative path is just the file name.
- component.onSetCover({ name: "photo.png", type: "file", parentDir: "/datasets/owner/dataset/v1" });
+ component.onSetCover({ name: "photo.png", type: "file", parentDir: "/dataset/owner/dataset/v1" });
expect(emitted).toEqual(["photo.png"]);
});
diff --git a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
index 604d86a9f9..f42ae00a81 100644
--- a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
+++ b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
@@ -111,7 +111,7 @@ describe("DatasetSelectionModalComponent", () => {
it("ngOnInit initializes selectedDataset and selectedVersion from data.selectedPath", () => {
modalData.fileMode = true;
- modalData.selectedPath = `/datasets/${OWNER}/myds/v1`;
+ modalData.selectedPath = `/dataset/${OWNER}/myds/v1`;
build();
@@ -154,7 +154,7 @@ describe("DatasetSelectionModalComponent", () => {
expect(datasetService.retrieveDatasetVersionFileTree).toHaveBeenCalledWith(10, 100);
expect(component.fileTree).toEqual([fileNode]);
- expect(component.selectedPath).toBe(`/datasets/${OWNER}/myds/v1`);
+ expect(component.selectedPath).toBe(`/dataset/${OWNER}/myds/v1`);
});
it("onFileSelected sets selectedPath to the node's full path in file mode", () => {
@@ -303,12 +303,12 @@ describe("DatasetSelectionModalComponent", () => {
confirmButton().click();
- expect(modalRef.close).toHaveBeenCalledWith(`/datasets/${OWNER}/myds/v1`);
+ expect(modalRef.close).toHaveBeenCalledWith(`/dataset/${OWNER}/myds/v1`);
});
it("takes the path from the file tree's selection in file mode", () => {
modalData.fileMode = true;
- modalData.selectedPath = `/datasets/${OWNER}/myds/v1`;
+ modalData.selectedPath = `/dataset/${OWNER}/myds/v1`;
build();
fixture.debugElement
diff --git a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
index 97ecf579ca..b5768aa685 100644
--- a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
+++ b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
@@ -85,7 +85,7 @@ export class DatasetSelectionModalComponent implements OnInit {
const selectedPath = this.data.selectedPath;
if (selectedPath) {
const segments = selectedPath.split("/").filter(part => part.length > 0);
- // TODO(datasets-prefix): require the prefix once all ml model support PRs are done.
+ // TODO(dataset-prefix): require the prefix once all ml model support PRs are done.
if ((Object.values(ResourceType) as string[]).includes(segments[0])) {
segments.shift();
}
@@ -124,7 +124,7 @@ export class DatasetSelectionModalComponent implements OnInit {
this.fileTree = data.fileNodes;
});
if (!this.data.fileMode) {
- this.selectedPath = `/${ResourceType.Datasets}/${this.selectedDataset.ownerEmail}/${this.selectedDataset.dataset.name}/${this.selectedVersion.name}`;
+ this.selectedPath = `/${ResourceType.Dataset}/${this.selectedDataset.ownerEmail}/${this.selectedDataset.dataset.name}/${this.selectedVersion.name}`;
}
}
}
diff --git a/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts b/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
index 394e9773cc..f5ba4045e8 100644
--- a/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
+++ b/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
@@ -101,11 +101,11 @@ describe("DatasetVersionSelectorComponent", () => {
});
it("shows the chosen path in a read-only box", () => {
- setFormControl("/datasets/owner@x.com/myds/v1");
+ setFormControl("/dataset/owner@x.com/myds/v1");
fixture.detectChanges();
const box = selectionBox()!;
- expect(box.value).toBe("/datasets/owner@x.com/myds/v1");
+ expect(box.value).toBe("/dataset/owner@x.com/myds/v1");
// The path is only editable through the picker, never by typing.
expect(box.readOnly).toBe(true);
expect(box.required).toBe(true);
@@ -114,13 +114,13 @@ describe("DatasetVersionSelectorComponent", () => {
it("opens the picker from the button and shows whatever it returns", () => {
setFormControl("");
fixture.detectChanges();
- modalServiceSpy.create.mockReturnValue({ afterClose: of("/datasets/owner@x.com/myds/v2") });
+ modalServiceSpy.create.mockReturnValue({ afterClose: of("/dataset/owner@x.com/myds/v2") });
openButton().click();
fixture.detectChanges();
expect(modalServiceSpy.create).toHaveBeenCalledTimes(1);
- expect(selectionBox()!.value).toBe("/datasets/owner@x.com/myds/v2");
+ expect(selectionBox()!.value).toBe("/dataset/owner@x.com/myds/v2");
});
});
});
diff --git a/sql/changelog.xml b/sql/changelog.xml
index c0f735a09c..b9c96431a5 100644
--- a/sql/changelog.xml
+++ b/sql/changelog.xml
@@ -104,6 +104,11 @@
<sqlFile path="sql/updates/38.sql"/>
</changeSet>
+ <!-- Singularize the resource-type prefix on stored dataset paths -->
+ <changeSet id="39" author="tanishqgandhi1908">
+ <sqlFile path="sql/updates/39.sql"/>
+ </changeSet>
+
<!-- example changeSet
<changeSet id="1" author="author">
<sqlFile path="sql/updates/1.sql"/>
diff --git a/sql/updates/39.sql b/sql/updates/39.sql
new file mode 100644
index 0000000000..4c48244bdd
--- /dev/null
+++ b/sql/updates/39.sql
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+\c texera_db
+
+SET search_path TO texera_db;
+
+BEGIN;
+
+-- The resource-type prefix on dataset logical paths is singular, matching the table
+-- it names: /dataset/ownerEmail/datasetName/versionName/... This supersedes 36.sql,
+-- which introduced the prefix in its plural form; that changeSet is left as it
+-- shipped, so databases that already recorded it are brought forward here instead.
+--
+-- Stored workflows carry such paths inside workflow.content and
+-- workflow_version.content, in two operator properties:
+-- * fileName (scan-source operators): /owner/name/version/file
+-- * datasetVersionPath (file-lister operator): /owner/name/version
+--
+-- Both are normalized to that form: a "/datasets/" prefix has its leading segment
+-- rewritten, and a path with no prefix gets "dataset" prepended. The prefixed case
+-- is tested first, since such a path looks unprefixed to the other branch and must
+-- not be prefixed twice.
+--
+-- Either case applies only when the path's owner and name segments match an
+-- existing (user.email, dataset.name) pair -- unique, and read at parts 1 and 2
+-- unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off the
+-- plain filesystem paths and URLs this column also holds: "/datasets" is an
+-- ordinary directory name, and FileResolver tries localResolveFunc first, so a
+-- fileName of /datasets/imdb/movies.csv can be a working local mount. It matches
+-- no dataset, so it is left alone.
+--
+-- Values already in the target form match neither case, making this idempotent.
+-- jsonb_set uses create_missing = false so absent properties are never added.
+
+DO $$
+DECLARE
+ wf_count INT := 0;
+ wv_count INT := 0;
+BEGIN
+ WITH affected AS (
+ SELECT w.wid
+ FROM workflow w,
+ jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(w.content::jsonb -> 'operators') = 'array'
+ THEN w.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) AS op,
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}' AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}' AS dvp
+ ) f
+ WHERE jsonb_typeof(w.content::jsonb -> 'operators') = 'array'
+ AND (
+ (f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 3)))
+ OR
+ (f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3)))
+ OR
+ (f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 2)))
+ OR
+ (f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2)))
+ )
+ GROUP BY w.wid
+ ),
+ updated AS (
+ UPDATE workflow w
+ SET content = jsonb_set(w.content::jsonb, '{operators}', (
+ SELECT jsonb_agg(
+ jsonb_set(
+ jsonb_set(
+ op,
+ '{operatorProperties,fileName}',
+ CASE
+ WHEN f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.fn, '^/datasets/', '/dataset/'))
+ WHEN f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.fn, '/'))
+ ELSE COALESCE(op #> '{operatorProperties,fileName}', 'null'::jsonb)
+ END,
+ false
+ ),
+ '{operatorProperties,datasetVersionPath}',
+ CASE
+ WHEN f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.dvp, '^/datasets/', '/dataset/'))
+ WHEN f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.dvp, '/'))
+ ELSE COALESCE(op #> '{operatorProperties,datasetVersionPath}', 'null'::jsonb)
+ END,
+ false
+ )
+ ORDER BY ord
+ )
+ FROM jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(w.content::jsonb -> 'operators') = 'array'
+ THEN w.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) WITH ORDINALITY AS t(op, ord),
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}' AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}' AS dvp
+ ) f
+ ))::text
+ FROM affected a
+ WHERE w.wid = a.wid
+ RETURNING 1
+ )
+ SELECT count(*) INTO wf_count FROM updated;
+
+ WITH affected AS (
+ SELECT wv.vid
+ FROM workflow_version wv,
+ jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(wv.content::jsonb -> 'operators') = 'array'
+ THEN wv.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) AS op,
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}' AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}' AS dvp
+ ) f
+ WHERE jsonb_typeof(wv.content::jsonb -> 'operators') = 'array'
+ AND (
+ (f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 3)))
+ OR
+ (f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3)))
+ OR
+ (f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 2)))
+ OR
+ (f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2)))
+ )
+ GROUP BY wv.vid
+ ),
+ updated AS (
+ UPDATE workflow_version wv
+ SET content = jsonb_set(wv.content::jsonb, '{operators}', (
+ SELECT jsonb_agg(
+ jsonb_set(
+ jsonb_set(
+ op,
+ '{operatorProperties,fileName}',
+ CASE
+ WHEN f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.fn, '^/datasets/', '/dataset/'))
+ WHEN f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.fn, '/'))
+ ELSE COALESCE(op #> '{operatorProperties,fileName}', 'null'::jsonb)
+ END,
+ false
+ ),
+ '{operatorProperties,datasetVersionPath}',
+ CASE
+ WHEN f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.dvp, '^/datasets/', '/dataset/'))
+ WHEN f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1)
+ AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.dvp, '/'))
+ ELSE COALESCE(op #> '{operatorProperties,datasetVersionPath}', 'null'::jsonb)
+ END,
+ false
+ )
+ ORDER BY ord
+ )
+ FROM jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(wv.content::jsonb -> 'operators') = 'array'
+ THEN wv.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) WITH ORDINALITY AS t(op, ord),
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}' AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}' AS dvp
+ ) f
+ ))::text
+ FROM affected a
+ WHERE wv.vid = a.vid
+ RETURNING 1
+ )
+ SELECT count(*) INTO wv_count FROM updated;
+
+ RAISE NOTICE 'Normalized the resource-type path prefix in % workflow and % workflow_version row(s).', wf_count, wv_count;
+END $$;
+
+COMMIT;