You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "AnzhiZhang (via GitHub)" <gi...@apache.org> on 2025/09/04 13:35:15 UTC
[PR] feat: allow datasets with the same name across different users [texera]
AnzhiZhang opened a new pull request, #3715:
URL: https://github.com/apache/texera/pull/3715
resolves #3683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365246537
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Yes it is correct. This is creating `DashboardDataset` and pass these params. It is not a self assignment, the left side `dataset` is the parameter name, right side is the object we are passing to the constructor.
Line 321:
```scala
val dataset = record.into(DATASET).into(classOf[Dataset])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365246537
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Yes it is correct. This is creating `DashboardDataset` and pass these params. It is not a self assignment, the left side `dataset` is the argument name, right side is the object we are passing to the constructor.
Line 321:
```scala
val dataset = record.into(DATASET).into(classOf[Dataset])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2331591158
##########
core/scripts/sql/updates/13.sql:
##########
@@ -0,0 +1,39 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Update the table schema
+-- ============================================
+
+BEGIN;
+
+-- 1. Add new column repository_name to dataset table.
+ALTER TABLE dataset
+ ADD COLUMN repository_name varchar(128);
+
+-- 2. Copy the data from name column to repository_name column.
+UPDATE dataset
+SET repository_name = name;
Review Comment:
This is intended for backward compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2400199787
##########
core/config/src/main/scala/edu/uci/ics/amber/config/StorageConfig.scala:
##########
@@ -65,15 +65,15 @@ object StorageConfig {
conf.getInt("storage.iceberg.table.commit.retry.max-wait-ms")
// LakeFS specifics
- val lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
+ var lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
Review Comment:
Setting these environment variables in the test code is too late for the `StorageConfig` to read. I tried `lazy val` as well, and it also does not work. Do you have any other suggestions?
```scala
System.setProperty("STORAGE_S3_ENDPOINT", minioEndpoint)
System.setProperty("STORAGE_LAKEFS_ENDPOINT", s"$lakefsBaseUrl/api/v1")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2395254148
##########
core/workflow-core/src/main/scala/edu/uci/ics/amber/core/storage/util/LakeFSStorageClient.scala:
##########
@@ -77,22 +77,23 @@ object LakeFSStorageClient {
* Initializes a new repository in LakeFS.
*
* @param repoName Name of the repository.
- * @param defaultBranch Default branch name, usually "main".
*/
def initRepo(
repoName: String
): Repository = {
+ // validate repoName
+ // https://docs.lakefs.io/latest/understand/model/#repository
Review Comment:
merge this comment into one line
##########
core/config/src/main/scala/edu/uci/ics/amber/config/StorageConfig.scala:
##########
@@ -65,15 +65,15 @@ object StorageConfig {
conf.getInt("storage.iceberg.table.commit.retry.max-wait-ms")
// LakeFS specifics
- val lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
+ var lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
Review Comment:
This change seem to be unnecessary
##########
core/workflow-core/src/main/scala/edu/uci/ics/amber/core/storage/FileResolver.scala:
##########
@@ -80,7 +80,7 @@ object FileResolver {
*
* The fileName format should be: /ownerEmail/datasetName/versionName/fileRelativePath
* e.g. /bob@texera.com/twitterDataset/v1/california/irvine/tw1.csv
- * The output dataset URI format is: {DATASET_FILE_URI_SCHEME}:///{did}/{versionHash}/file-path
+ * The output dataset URI format is: {DATASET_FILE_URI_SCHEME}:///{repositoryName}/{versionHash}/fileRelativePath
* e.g. {DATASET_FILE_URI_SCHEME}:///15/adeq233td/some/dir/file.txt
Review Comment:
update this comment also
##########
core/config/src/main/scala/edu/uci/ics/amber/config/StorageConfig.scala:
##########
@@ -65,15 +65,15 @@ object StorageConfig {
conf.getInt("storage.iceberg.table.commit.retry.max-wait-ms")
// LakeFS specifics
- val lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
+ var lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
Review Comment:
I see that this is for the mock lakeFS and S3. Can you try to set the environment variable to change this value? i.e. see `conf/src/main/resources/storage.conf`, each variable has a env var corresponding to it, which can be used to change the actual value of the config value
##########
core/config/src/main/scala/edu/uci/ics/amber/config/StorageConfig.scala:
##########
@@ -65,15 +65,15 @@ object StorageConfig {
conf.getInt("storage.iceberg.table.commit.retry.max-wait-ms")
// LakeFS specifics
- val lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
+ var lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
val lakefsApiSecret: String = conf.getString("storage.lakefs.auth.api-secret")
val lakefsUsername: String = conf.getString("storage.lakefs.auth.username")
val lakefsPassword: String = conf.getString("storage.lakefs.auth.password")
val lakefsBlockStorageType: String = conf.getString("storage.lakefs.block-storage.type")
val lakefsBucketName: String = conf.getString("storage.lakefs.block-storage.bucket-name")
// S3 specifics
- val s3Endpoint: String = conf.getString("storage.s3.endpoint")
+ var s3Endpoint: String = conf.getString("storage.s3.endpoint")
Review Comment:
Ditto
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
accessPrivilege = datasetAccess.getPrivilege,
ownerEmail = ownerEmail,
Review Comment:
Ditto
##########
deployment/k8s/texera-helmchart/files/texera_ddl.sql:
##########
@@ -264,13 +264,14 @@ CREATE TABLE IF NOT EXISTS public_project
-- dataset
CREATE TABLE IF NOT EXISTS dataset
(
- did SERIAL PRIMARY KEY,
- owner_uid INT NOT NULL,
- name VARCHAR(128) NOT NULL,
- is_public BOOLEAN NOT NULL DEFAULT TRUE,
+ did SERIAL PRIMARY KEY,
+ owner_uid INT NOT NULL,
+ name VARCHAR(128) NOT NULL,
+ repository_name VARCHAR(128),
+ is_public BOOLEAN NOT NULL DEFAULT TRUE,
is_downloadable BOOLEAN NOT NULL DEFAULT TRUE,
- description VARCHAR(512) NOT NULL,
- creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ description VARCHAR(512) NOT NULL,
+ creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
Review Comment:
revert this unnecessary change
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
This is correct. `dataset = dataset`, the left side is the parameter name, and right side is the actual variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2395501668
##########
deployment/k8s/texera-helmchart/files/texera_ddl.sql:
##########
@@ -264,13 +264,14 @@ CREATE TABLE IF NOT EXISTS public_project
-- dataset
CREATE TABLE IF NOT EXISTS dataset
(
- did SERIAL PRIMARY KEY,
- owner_uid INT NOT NULL,
- name VARCHAR(128) NOT NULL,
- is_public BOOLEAN NOT NULL DEFAULT TRUE,
+ did SERIAL PRIMARY KEY,
+ owner_uid INT NOT NULL,
+ name VARCHAR(128) NOT NULL,
+ repository_name VARCHAR(128),
+ is_public BOOLEAN NOT NULL DEFAULT TRUE,
is_downloadable BOOLEAN NOT NULL DEFAULT TRUE,
- description VARCHAR(512) NOT NULL,
- creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ description VARCHAR(512) NOT NULL,
+ creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
Review Comment:
@bobbai00 It will be good if you don't approve this PR before these comments are taken care of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365246537
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Yes it is correct. This is creating `DashboardDataset` and pass these params. Left side `dataset` is the parameter name, right side is the object we are passing to the constructor.
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
accessPrivilege = datasetAccess.getPrivilege,
ownerEmail = ownerEmail,
Review Comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2344530155
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
Considering the dataset name will be used in the URI, this is intended to require the user to input a meaningful name (not an empty string, weird characters, etc.). We can discuss that. I am fine with removing such requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #3715:
URL: https://github.com/apache/texera/pull/3715#issuecomment-3285875637
@AnzhiZhang It will be good to divide this PR to smaller PRs. I think it's doable. If needed, please schedule a call with me and @bobbai00 to discuss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on PR #3715:
URL: https://github.com/apache/texera/pull/3715#issuecomment-3273425172
Parts of this PR are opened in #3723 and #3724. Once they are merged, these changes should disappear from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2400292919
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
Please do as suggested above.
1. create a separate static function `validateDatasetName` at `object DatasetResource`
2. use it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2399530064
##########
deployment/k8s/texera-helmchart/files/texera_ddl.sql:
##########
@@ -264,13 +264,14 @@ CREATE TABLE IF NOT EXISTS public_project
-- dataset
CREATE TABLE IF NOT EXISTS dataset
(
- did SERIAL PRIMARY KEY,
- owner_uid INT NOT NULL,
- name VARCHAR(128) NOT NULL,
- is_public BOOLEAN NOT NULL DEFAULT TRUE,
+ did SERIAL PRIMARY KEY,
+ owner_uid INT NOT NULL,
+ name VARCHAR(128) NOT NULL,
+ repository_name VARCHAR(128),
+ is_public BOOLEAN NOT NULL DEFAULT TRUE,
is_downloadable BOOLEAN NOT NULL DEFAULT TRUE,
- description VARCHAR(512) NOT NULL,
- creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ description VARCHAR(512) NOT NULL,
+ creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
Review Comment:
We do NOT want to include such style changes in this PR as they have nothing to do with this PR and they can make the PR harder to review. Please revert it as suggested by @bobbai00 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2400259326
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
Please check the current code for validating the dataset name in `DatasetResource`, and let me know if you would prefer a separate static function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2374208879
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
This is updated, please review @bobbai00
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2400288764
##########
core/config/src/main/scala/edu/uci/ics/amber/config/StorageConfig.scala:
##########
@@ -65,15 +65,15 @@ object StorageConfig {
conf.getInt("storage.iceberg.table.commit.retry.max-wait-ms")
// LakeFS specifics
- val lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
+ var lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
Review Comment:
ok in this case, please leave the comments at these two variables that: this is var because during the testing the values will be replaced by mock lakeFS and S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365898664
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Here is a tutorial and a document page for reference:
https://docs.scala-lang.org/tour/named-arguments.html
https://docs.scala-lang.org/sips/named-and-default-arguments.html#named-arguments
It is indeed a good practice to have a similar name on the left and right sides if we are accepting a specific object. We will then know that we are passing the correct variable to the correct argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365876233
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
I still had a hard time understanding how the compiler can differentiate between these two variables of `dataset`. @bobbai00 can you chime in? Maybe we can change it slightly to improve the readability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2343137696
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
You can remove this. The datasetName is now user-given name, not the actual repository name. So you don't need to validate the datasetName using lakeFS rule.
##########
core/workflow-core/src/main/scala/edu/uci/ics/amber/core/storage/util/LakeFSStorageClient.scala:
##########
@@ -74,25 +74,36 @@ object LakeFSStorageClient {
}
/**
- * Initializes a new repository in LakeFS.
+ * Validates the repository name against LakeFS naming conventions.
*
- * @param repoName Name of the repository.
- * @param defaultBranch Default branch name, usually "main".
+ * @param repoName name of the repository to validate.
+ * @throws IllegalArgumentException if the repository name is invalid.
*/
- def initRepo(
- repoName: String
- ): Repository = {
+ def validateRepositoryName(repoName: String): Unit = {
Review Comment:
As stated above, this method seems not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2361883285
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365215462
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
accessPrivilege = datasetAccess.getPrivilege,
ownerEmail = ownerEmail,
Review Comment:
Is this self assignment correct?
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Is this self assignment correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dataset): allow datasets to have the same name across different users [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 merged PR #3715:
URL: https://github.com/apache/texera/pull/3715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2400296782
##########
core/config/src/main/scala/edu/uci/ics/amber/config/StorageConfig.scala:
##########
@@ -65,15 +65,15 @@ object StorageConfig {
conf.getInt("storage.iceberg.table.commit.retry.max-wait-ms")
// LakeFS specifics
- val lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
+ var lakefsEndpoint: String = conf.getString("storage.lakefs.endpoint")
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2400242147
##########
deployment/k8s/texera-helmchart/files/texera_ddl.sql:
##########
@@ -264,13 +264,14 @@ CREATE TABLE IF NOT EXISTS public_project
-- dataset
CREATE TABLE IF NOT EXISTS dataset
(
- did SERIAL PRIMARY KEY,
- owner_uid INT NOT NULL,
- name VARCHAR(128) NOT NULL,
- is_public BOOLEAN NOT NULL DEFAULT TRUE,
+ did SERIAL PRIMARY KEY,
+ owner_uid INT NOT NULL,
+ name VARCHAR(128) NOT NULL,
+ repository_name VARCHAR(128),
+ is_public BOOLEAN NOT NULL DEFAULT TRUE,
is_downloadable BOOLEAN NOT NULL DEFAULT TRUE,
- description VARCHAR(512) NOT NULL,
- creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ description VARCHAR(512) NOT NULL,
+ creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
Review Comment:
reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #3715:
URL: https://github.com/apache/texera/pull/3715#issuecomment-3357648037
@AnzhiZhang After all comments by @bobbai00 and @aicam are taken care of, I will approve this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2366435399
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
@bobbai00 Let's discuss it in our next in-person meeting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2331068509
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -288,6 +315,7 @@ class DatasetResource {
createdDataset.getDid,
createdDataset.getOwnerUid,
createdDataset.getName,
+ createdDataset.getRepositoryName,
Review Comment:
The return statement for `DashboardDataset` constructor is missing a parameter. Based on the context, it should include `createdDataset.getRepositoryName` but the constructor expects 6 parameters: `did`, `ownerUid`, `name`, `repositoryName`, `isPublic`, `isDownloadable`, `description`, `creationTime`, `lastModifiedTime`. The current call only provides 9 parameters but seems to be missing proper parameter alignment.
##########
core/scripts/sql/updates/13.sql:
##########
@@ -0,0 +1,39 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Update the table schema
+-- ============================================
+
+BEGIN;
+
+-- 1. Add new column repository_name to dataset table.
+ALTER TABLE dataset
+ ADD COLUMN repository_name varchar(128);
+
+-- 2. Copy the data from name column to repository_name column.
+UPDATE dataset
+SET repository_name = name;
Review Comment:
The migration script copies dataset names directly to repository_name, but the new system expects repository names in format `dataset-{did}`. This migration will create inconsistency with the new repository naming scheme. The repository_name should be set to `CONCAT('dataset-', did)` instead of copying the name directly.
```suggestion
SET repository_name = CONCAT('dataset-', did);
```
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -1280,29 +1322,29 @@ class DatasetResource {
private def resolveDatasetAndPath(
encodedUrl: String,
- datasetName: String,
+ repositoryName: String,
commitHash: String,
uid: Integer
): Either[Response, (String, String, String)] = {
val decodedPathStr = URLDecoder.decode(encodedUrl, StandardCharsets.UTF_8.name())
- (Option(datasetName), Option(commitHash)) match {
+ (Option(repositoryName), Option(commitHash)) match {
case (Some(_), None) | (None, Some(_)) =>
// Case 1: Only one parameter is provided (error case)
Left(
Response
.status(Response.Status.BAD_REQUEST)
.entity(
- "Both datasetName and commitHash must be provided together, or neither should be provided."
+ "Both repositoryName and commitHash must be provided together, or neither should be provided."
)
.build()
)
- case (Some(dsName), Some(commit)) =>
- // Case 2: datasetName and commitHash are provided, validate access
+ case (Some(repositoryName), Some(commit)) =>
+ // Case 2: repositoryName and commitHash are provided, validate access
val response = withTransaction(context) { ctx =>
val datasetDao = new DatasetDao(ctx.configuration())
- val datasets = datasetDao.fetchByName(dsName).asScala.toList
+ val datasets = datasetDao.fetchByRepositoryName(repositoryName).asScala.toList
Review Comment:
The code attempts to call `fetchByRepositoryName()` on `DatasetDao`, but this method may not exist. The existing code pattern suggests `fetchByName()` was used previously for dataset names. You need to ensure `fetchByRepositoryName()` method is implemented in `DatasetDao` or use an alternative query approach.
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -1311,17 +1353,18 @@ class DatasetResource {
// Standard read access check only - download restrictions handled per endpoint
// Non-download operations (viewing) should work for all public datasets
- (dsName, commit, decodedPathStr)
+ (repositoryName, commit, decodedPathStr)
}
Right(response)
case (None, None) =>
- // Case 3: Neither datasetName nor commitHash are provided, resolve normally
+ // Case 3: Neither repositoryName nor commitHash are provided, resolve normally
val response = withTransaction(context) { ctx =>
val fileUri = FileResolver.resolve(decodedPathStr)
val document = DocumentFactory.openReadonlyDocument(fileUri).asInstanceOf[OnDataset]
val datasetDao = new DatasetDao(ctx.configuration())
- val datasets = datasetDao.fetchByName(document.getDatasetName()).asScala.toList
+ val datasets =
+ datasetDao.fetchByRepositoryName(document.getRepositoryName()).asScala.toList
Review Comment:
Same issue as above - `fetchByRepositoryName()` method may not exist in `DatasetDao`. This could cause runtime errors when this code path is executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #3715:
URL: https://github.com/apache/texera/pull/3715#issuecomment-3263191432
The PR should include test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on PR #3715:
URL: https://github.com/apache/texera/pull/3715#issuecomment-3258322042
@chenlica Updated. Tests have not been added yet. Should I add some tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2331591621
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -1280,29 +1322,29 @@ class DatasetResource {
private def resolveDatasetAndPath(
encodedUrl: String,
- datasetName: String,
+ repositoryName: String,
commitHash: String,
uid: Integer
): Either[Response, (String, String, String)] = {
val decodedPathStr = URLDecoder.decode(encodedUrl, StandardCharsets.UTF_8.name())
- (Option(datasetName), Option(commitHash)) match {
+ (Option(repositoryName), Option(commitHash)) match {
case (Some(_), None) | (None, Some(_)) =>
// Case 1: Only one parameter is provided (error case)
Left(
Response
.status(Response.Status.BAD_REQUEST)
.entity(
- "Both datasetName and commitHash must be provided together, or neither should be provided."
+ "Both repositoryName and commitHash must be provided together, or neither should be provided."
)
.build()
)
- case (Some(dsName), Some(commit)) =>
- // Case 2: datasetName and commitHash are provided, validate access
+ case (Some(repositoryName), Some(commit)) =>
+ // Case 2: repositoryName and commitHash are provided, validate access
val response = withTransaction(context) { ctx =>
val datasetDao = new DatasetDao(ctx.configuration())
- val datasets = datasetDao.fetchByName(dsName).asScala.toList
+ val datasets = datasetDao.fetchByRepositoryName(repositoryName).asScala.toList
Review Comment:
We created a new repository name, this is correct.
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -1311,17 +1353,18 @@ class DatasetResource {
// Standard read access check only - download restrictions handled per endpoint
// Non-download operations (viewing) should work for all public datasets
- (dsName, commit, decodedPathStr)
+ (repositoryName, commit, decodedPathStr)
}
Right(response)
case (None, None) =>
- // Case 3: Neither datasetName nor commitHash are provided, resolve normally
+ // Case 3: Neither repositoryName nor commitHash are provided, resolve normally
val response = withTransaction(context) { ctx =>
val fileUri = FileResolver.resolve(decodedPathStr)
val document = DocumentFactory.openReadonlyDocument(fileUri).asInstanceOf[OnDataset]
val datasetDao = new DatasetDao(ctx.configuration())
- val datasets = datasetDao.fetchByName(document.getDatasetName()).asScala.toList
+ val datasets =
+ datasetDao.fetchByRepositoryName(document.getRepositoryName()).asScala.toList
Review Comment:
Same as above - We created a new repository name, this is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2331594158
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -288,6 +315,7 @@ class DatasetResource {
createdDataset.getDid,
createdDataset.getOwnerUid,
createdDataset.getName,
+ createdDataset.getRepositoryName,
Review Comment:
Copilot misunderstood the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2399220139
##########
deployment/k8s/texera-helmchart/files/texera_ddl.sql:
##########
@@ -264,13 +264,14 @@ CREATE TABLE IF NOT EXISTS public_project
-- dataset
CREATE TABLE IF NOT EXISTS dataset
(
- did SERIAL PRIMARY KEY,
- owner_uid INT NOT NULL,
- name VARCHAR(128) NOT NULL,
- is_public BOOLEAN NOT NULL DEFAULT TRUE,
+ did SERIAL PRIMARY KEY,
+ owner_uid INT NOT NULL,
+ name VARCHAR(128) NOT NULL,
+ repository_name VARCHAR(128),
+ is_public BOOLEAN NOT NULL DEFAULT TRUE,
is_downloadable BOOLEAN NOT NULL DEFAULT TRUE,
- description VARCHAR(512) NOT NULL,
- creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ description VARCHAR(512) NOT NULL,
+ creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
Review Comment:
This is to make sure types are aligned vertically for readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2395286324
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
Semantically,
- datasetName is user given
- repositoryName is allocated by internal services.
Therefore, although LakeFS has a good validation rule, it is misleading to call `LakeFSStorageClient.validateRepositoryName()` on the `datasetName`.
You can do the following here:
- create a static function in `DatasetResource` named `validateDatasetName`. This function does the exact same thing as `validateRepositoryName`.
- call validateDatasetName here.
This duplication is meaningful as it makes the semantic consistent. Also it provides the flexibility for us to modify the validation rule later, without affecting the lakefs repository name validation
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
Semantically,
- datasetName is user given
- repositoryName is allocated by internal services.
Therefore, although LakeFS has a good validation rule, it is misleading to call `LakeFSStorageClient.validateRepositoryName()` on the `datasetName`.
You can do the following here:
- create a static function in `DatasetResource` named `validateDatasetName`. This function does the exact same thing as `validateRepositoryName`.
- call validateDatasetName here.
This duplication is meaningful as it makes the semantic consistent. Also it provides the flexibility for us to modify the validation rule later, without affecting the lakefs repository name validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #3715:
URL: https://github.com/apache/texera/pull/3715#issuecomment-3255850200
@AnzhiZhang Please summarize the final design (after our internal discussion) and give an overview of the changes. See if we can split the PR into smaller PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2361883285
##########
core/file-service/src/main/scala/edu/uci/ics/texera/service/resource/DatasetResource.scala:
##########
@@ -239,43 +239,70 @@ class DatasetResource {
withTransaction(context) { ctx =>
val uid = user.getUid
- val datasetDao: DatasetDao = new DatasetDao(ctx.configuration())
val datasetUserAccessDao: DatasetUserAccessDao = new DatasetUserAccessDao(ctx.configuration())
val datasetName = request.datasetName
val datasetDescription = request.datasetDescription
val isDatasetPublic = request.isDatasetPublic
val isDatasetDownloadable = request.isDatasetDownloadable
- // Check if a dataset with the same name already exists
- if (!datasetDao.fetchByName(datasetName).isEmpty) {
- throw new BadRequestException("Dataset with the same name already exists")
- }
-
- // Initialize the repository in LakeFS
+ // validate dataset name
try {
- LakeFSStorageClient.initRepo(datasetName)
+ LakeFSStorageClient.validateRepositoryName(datasetName)
Review Comment:
resolved
##########
core/workflow-core/src/main/scala/edu/uci/ics/amber/core/storage/util/LakeFSStorageClient.scala:
##########
@@ -74,25 +74,36 @@ object LakeFSStorageClient {
}
/**
- * Initializes a new repository in LakeFS.
+ * Validates the repository name against LakeFS naming conventions.
*
- * @param repoName Name of the repository.
- * @param defaultBranch Default branch name, usually "main".
+ * @param repoName name of the repository to validate.
+ * @throws IllegalArgumentException if the repository name is invalid.
*/
- def initRepo(
- repoName: String
- ): Repository = {
+ def validateRepositoryName(repoName: String): Unit = {
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: allow datasets with the same name across different users [texera]
Posted by "AnzhiZhang (via GitHub)" <gi...@apache.org>.
AnzhiZhang commented on code in PR #3715:
URL: https://github.com/apache/texera/pull/3715#discussion_r2365246537
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Yes it is correct. This is creating `DashboardDataset` and pass these params. Left side `dataset` is the parameter name, right side is the object we are passing to the constructor.
Line 321:
```
val dataset = record.into(DATASET).into(classOf[Dataset])
```
##########
core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard/hub/HubResource.scala:
##########
@@ -326,7 +326,7 @@ object HubResource {
dataset = dataset,
Review Comment:
Yes it is correct. This is creating `DashboardDataset` and pass these params. Left side `dataset` is the parameter name, right side is the object we are passing to the constructor.
Line 321:
```scala
val dataset = record.into(DATASET).into(classOf[Dataset])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org