You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Sentiaus (via GitHub)" <gi...@apache.org> on 2026/07/16 09:21:36 UTC
[PR] feat(file-service): Backend Routes [texera]
Sentiaus opened a new pull request, #6447:
URL: https://github.com/apache/texera/pull/6447
### What changes were proposed in this PR?
This PR adds server-side Google Drive export endpoints for datasets and workflows.
**New endpoints:**
- `POST /dataset/{did}/drive-export` — Exports a dataset version (specified by `dvid` or `latest=true`) as a zip archive directly to a Google Drive resumable upload session URI.
- `POST /dataset/drive-export/file` — Exports a single file from a dataset version to Drive.
- `POST /workflow/{wid}/export/drive` — Exports a workflow's JSON content to Drive.
**Security:**
- All three endpoints validate that the session URI starts with `https://www.googleapis.com/upload/drive/` before making any outbound connection, preventing SSRF.
- Dataset endpoints check read access and the dataset's `isDownloadable` flag before streaming data.
- Workflow endpoint checks read access via `WorkflowAccessResource`.
**Size limits:**
- Dataset version export: capped at 500 MB (`DRIVE_EXPORT_MAX_DATASET_BYTES`). The zip is built in memory before the PUT to Drive; this limit will be lifted once chunked/streaming resumable upload is implemented.
- Single file export: 5 TB (`DRIVE_EXPORT_MAX_FILE_BYTES`) — Google Drive's resumable upload ceiling.
- Workflow export: capped at 500 MB (`DRIVE_EXPORT_MAX_BYTES`) since the JSON content is read into a byte array before upload.
**Config (`user-system.conf`):**
- Adds `clientSecret` and `apiKey` fields (empty by default, overridable via `USER_SYS_GOOGLE_CLIENT_SECRET` / `USER_SYS_GOOGLE_API_KEY` env vars) so the frontend can fetch them from `GoogleAuthResource`.
**`GoogleAuthResource`:**
- Adds `GET /auth/google/drive/apikey` endpoint returning `UserSystemConfig.googleApiKey` so the frontend can initialize the Google Picker without embedding credentials in the build.
### Any related issues, documentation, discussions?
Part of the Google Drive export feature. The frontend PR (stacked on this one) wires up the UI.
### How was this PR tested?
- `DatasetResourceSpec` — new integration tests (using `MockTexeraDB` + `MockLakeFS` testcontainers) cover:
- Successful version export to Drive (verifying the PUT to the session URI is made)
- Successful single-file export to Drive
- Rejection of non-googleapis session URIs (`400 Bad Request`)
- `403` when the dataset is not downloadable
- `400` when both `dvid` and `latest` are provided (or neither)
- `WorkflowResource` endpoint tested manually (recompile + restart required to pick up the new route).
- SSRF validation verified: passing a non-`https://www.googleapis.com/upload/drive/` URI returns `400`.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-5986677009
This pull request has been automatically closed due to 37 days of inactivity.
If you would like to continue this work, please re-open the PR or open a new
one referencing this. Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file-service): Backend Routes [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-4990209217
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@Ma77Ball`, `@carloea2`, `@VuMartin`
You can notify them by mentioning `@Ma77Ball`, `@carloea2`, `@VuMartin` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-5861803178
This pull request has been automatically marked as **stale** because it has
not had any activity for 30 days. It will be closed in **7 days** if no
further activity occurs.
If this PR is still relevant, please:
- Push new commits or rebase onto the latest `main`, or
- Comment with a status update, or
- Address any open review feedback.
To exempt this PR from auto-close, a maintainer can add the `pinned` label.
Thank you for your contribution to Apache Texera (incubating)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "Sentiaus (via GitHub)" <gi...@apache.org>.
Sentiaus commented on code in PR #6447:
URL: https://github.com/apache/texera/pull/6447#discussion_r3636495632
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1361,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
Review Comment:
@xuang7 removed the building of the ZIP and Drive PUT request from withTransaction.
I'm not able to use StreamingOutput because this would require implementing the logic + error handling from hitting Google's resumable API repeatedly (This is the end goal), however this would add additional complexity that hasn't been discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on code in PR #6447:
URL: https://github.com/apache/texera/pull/6447#discussion_r3610273350
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1298,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
+ if ((dvid != null && latest != null) || (dvid == null && latest == null)) {
+ throw new BadRequestException("Specify exactly one: dvid=<ID> OR latest=true")
+ }
+
+ val uid = user.getUid
+ if (!userHasReadAccess(ctx, did, uid)) {
+ throw new ForbiddenException(ERR_USER_HAS_NO_ACCESS_TO_DATASET_MESSAGE)
+ }
+
+ val dataset = getDatasetByID(ctx, did)
+ if (!userOwnDataset(ctx, did, uid) && !dataset.getIsDownloadable) {
+ throw new ForbiddenException("Dataset download is not allowed")
+ }
+
+ val datasetVersion = if (dvid != null) {
+ getDatasetVersionByID(ctx, dvid)
+ } else {
+ getLatestDatasetVersion(ctx, did).getOrElse(
+ throw new NotFoundException(ERR_DATASET_VERSION_NOT_FOUND_MESSAGE)
+ )
+ }
+
+ val repositoryName = dataset.getRepositoryName
+ val versionHash = datasetVersion.getVersionHash
+ val objects = withLakeFSErrorHandling(
+ s"listing files of version '$versionHash' of dataset '${dataset.getName}'"
+ ) {
+ LakeFSStorageClient.retrieveObjectsOfVersion(repositoryName, versionHash)
+ }
+
+ if (objects.isEmpty) {
+ throw new NotFoundException(s"No objects found in version $versionHash")
+ }
+
+ val totalBytes = objects.map(_.getSizeBytes.longValue()).sum
+ // TODO: remove this limit once chunked resumable upload to Drive is implemented
+ if (totalBytes > DRIVE_EXPORT_MAX_DATASET_BYTES) {
+ throw new BadRequestException("Dataset version exceeds the 500 MB export limit.")
+ }
+ if (totalBytes > DRIVE_EXPORT_MAX_FILE_BYTES) {
+ throw new ForbiddenException("Dataset version exceeds the 5 TB Google Drive export limit.")
+ }
Review Comment:
The 500 MB check is the first to run for any size over 500 MB, so this 5 TB branch is unreachable dead code here. I recommend dropping it until the feature is added:
```suggestion
if (totalBytes > DRIVE_EXPORT_MAX_DATASET_BYTES) {
throw new BadRequestException("Dataset version exceeds the 500 MB export limit.")
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file-service): Backend Routes [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-5000068554
Hi @Sentiaus, could you please make the PR title more informative? Also I am lack of context, there is no issue or discussion linked in the PR description, could you please add that and hopefully that can make the description shorter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file-service): Backend Routes [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-4990231866
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/6447?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.27%. Comparing base ([`0a5693c`](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cff5644`](https://app.codecov.io/gh/apache/texera/commit/cff5644ddd25e3f599224745a28bf27bc095833b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 239 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #6447 +/- ##
=========================================
Coverage 55.27% 55.27%
Complexity 2991 2991
=========================================
Files 1117 1117
Lines 43258 43259 +1
Branches 4668 4668
=========================================
+ Hits 23912 23913 +1
Misses 17938 17938
Partials 1408 1408
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.36% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.80% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.30% <ø> (+0.74%)` | :arrow_up: | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.02% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.87% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.57% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.20% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [python](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.76% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/6447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.14% <ø> (ø)` | | Carriedforward from [0a5693c](https://app.codecov.io/gh/apache/texera/commit/0a5693cb9a2e6210163e5588924bbd31d3730536?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/6447?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
xuang7 commented on code in PR #6447:
URL: https://github.com/apache/texera/pull/6447#discussion_r3626225070
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1361,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
+ if ((dvid != null && latest != null) || (dvid == null && latest == null)) {
+ throw new BadRequestException("Specify exactly one: dvid=<ID> OR latest=true")
+ }
+
+ val uid = user.getUid
+ if (!userHasReadAccess(ctx, did, uid)) {
+ throw new ForbiddenException(ERR_USER_HAS_NO_ACCESS_TO_DATASET_MESSAGE)
+ }
+
+ val dataset = getDatasetByID(ctx, did)
+ if (!userOwnDataset(ctx, did, uid) && !dataset.getIsDownloadable) {
+ throw new ForbiddenException("Dataset download is not allowed")
+ }
+
+ val datasetVersion = if (dvid != null) {
+ getDatasetVersionByID(ctx, dvid)
+ } else {
+ getLatestDatasetVersion(ctx, did).getOrElse(
+ throw new NotFoundException(ERR_DATASET_VERSION_NOT_FOUND_MESSAGE)
+ )
+ }
+
+ val repositoryName = dataset.getRepositoryName
+ val versionHash = datasetVersion.getVersionHash
+ val objects = withLakeFSErrorHandling(
+ s"listing files of version '$versionHash' of dataset '${dataset.getName}'"
+ ) {
+ LakeFSStorageClient.retrieveObjectsOfVersion(repositoryName, versionHash)
+ }
+
+ if (objects.isEmpty) {
+ throw new NotFoundException(s"No objects found in version $versionHash")
+ }
+
+ val totalBytes = objects.map(_.getSizeBytes.longValue()).sum
+ // TODO: remove this limit once chunked resumable upload to Drive is implemented
+ if (totalBytes > DRIVE_EXPORT_MAX_DATASET_BYTES) {
+ throw new BadRequestException("Dataset version exceeds the 500 MB export limit.")
+ }
+ if (totalBytes > DRIVE_EXPORT_MAX_FILE_BYTES) {
+ throw new ForbiddenException("Dataset version exceeds the 5 TB Google Drive export limit.")
+ }
+
+ // Build the zip in memory
+ val baos = new ByteArrayOutputStream()
+ val zipOut = new java.util.zip.ZipOutputStream(baos)
+ try {
+ objects.foreach { obj =>
+ val filePath = obj.getPath
+ val file = withLakeFSErrorHandling(s"downloading file '$filePath' for Drive export") {
+ LakeFSStorageClient.getFileFromRepo(repositoryName, versionHash, filePath)
+ }
+ zipOut.putNextEntry(new java.util.zip.ZipEntry(filePath))
+ Files.copy(Paths.get(file.toURI), zipOut)
+ zipOut.closeEntry()
+ }
+ } finally {
+ zipOut.close()
+ }
+
+ val zipBytes = baos.toByteArray
+ val sessionUri = request.sessionUri
+
+ // PUT the zip to the Google Drive session URI
+ val conn = new URL(sessionUri).openConnection().asInstanceOf[HttpURLConnection]
+ try {
+ conn.setDoOutput(true)
+ conn.setRequestMethod("PUT")
+ conn.setRequestProperty("Content-Type", "application/zip")
+ conn.setFixedLengthStreamingMode(zipBytes.length)
+ val out = conn.getOutputStream
+ out.write(zipBytes)
+ out.close()
+
+ val code = conn.getResponseCode
+ if (code < 200 || code >= 300) {
+ val body =
+ Option(conn.getErrorStream).map(s => new String(s.readAllBytes())).getOrElse("")
+ if (body.contains("storageQuotaExceeded")) {
+ throw new ForbiddenException("Google Drive storage quota exceeded.")
+ }
+ throw new WebApplicationException(
+ s"Google Drive upload failed with HTTP $code",
+ Response.Status.BAD_GATEWAY
+ )
+ }
+ } finally {
+ conn.disconnect()
+ }
+
+ Response.ok(Map("message" -> "Dataset exported to Google Drive")).build()
+ }
+ }
+
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/drive-export/file")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportFileToDrive(
+ request: DriveFileExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ val uid = user.getUid
+ resolveDatasetAndPath(request.filePath, null, null, uid) match {
Review Comment:
resolveDatasetAndPath only checks read access ("download restrictions handled per endpoint" per its comment), and this endpoint never applies the owner-or-downloadable check that exportDatasetToDrive does at line 1333. Any user with read access to a non-downloadable dataset can export its files to their own Drive.
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1361,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
Review Comment:
The LakeFS downloads, in-memory zip, and Drive PUT all run inside withTransaction, holding one of the pool's 10 DB connections for the whole transfer (unbounded, since no read timeout is set). Note getDatasetVersionZip avoids this: its downloads run via StreamingOutput after the transaction commits.
##########
common/config/src/main/resources/user-system.conf:
##########
@@ -27,6 +27,12 @@ user-sys {
clientId = ""
clientId = ${?USER_SYS_GOOGLE_CLIENT_ID}
+ clientSecret = ""
+ clientSecret = ${?USER_SYS_GOOGLE_CLIENT_SECRET}
Review Comment:
ClientSecret is added but referenced nowhere in this PR. If the reworked frontend PR no longer does a server-side exchange, please drop this entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file-service): Backend Routes [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-4990254988
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 8 worse · ⚪ 7 noise (<±5%) · 0 without baseline
> Compared against main `c7473a1` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/29486850070)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 412 | 0.251 | 22,744/35,085/35,085 us | 🔴 +14.2% / 🔴 +147.9% |
| 🔴 | bs=100 sw=10 sl=64 | 914 | 0.558 | 107,059/134,537/134,537 us | 🔴 +20.4% / 🔴 +37.6% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,102 | 0.673 | 902,472/1,012,393/1,012,393 us | ⚪ within ±5% / 🔴 +7.5% |
<details><summary>Baseline details</summary>
Latest main `c7473a1` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 412 tuples/sec | 455 tuples/sec | 847.3 tuples/sec | -9.5% | -51.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.251 MB/s | 0.278 MB/s | 0.517 MB/s | -9.7% | -51.5% |
| bs=10 sw=10 sl=64 | p50 | 22,744 us | 19,911 us | 11,621 us | +14.2% | +95.7% |
| bs=10 sw=10 sl=64 | p95 | 35,085 us | 33,454 us | 14,154 us | +4.9% | +147.9% |
| bs=10 sw=10 sl=64 | p99 | 35,085 us | 33,454 us | 19,069 us | +4.9% | +84.0% |
| bs=100 sw=10 sl=64 | throughput | 914 tuples/sec | 1,003 tuples/sec | 1,104 tuples/sec | -8.9% | -17.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.558 MB/s | 0.612 MB/s | 0.674 MB/s | -8.8% | -17.2% |
| bs=100 sw=10 sl=64 | p50 | 107,059 us | 98,330 us | 91,279 us | +8.9% | +17.3% |
| bs=100 sw=10 sl=64 | p95 | 134,537 us | 111,747 us | 97,771 us | +20.4% | +37.6% |
| bs=100 sw=10 sl=64 | p99 | 134,537 us | 111,747 us | 107,798 us | +20.4% | +24.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,102 tuples/sec | 1,103 tuples/sec | 1,127 tuples/sec | -0.1% | -2.2% |
| bs=1000 sw=10 sl=64 | MB/s | 0.673 MB/s | 0.673 MB/s | 0.688 MB/s | 0.0% | -2.2% |
| bs=1000 sw=10 sl=64 | p50 | 902,472 us | 908,979 us | 902,553 us | -0.7% | -0.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,012,393 us | 1,026,081 us | 941,491 us | -1.3% | +7.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,012,393 us | 1,026,081 us | 979,025 us | -1.3% | +3.4% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,486.00,200,128000,412,0.251,22743.66,35085.16,35085.16
1,100,10,64,20,2188.24,2000,1280000,914,0.558,107058.69,134537.09,134537.09
2,1000,10,64,20,18144.75,20000,12800000,1102,0.673,902472.17,1012392.81,1012392.81
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
xuang7 commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-5297333163
Hi @Sentiaus, just checking in on the status of this PR. Would you like to continue working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-5433041302
This PR conflicts with its base branch and needs to be rebased before review can continue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "Sentiaus (via GitHub)" <gi...@apache.org>.
Sentiaus commented on PR #6447:
URL: https://github.com/apache/texera/pull/6447#issuecomment-5001349823
@Yicong-Huang sorry, I was making a last revision to the PR. Added the discussion and previous PR that I just closed. This is related to the google drive export discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #6447:
URL: https://github.com/apache/texera/pull/6447#discussion_r3741174276
##########
amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala:
##########
@@ -782,6 +787,41 @@ class WorkflowResource extends LazyLogging {
.fetchOneInto(classOf[String])
}
+ @POST
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{wid}/export/drive")
+ def exportWorkflowToDrive(
+ @PathParam("wid") wid: Integer,
+ request: WorkflowResource.DriveExportRequest,
+ @Auth sessionUser: SessionUser
+ ): Unit = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/"))
+ throw new BadRequestException("Invalid session URI")
+ if (!WorkflowAccessResource.hasReadAccess(wid, sessionUser.getUid))
+ throw new ForbiddenException("No sufficient access privilege.")
+ val content = workflowDao.fetchOneByWid(wid).getContent.getBytes("UTF-8")
+ if (content.length > WorkflowResource.DRIVE_EXPORT_MAX_BYTES)
+ throw new BadRequestException("Workflow content exceeds the 500 MB export limit.")
+ val conn = new URL(request.sessionUri).openConnection().asInstanceOf[HttpURLConnection]
+ try {
+ conn.setDoOutput(true)
+ conn.setRequestMethod("PUT")
+ conn.setRequestProperty("Content-Type", "application/json")
+ conn.setRequestProperty("Content-Length", content.length.toString)
+ conn.getOutputStream.write(content)
+ val code = conn.getResponseCode
+ if (code < 200 || code >= 300) {
+ val body = Option(conn.getErrorStream).map(s => new String(s.readAllBytes())).getOrElse("")
+ if (body.contains("storageQuotaExceeded"))
+ throw new ForbiddenException("Google Drive storage quota exceeded.")
+ throw new InternalServerErrorException(s"Google Drive upload failed: $body")
Review Comment:
This is the third copy of the same Drive upload; `DatasetResource.scala:1319` and `:1425` carry the other two. They have already drifted. Both DatasetResource copies return `502 BAD_GATEWAY` with just the status code. This one returns 500 and echoes Google's raw error body to the caller — so our client is told we broke, not the upstream.
I'd extract one helper holding the allowlist check, the PUT and the status handling. Patching the three copies to agree instead leaves the next edit free to split them again.
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1298,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
Review Comment:
Confirming @xuang7's thread rather than reopening it. At head `febaff79` this `withTransaction` still closes at line 1417. The LakeFS downloads (`:1373`), the in-memory zip (`:1367`) and the Drive PUT (`:1387-1414`) all run inside it, holding one pooled connection for the whole transfer with no read timeout.
`getDatasetVersionZip` avoids this by returning a `StreamingOutput` (`:1278-1301`). The full resumable-upload rework isn't needed here: keep the access checks and `retrieveObjectsOfVersion` inside the transaction, and move everything from `val totalBytes` onward outside it.
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1298,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
+ if ((dvid != null && latest != null) || (dvid == null && latest == null)) {
+ throw new BadRequestException("Specify exactly one: dvid=<ID> OR latest=true")
+ }
+
+ val uid = user.getUid
+ if (!userHasReadAccess(ctx, did, uid)) {
+ throw new ForbiddenException(ERR_USER_HAS_NO_ACCESS_TO_DATASET_MESSAGE)
+ }
+
+ val dataset = getDatasetByID(ctx, did)
+ if (!userOwnDataset(ctx, did, uid) && !dataset.getIsDownloadable) {
+ throw new ForbiddenException("Dataset download is not allowed")
+ }
+
+ val datasetVersion = if (dvid != null) {
+ getDatasetVersionByID(ctx, dvid)
+ } else {
+ getLatestDatasetVersion(ctx, did).getOrElse(
+ throw new NotFoundException(ERR_DATASET_VERSION_NOT_FOUND_MESSAGE)
+ )
+ }
Review Comment:
`?latest=false` reaches this branch and exports the latest version. The guard above only rejects "both set" and "neither set", so a non-null `false` falls straight through — an explicit "not the latest one" is honoured as "yes, the latest one". `getDatasetVersionZip` gates on `Boolean.TRUE.equals` and rejects everything else (`:1253`, `:1258`).
```suggestion
} else if (java.lang.Boolean.TRUE.equals(latest)) {
getLatestDatasetVersion(ctx, did).getOrElse(
throw new NotFoundException(ERR_DATASET_VERSION_NOT_FOUND_MESSAGE)
)
} else {
throw new BadRequestException("Specify exactly one: dvid=<ID> OR latest=true")
}
```
##########
amber/src/main/scala/org/apache/texera/web/resource/auth/GoogleAuthResource.scala:
##########
@@ -48,11 +48,17 @@ object GoogleAuthResource {
@Path("/auth/google")
class GoogleAuthResource {
final private lazy val clientId = UserSystemConfig.googleClientId
+ final private lazy val driveApiKey = UserSystemConfig.googleApiKey
@GET
@Path("/clientid")
def getClientId: String = clientId
+ @GET
+ @Path("/drive/apikey")
+ @Produces(Array(MediaType.TEXT_PLAIN))
+ def getDriveApiKey: String = driveApiKey
Review Comment:
This endpoint has no `@RolesAllowed`. Amber is also the one service that doesn't call `RoleAnnotationEnforcer.enforce`, so nothing rejects an anonymous request — any caller can read the deployment's Google API key and spend its quota.
The neighbouring `getClientId` is unannotated for a reason: the OAuth client id must be readable before login. The Picker key is only needed after login, so it doesn't inherit that. Adding `@RolesAllowed(Array("REGULAR", "ADMIN"))` also needs the `javax.annotation.security.RolesAllowed` import.
##########
file-service/src/test/scala/org/apache/texera/service/resource/DatasetResourceSpec.scala:
##########
@@ -2752,6 +2752,95 @@ class DatasetResourceSpec
fetchPartRows(uploadId) shouldBe empty
}
+ // ===========================================================================
+ // exportDatasetToDrive tests
+ // ===========================================================================
+
+ "exportDatasetToDrive" should "reject a session URI that does not start with the googleapis upload prefix" in {
+ val request = DatasetResource.DriveExportRequest("http://test.com/upload")
+ intercept[BadRequestException] {
+ datasetResource.exportDatasetToDrive(baseDataset.getDid, null, null, request, sessionUser)
Review Comment:
This test passes whether or not the session-URI check exists. With `dvid` and `latest` both null it trips the "Specify exactly one" guard first, which throws the same `BadRequestException`. The call is byte-identical to the one at `:2782` that tests that other guard, so `exportDatasetToDrive`'s SSRF control is currently unverified.
```suggestion
val latest: java.lang.Boolean = true
datasetResource.exportDatasetToDrive(baseDataset.getDid, null, latest, request, sessionUser)
```
##########
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:
##########
@@ -1298,6 +1305,187 @@ class DatasetResource extends LazyLogging {
}
}
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{did}/drive-export")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportDatasetToDrive(
+ @PathParam("did") did: Integer,
+ @QueryParam("dvid") dvid: Integer,
+ @QueryParam("latest") latest: java.lang.Boolean,
+ request: DriveExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ withTransaction(context) { ctx =>
+ if ((dvid != null && latest != null) || (dvid == null && latest == null)) {
+ throw new BadRequestException("Specify exactly one: dvid=<ID> OR latest=true")
+ }
+
+ val uid = user.getUid
+ if (!userHasReadAccess(ctx, did, uid)) {
+ throw new ForbiddenException(ERR_USER_HAS_NO_ACCESS_TO_DATASET_MESSAGE)
+ }
+
+ val dataset = getDatasetByID(ctx, did)
+ if (!userOwnDataset(ctx, did, uid) && !dataset.getIsDownloadable) {
+ throw new ForbiddenException("Dataset download is not allowed")
+ }
+
+ val datasetVersion = if (dvid != null) {
+ getDatasetVersionByID(ctx, dvid)
+ } else {
+ getLatestDatasetVersion(ctx, did).getOrElse(
+ throw new NotFoundException(ERR_DATASET_VERSION_NOT_FOUND_MESSAGE)
+ )
+ }
+
+ val repositoryName = dataset.getRepositoryName
+ val versionHash = datasetVersion.getVersionHash
+ val objects = withLakeFSErrorHandling(
+ s"listing files of version '$versionHash' of dataset '${dataset.getName}'"
+ ) {
+ LakeFSStorageClient.retrieveObjectsOfVersion(repositoryName, versionHash)
+ }
+
+ if (objects.isEmpty) {
+ throw new NotFoundException(s"No objects found in version $versionHash")
+ }
+
+ val totalBytes = objects.map(_.getSizeBytes.longValue()).sum
+ // TODO: remove this limit once chunked resumable upload to Drive is implemented
+ if (totalBytes > DRIVE_EXPORT_MAX_DATASET_BYTES) {
+ throw new BadRequestException("Dataset version exceeds the 500 MB export limit.")
+ }
+ if (totalBytes > DRIVE_EXPORT_MAX_FILE_BYTES) {
+ throw new ForbiddenException("Dataset version exceeds the 5 TB Google Drive export limit.")
+ }
+
+ // Build the zip in memory
+ val baos = new ByteArrayOutputStream()
+ val zipOut = new java.util.zip.ZipOutputStream(baos)
+ try {
+ objects.foreach { obj =>
+ val filePath = obj.getPath
+ val file = withLakeFSErrorHandling(s"downloading file '$filePath' for Drive export") {
+ LakeFSStorageClient.getFileFromRepo(repositoryName, versionHash, filePath)
+ }
+ zipOut.putNextEntry(new java.util.zip.ZipEntry(filePath))
+ Files.copy(Paths.get(file.toURI), zipOut)
+ zipOut.closeEntry()
+ }
+ } finally {
+ zipOut.close()
+ }
+
+ val zipBytes = baos.toByteArray
+ val sessionUri = request.sessionUri
+
+ // PUT the zip to the Google Drive session URI
+ val conn = new URL(sessionUri).openConnection().asInstanceOf[HttpURLConnection]
+ try {
+ conn.setDoOutput(true)
+ conn.setRequestMethod("PUT")
+ conn.setRequestProperty("Content-Type", "application/zip")
+ conn.setFixedLengthStreamingMode(zipBytes.length)
+ val out = conn.getOutputStream
+ out.write(zipBytes)
+ out.close()
+
+ val code = conn.getResponseCode
+ if (code < 200 || code >= 300) {
+ val body =
+ Option(conn.getErrorStream).map(s => new String(s.readAllBytes())).getOrElse("")
+ if (body.contains("storageQuotaExceeded")) {
+ throw new ForbiddenException("Google Drive storage quota exceeded.")
+ }
+ throw new WebApplicationException(
+ s"Google Drive upload failed with HTTP $code",
+ Response.Status.BAD_GATEWAY
+ )
+ }
+ } finally {
+ conn.disconnect()
+ }
+
+ Response.ok(Map("message" -> "Dataset exported to Google Drive")).build()
+ }
+ }
+
+ @POST
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/drive-export/file")
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ def exportFileToDrive(
+ request: DriveFileExportRequest,
+ @Auth user: SessionUser
+ ): Response = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/")) {
+ throw new BadRequestException("Invalid session URI")
+ }
+ val uid = user.getUid
+ resolveDatasetAndPath(request.filePath, null, null, uid) match {
+ case Left(errorResponse) => errorResponse
+ case Right((repositoryName, commitHash, filePath)) =>
+ val presignedUrl =
+ withLakeFSErrorHandling(s"generating presigned URL for file '$filePath'") {
+ LakeFSStorageClient.getFilePresignedUrl(repositoryName, commitHash, filePath)
+ }
+
+ val minioConn = new URL(presignedUrl).openConnection().asInstanceOf[HttpURLConnection]
+ try {
+ minioConn.setRequestMethod("GET")
+ val contentLength = minioConn.getContentLengthLong
+ if (contentLength > DRIVE_EXPORT_MAX_FILE_BYTES) {
+ throw new ForbiddenException("File exceeds the 5 TB Google Drive export limit.")
+ }
+ val minioStream = minioConn.getInputStream
+
+ val driveConn =
+ new URL(request.sessionUri).openConnection().asInstanceOf[HttpURLConnection]
+ try {
+ driveConn.setDoOutput(true)
+ driveConn.setRequestMethod("PUT")
+ driveConn.setRequestProperty("Content-Type", "application/octet-stream")
+ if (contentLength > 0) {
+ driveConn.setFixedLengthStreamingMode(contentLength)
+ val out = driveConn.getOutputStream
+ minioStream.transferTo(out)
+ out.close()
+ } else {
+ val bytes = minioStream.readAllBytes()
Review Comment:
A question rather than a claim. This branch runs when `getContentLengthLong` returns `-1`. By then the `DRIVE_EXPORT_MAX_FILE_BYTES` check at line 1441 has already passed, since `-1` is not greater than the cap, so `readAllBytes()` pulls the whole object onto the heap unbounded.
Can a MinIO presigned GET omit `Content-Length`? If it can, this path needs its own cap; if not, the branch is dead.
##########
amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala:
##########
@@ -782,6 +787,41 @@ class WorkflowResource extends LazyLogging {
.fetchOneInto(classOf[String])
}
+ @POST
+ @Consumes(Array(MediaType.APPLICATION_JSON))
+ @RolesAllowed(Array("REGULAR", "ADMIN"))
+ @Path("/{wid}/export/drive")
+ def exportWorkflowToDrive(
+ @PathParam("wid") wid: Integer,
+ request: WorkflowResource.DriveExportRequest,
+ @Auth sessionUser: SessionUser
+ ): Unit = {
+ if (!request.sessionUri.startsWith("https://www.googleapis.com/upload/drive/"))
+ throw new BadRequestException("Invalid session URI")
+ if (!WorkflowAccessResource.hasReadAccess(wid, sessionUser.getUid))
+ throw new ForbiddenException("No sufficient access privilege.")
+ val content = workflowDao.fetchOneByWid(wid).getContent.getBytes("UTF-8")
+ if (content.length > WorkflowResource.DRIVE_EXPORT_MAX_BYTES)
+ throw new BadRequestException("Workflow content exceeds the 500 MB export limit.")
+ val conn = new URL(request.sessionUri).openConnection().asInstanceOf[HttpURLConnection]
+ try {
+ conn.setDoOutput(true)
+ conn.setRequestMethod("PUT")
+ conn.setRequestProperty("Content-Type", "application/json")
+ conn.setRequestProperty("Content-Length", content.length.toString)
+ conn.getOutputStream.write(content)
Review Comment:
`HttpURLConnection` writes its own `Content-Length` from the buffered body, so this header never takes effect and the payload is buffered rather than streamed. Both sibling endpoints in this PR use `setFixedLengthStreamingMode` and close the stream (`DatasetResource.scala:1392`/`:1395`, `:1453`/`:1456`); this one does neither.
```suggestion
conn.setFixedLengthStreamingMode(content.length)
val out = conn.getOutputStream
out.write(content)
out.close()
```
##########
common/config/src/main/resources/user-system.conf:
##########
@@ -27,6 +27,12 @@ user-sys {
clientId = ""
clientId = ${?USER_SYS_GOOGLE_CLIENT_ID}
+ clientSecret = ""
+ clientSecret = ${?USER_SYS_GOOGLE_CLIENT_SECRET}
+
+ apiKey = ""
+ apiKey = ${?USER_SYS_GOOGLE_API_KEY}
Review Comment:
`USER_SYS_GOOGLE_API_KEY` needs the same three companion entries its sibling `USER_SYS_GOOGLE_CLIENT_ID` has: `EnvironmentalVariable.scala:122`, `bin/k8s/values.yaml:321` and `bin/k8s/values-development.yaml:347`.
Without the k8s entries the variable is never injected. `googleApiKey` then resolves to `""` and `/auth/google/drive/apikey` serves an empty string. So the Picker breaks in Kubernetes only, while local dev keeps working because you export the variable by hand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Added backend routes to export workflows + datasets to google drive [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6447: feat: Added backend routes to export workflows + datasets to google drive
URL: https://github.com/apache/texera/pull/6447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org