You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "promer94 (via GitHub)" <gi...@apache.org> on 2023/03/29 17:35:53 UTC
[GitHub] [incubator-opendal] promer94 opened a new pull request, #1798: chore(bindings/nodejs): update presign example and init benchmark
promer94 opened a new pull request, #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798
* slightly improve the presign example
* init a simple benchmark using aws-sdk-js-v3 and benny
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152828118
##########
bindings/nodejs/generated.js:
##########
@@ -36,7 +36,7 @@ function isMusl() {
// For Node 10
if (!process.report || typeof process.report.getReport !== 'function') {
try {
- const lddPath = require('child_process').execSync('which ldd').toString().trim();
+ const lddPath = require('child_process').execSync('which ldd').toString().trim()
Review Comment:
This file should not be touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152818329
##########
bindings/nodejs/package.json:
##########
Review Comment:
> In fact, this is not a troublesome matter.
Just no needs :rofl:, We can spend our time towards more valuable tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] promer94 commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "promer94 (via GitHub)" <gi...@apache.org>.
promer94 commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152843808
##########
bindings/nodejs/generated.js:
##########
@@ -36,7 +36,7 @@ function isMusl() {
// For Node 10
if (!process.report || typeof process.report.getReport !== 'function') {
try {
- const lddPath = require('child_process').execSync('which ldd').toString().trim();
+ const lddPath = require('child_process').execSync('which ldd').toString().trim()
Review Comment:
I didn't intent to modify this file. But everytime i run `yarn build`, This file's license header will be remove and the content will be changed. So i have manually add it back. 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): update presign example and init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152613958
##########
bindings/nodejs/examples/presign.js:
##########
Review Comment:
This example is too complex to me. I prefer to only have an example for `presign`. Maybe we can remove this file, and add example for `presign` function. Like we does for
https://github.com/apache/incubator-opendal/blob/3ac7f07fd4f85b757be5552270fec2af7cf900cc/core/src/types/operator/operator.rs#L919-L944
##########
bindings/nodejs/benchmark/index.js:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+const { suite, add, cycle, complete } = require('benny')
+const { Operator } = require('../index')
+const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3')
+const { Upload } = require('@aws-sdk/lib-storage')
+const crypto = require('crypto')
+
+const opendal = new Operator('s3', {
+ root: '/',
+ bucket: 'benchmark',
+ endpoint: process.env.AWS_S3_ENDPOINT,
+})
+
+const client = new S3Client({
+ endpoint: process.env.AWS_S3_ENDPOINT,
+ region: process.env.AWS_S3_REGION,
+ credentials: {
+ accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+ secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
+ },
+ bucket: 'benchmark',
+})
+
+const s3 = {
+ write: (path, data) => {
+ const command = new PutObjectCommand({
+ Body: data,
+ Key: path,
+ Bucket: 'benchmark',
+ })
+ return client.send(command)
+ },
+ parallelUpload: (path, data) => {
Review Comment:
I would rather remove this case to maintain a simpler benchmark. Starting from a simple baseline is more beneficial for us. Additionally, I am particularly interested in the `read` performance as opposed to the `write` performance.
##########
bindings/nodejs/package.json:
##########
@@ -43,11 +43,16 @@
"NOTICE"
],
"devDependencies": {
+ "@aws-sdk/abort-controller": "^3.296.0",
+ "@aws-sdk/client-s3": "^3.301.0",
+ "@aws-sdk/lib-storage": "^3.301.0",
+ "@aws-sdk/s3-request-presigner": "^3.301.0",
Review Comment:
**question**: Seems we didn't use this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo merged pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] suyanhanx commented on a diff in pull request #1798: chore(bindings/nodejs): update presign example and init benchmark
Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152610967
##########
bindings/nodejs/generated.js:
##########
Review Comment:
We don't need to update this file. It is automatically generated.
##########
bindings/nodejs/package.json:
##########
Review Comment:
Can we move these dependencies into the benchmark folder?
##########
bindings/nodejs/examples/presign.js:
##########
@@ -44,6 +49,22 @@ const server = http.createServer(async (req, res) => {
}
})
-server.listen(3000, () => {
+server.listen(3000, async () => {
console.log('Server is listening on port 3000.')
+
+ /* Get presigned upload url */
+ const presignRequest = await fetch('http://127.0.0.1:3000/presign/upload?path=opendal.txt&expires=3600').then((v) =>
+ v.json(),
+ )
+ /* Upload file use presign data */
+ await fetch(presignRequest.uri, {
Review Comment:
```suggestion
await fetch(presignRequest.url, {
```
Same above.
##########
bindings/nodejs/examples/presign.js:
##########
@@ -18,24 +18,29 @@
*/
const http = require('node:http')
-const url = require('node:url')
const { Operator } = require('../index')
const op = new Operator('s3', {
root: '/',
- bucket: 'example-bucket',
+ bucket: 'presign'
})
const server = http.createServer(async (req, res) => {
res.setHeader('Content-Type', 'text/json; charset=utf-8')
-
- if (req.url.startsWith('/presign') && req.method === 'GET') {
- const urlParts = url.parse(req.url, true)
- const path = urlParts.query.path
- const expires = urlParts.query.expires
-
- const presignedRequest = op.presignRead(path, parseInt(expires))
-
+ if (req.url.startsWith('/presign/download')) {
+ const url = new URL(req.url, 'http://locahost:3000')
+ const path = url.searchParams.get('path')
+ const expires = parseInt(url.searchParams.get('expires'), 10)
+ const presignedRequest = op.presignRead(path, expires)
+ /* Redirect client to download file from presigned url */
+ res.statusCode = 307
+ res.setHeader('Location', presignedRequest.uri)
Review Comment:
```suggestion
res.setHeader('Location', presignedRequest.url)
```
Sorry for updating the API. Please check the latest file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152805890
##########
bindings/nodejs/benchmark/constant.js:
##########
Review Comment:
The benchmark project's file structure is too cumbersome. We prefer not to have directories with many files that are 50 lines long. Could you please consolidate them and keep the structure simple?
Simple is the best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] suyanhanx commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152844946
##########
bindings/nodejs/generated.js:
##########
@@ -36,7 +36,7 @@ function isMusl() {
// For Node 10
if (!process.report || typeof process.report.getReport !== 'function') {
try {
- const lddPath = require('child_process').execSync('which ldd').toString().trim();
+ const lddPath = require('child_process').execSync('which ldd').toString().trim()
Review Comment:
I'll handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] suyanhanx commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152816004
##########
bindings/nodejs/package.json:
##########
Review Comment:
> We can split them in the future. Not a big issue.
🤣 In fact, this is not a troublesome matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] suyanhanx commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152813603
##########
bindings/nodejs/package.json:
##########
Review Comment:
> I guess we don't need to, let's keep it simple to get started.
My point is, only install them when need to run benchmarks. These deps are not necessary during development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152844923
##########
bindings/nodejs/generated.js:
##########
@@ -36,7 +36,7 @@ function isMusl() {
// For Node 10
if (!process.report || typeof process.report.getReport !== 'function') {
try {
- const lddPath = require('child_process').execSync('which ldd').toString().trim();
+ const lddPath = require('child_process').execSync('which ldd').toString().trim()
Review Comment:
Oh, it's our fault. We will fix it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on pull request #1798: chore(bindings/nodejs): update presign example and init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#issuecomment-1489678435
The example file has been removed from https://github.com/apache/incubator-opendal/pull/1799. As a result, this PR can now focus on benchmarking :rocket:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152806315
##########
bindings/nodejs/package.json:
##########
Review Comment:
I guess we don't need to, let's keep it simple to get started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152806315
##########
bindings/nodejs/package.json:
##########
Review Comment:
I guess we don't need to, let's keep simple to get started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] promer94 commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "promer94 (via GitHub)" <gi...@apache.org>.
promer94 commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152843808
##########
bindings/nodejs/generated.js:
##########
@@ -36,7 +36,7 @@ function isMusl() {
// For Node 10
if (!process.report || typeof process.report.getReport !== 'function') {
try {
- const lddPath = require('child_process').execSync('which ldd').toString().trim();
+ const lddPath = require('child_process').execSync('which ldd').toString().trim()
Review Comment:
I didn't intent to modify this file. But everytime i run `yarn build`, This file's license header will be remove and the content will be changed. So i have to manually add it back. 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152805890
##########
bindings/nodejs/benchmark/constant.js:
##########
Review Comment:
The benchmark project's file structure is too cumbersome. We prefer not to have directories with many files that are 50 lines long. Could you please consolidate them and keep the structure simple?
Simple is the best. We can do changes while required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1798: chore(bindings/nodejs): init benchmark
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1798:
URL: https://github.com/apache/incubator-opendal/pull/1798#discussion_r1152814470
##########
bindings/nodejs/package.json:
##########
Review Comment:
> My point is, only install them when need to run benchmarks. These deps are not necessary during development.
We can split them in the future. Not a big issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org