You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/09/02 06:03:10 UTC
[GitHub] [incubator-dolphinscheduler] yy0812 opened a new issue #3643: [Bug][API] OOM occurs when uploading large files
yy0812 opened a new issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643
**Describe the bug**
When uploading large files(353MB) in the resource center, the page progress bar is stuck at 100%, view the background API logs and find OOM
I fixed this after modifying the JVM parameters in the` /bin/dolphinscheduler-daemon.sh` script, but felt it was not a good idea to do so.Do you want to add a configuration item or an FAQ?
**Which version of Dolphin Scheduler:**
-[1.3.2-release]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yy0812 commented on issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
yy0812 commented on issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-686192341
> This actually is not a bug， I think we can Add it to FAQ, if you are interested in putting your solution to FAQ, please leave a message.
I can help to improve FAQ, but I don't know how to submit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yangyichao-mango edited a comment on issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
yangyichao-mango edited a comment on issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-685803145
This actually is not a bug， I think we can Add it to FAQ, if you are interested in putting your solution to FAQ, please leave a message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yangyichao-mango commented on issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
yangyichao-mango commented on issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-686488879
I think If we can implement it with streaming mode, it will be better. You can try it as baoqi way.
------------------ Original ------------------
From: Baoqi Wu <notifications@github.com>
Date: Thu,Sep 3,2020 5:19 PM
To: apache/incubator-dolphinscheduler <incubator-dolphinscheduler@noreply.github.com>
Cc: Yichao Yang <1048262223@qq.com>, Comment <comment@noreply.github.com>
Subject: Re: [apache/incubator-dolphinscheduler] [Bug][API] OOM occurs when uploading large files (#3643)
I think we may be able to do better for this. for multipart files in springboot & jetty, it should use streaming mode to not directly load the whole file in memory.
But according to spring boot's documentation: https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/web/servlet/MultipartProperties.html
`file-size-threshold specifies the size threshold after which files will be written to disk. The default is 0.`
The default is 0, so, it will parse uploaded multipart file into memory, instead of writing to temp file.
So, I guess set the spring.http.multipart.file-size-threshold to a value like 10MB. may avoid the OutOfMemory error
spring.http.multipart.file-size-threshold=10MB
Can you try whether this will reduce memory usage?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] xingchun-chen closed issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
xingchun-chen closed issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] Baoqi commented on issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
Baoqi commented on issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-686364687
I think we may be able to do better for this. for multipart files in springboot & jetty, it should use streaming mode to not directly load the whole file in memory.
But according to spring boot's documentation: https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/web/servlet/MultipartProperties.html
`file-size-threshold specifies the size threshold after which files will be written to disk. The default is 0.`
The default is 0, so, it will parse uploaded multipart file into memory, instead of writing to temp file.
So, I guess set the `spring.http.multipart.file-size-threshold` to a value like 10MB. may avoid the OutOfMemory error
spring.http.multipart.file-size-threshold=10MB
Can you try whether this will reduce memory usage?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yangyichao-mango commented on issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
yangyichao-mango commented on issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-685803145
This actually is not a bug， If you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs commented on issue #3643: [Bug][API] OOM occurs when uploading large files
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #3643:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-685809800
> This actually is not a bug， I think we can Add it to FAQ, if you are interested in putting your solution to FAQ, please leave a message.
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org