You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/09/06 02:53:54 UTC
[GitHub] [doris] Henry2SS opened a new issue, #12368: [Enhancement] broker load should not close file system when exception happens
Henry2SS opened a new issue, #12368:
URL: https://github.com/apache/doris/issues/12368
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
version: `trunk`
suppose there are two threads named `t1` and `t2` working at the same time.
`t1` init the `brokerFileSystem`, and `t2` gets the cache.
during `t1` working period, `t2` gets exception, and closes the file system.
so that `t1` cannot work normally.
### Solution
should not close file system when exception happens. Just let it be expired.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Henry2SS commented on issue #12368: [Enhancement] broker load should not close file system when exception happens
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on issue #12368:
URL: https://github.com/apache/doris/issues/12368#issuecomment-1237756135
> > In some senerio, `t1` is still using the file system to reading files. It means that the file system is not broken. If `t2` close the file system, it will leads `t1` to abort.
>
> When an exception occurs, it is likely that there is a problem with the broker. If it is not closed at this time, when will it be closed? If it is not closed, the import will always fail. A small probability of mistaken killing should be allowed.
`File System` itself will not be broken.
```
An abstract base class for a fairly generic filesystem. It may be implemented as a distributed filesystem, or as a "local" one that reflects the locally-connected disk.
```
And there is a cache, that stores the file system object. It has expiration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] whutpencil commented on issue #12368: [Enhancement] broker load should not close file system when exception happens
Posted by GitBox <gi...@apache.org>.
whutpencil commented on issue #12368:
URL: https://github.com/apache/doris/issues/12368#issuecomment-1237722982
> In some senerio, `t1` is still using the file system to reading files. It means that the file system is not broken. If `t2` close the file system, it will leads `t1` to abort.
When an exception occurs, it is likely that there is a problem with the broker. If it is not closed at this time, when will it be closed? If it is not closed, the import will always fail. A small probability of mistaken killing should be allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] whutpencil commented on issue #12368: [Enhancement] broker load should not close file system when exception happens
Posted by GitBox <gi...@apache.org>.
whutpencil commented on issue #12368:
URL: https://github.com/apache/doris/issues/12368#issuecomment-1237649120
There is no problem with this logic. If the filesystem cannot be used, it will still report an error next time if it is not closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] whutpencil commented on issue #12368: [Enhancement] broker load should not close file system when exception happens
Posted by GitBox <gi...@apache.org>.
whutpencil commented on issue #12368:
URL: https://github.com/apache/doris/issues/12368#issuecomment-1237723295
> In some senerio, `t1` is still using the file system to reading files. It means that the file system is not broken. If `t2` close the file system, it will leads `t1` to abort.
When an exception occurs, it is likely that there is a problem with the broker. If it is not closed at this time, when will it be closed? If it is not closed, the import will always fail. A small probability of mistaken killing should be allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Henry2SS commented on issue #12368: [Enhancement] broker load should not close file system when exception happens
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on issue #12368:
URL: https://github.com/apache/doris/issues/12368#issuecomment-1237718977
> There is no problem with this logic. If the filesystem cannot be used, it will still report an error next time if it is not closed.
In some senerio, `t1` is still using the file system to reading files. It means that the file system is not broken. If `t2` close the file system, it will leads `t1` to abort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Henry2SS commented on issue #12368: [Enhancement] broker load should not close file system when exception happens
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on issue #12368:
URL: https://github.com/apache/doris/issues/12368#issuecomment-1237802878
> > In some senerio, `t1` is still using the file system to reading files. It means that the file system is not broken. If `t2` close the file system, it will leads `t1` to abort.
>
> When an exception occurs, it is likely that there is a problem with the broker. If it is not closed at this time, when will it be closed? If it is not closed, the import will always fail. A small probability of mistaken killing should be allowed.
It throws an IOException. That seems like file system is not broken. Most of the exceptions are because of abnormal status of hdfs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] broker load should not close file system when exception happens [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed issue #12368: [Enhancement] broker load should not close file system when exception happens
URL: https://github.com/apache/doris/issues/12368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org