You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/08/16 14:30:54 UTC
[GitHub] [incubator-kyuubi] melin opened a new issue, #3246: [Improvement] InMemoryFIleIndex 缓存filelist信息，怎么解决两个sparksession 长期运行同步问题？
melin opened a new issue, #3246:
URL: https://github.com/apache/incubator-kyuubi/issues/3246
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
例如两个SparkSession A和B，在A中先查询表T1，接着在B中往表T1写入数据，之后在A中查询不到B 写入的数据。目前有两种做法：
1，关闭SparkSession A
2、调用 refresh table command。
这两种做法对于业务用户不知道什么时候去操作，不具可操作性。频繁刷新refresh，影响交互性能。
最理想做法是支持redis 缓存等集中式缓存。
### How should we improve?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3246: [Improvement] InMemoryFIleIndex 缓存filelist信息，怎么解决两个sparksession 长期运行同步问题？
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3246:
URL: https://github.com/apache/incubator-kyuubi/issues/3246#issuecomment-1217372055
It's by design in Hive tables and Spark buildin file format tables, the datalake table formats like delta and iceberg could resolve your problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #3246: [Improvement] InMemoryFIleIndex 缓存filelist信息，怎么解决两个sparksession 长期运行同步问题？
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #3246: [Improvement] InMemoryFIleIndex 缓存filelist信息，怎么解决两个sparksession 长期运行同步问题？
URL: https://github.com/apache/incubator-kyuubi/issues/3246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #3246: [Improvement] InMemoryFIleIndex 缓存filelist信息，怎么解决两个sparksession 长期运行同步问题？
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #3246:
URL: https://github.com/apache/incubator-kyuubi/issues/3246#issuecomment-1217379677
I have a [PR](https://github.com/apache/spark/pull/37355)(maybe not solve your problem yet, but going the right direction with follow-up might achieve that goal).
Sadly, it haven't get much attention from spark community yet, I do see a chance to merge it and do followups
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org