You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@livy.apache.org by GitBox <gi...@apache.org> on 2020/04/24 23:33:15 UTC
[GitHub] [incubator-livy] mzhang-code opened a new pull request #292: [LIVY-764] Give code block a filename and cache it in linecache
mzhang-code opened a new pull request #292:
URL: https://github.com/apache/incubator-livy/pull/292
[LIVY-764] Give code block a filename and cache it in linecache
## What changes were proposed in this pull request?
Today python shell (`fake_shell.py`) uses 'stdin' as file name to compile code blocks and never caches them in linecache. This breaks the user code that depends on `inspect.findsource`, like `torch.jit.script`
To enable code block caching at the backend, we can use the way how ipython does: https://github.com/ipython/ipython/blob/44bd47d5ac27805a297798d39ad7e31f63181468/IPython/core/compilerop.py#L54
## How was this patch tested?
E2E test with local livy server
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] mzhang-code commented on pull request #292: [LIVY-764] Give code block a filename and cache it in linecache
Posted by GitBox <gi...@apache.org>.
mzhang-code commented on pull request #292:
URL: https://github.com/apache/incubator-livy/pull/292#issuecomment-619282542
This change enables `inspect.findsource` and `torch.jit.script`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] mzhang-code commented on a change in pull request #292: [LIVY-764] Give code block a filename and cache it in linecache
Posted by GitBox <gi...@apache.org>.
mzhang-code commented on a change in pull request #292:
URL: https://github.com/apache/incubator-livy/pull/292#discussion_r416047727
##########
File path: repl/src/main/resources/fake_shell.py
##########
@@ -212,20 +215,35 @@ def __init__(self, exc_info):
class NormalNode(object):
def __init__(self, code):
- self.code = compile(code, '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1)
+ self.cell_name = self.code_name(code)
+ linecache_entry = (len(code), time.time(), [line+'\n' for line in code.splitlines()], self.cell_name)
+ linecache.cache[self.cell_name] = linecache_entry
+ linecache._livy_cache[self.cell_name] = linecache_entry
+
+ self.code = compile(code, self.cell_name, 'exec', ast.PyCF_ONLY_AST, 1)
+
+ def code_name(self, code, number=0):
Review comment:
Makes sense. Will fix it together with unit test fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] vkuzmin-uber commented on a change in pull request #292: [LIVY-764] Give code block a filename and cache it in linecache
Posted by GitBox <gi...@apache.org>.
vkuzmin-uber commented on a change in pull request #292:
URL: https://github.com/apache/incubator-livy/pull/292#discussion_r414981126
##########
File path: repl/src/main/resources/fake_shell.py
##########
@@ -212,20 +215,35 @@ def __init__(self, exc_info):
class NormalNode(object):
def __init__(self, code):
- self.code = compile(code, '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1)
+ self.cell_name = self.code_name(code)
+ linecache_entry = (len(code), time.time(), [line+'\n' for line in code.splitlines()], self.cell_name)
+ linecache.cache[self.cell_name] = linecache_entry
+ linecache._livy_cache[self.cell_name] = linecache_entry
+
+ self.code = compile(code, self.cell_name, 'exec', ast.PyCF_ONLY_AST, 1)
+
+ def code_name(self, code, number=0):
Review comment:
nit: It seems like this doesn't need self and util method that should be static and maybe private ?
```
@staticmethod
def _code_name(code, number=0):
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org