You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by GitBox <gi...@apache.org> on 2020/07/19 20:53:07 UTC

[GitHub] [incubator-heron] Code0x58 opened a new pull request #3574: Decode string when deserializing Jaba object in python

Code0x58 opened a new pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574


   Re. https://heronstreaming.slack.com/archives/C0VKQLKV0/p1595162763201800
   
   This also uses the longer DataWatcher handler signature to avoid a
   TypeError which it relies on to pass retry with the shorter signature.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 edited a comment on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
thinker0 edited a comment on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-662158049


   +1 
   really good work.
   Approved


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661021922


   @Code0x58 another  
   ```
   [2020-07-20 21:33:24 +0900] [INFO]: Setting topology info for topology: heron-ladp_ladm_perf_adgroup_breakdown_report-kafka-to-ladp-log-es-eco
   [2020-07-20 21:33:24 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/tracker.py", line 77, in _convert_java_value
       'raw' : utils.hex_escape(kv.serialized_value)}
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/utils.py", line 44, in hex_escape
       return ''.join(ch if ch in printable else r'0x{0:02x}'.format(ord(ch)) for ch in bin_str)
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/utils.py", line 44, in <genexpr>
       return ''.join(ch if ch in printable else r'0x{0:02x}'.format(ord(ch)) for ch in bin_str)
   TypeError: 'in <string>' requires string as left operand, not int
   [2020-07-20 21:33:24 +0900] [ERROR]: Caught exception while triggering callback: 'in <string>' requires string as left operand, not int
   [2020-07-20 21:33:24 +0900] [INFO]: Unregister a watch with uid: e56c8661-0da9-4793-a2a8-60d733e74528
   [2020-07-20 21:33:24 +0900] [INFO]: Adding data watch for path: /heron/tmasters/heron-ladp_ladm_perf_adgroup_breakdown_report-kafka-to-ladp-log-es-eco
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] joshfischer1108 commented on a change in pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
joshfischer1108 commented on a change in pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457001709



##########
File path: heron/statemgrs/src/python/zkstatemanager.py
##########
@@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching):
 
     # pylint: disable=unused-variable, unused-argument
     @self.client.DataWatch(path)
-    def watch_topology(data, stats):
+    def watch_topology(data, stats, event):
       """ watch topology """
       if data:
         topology = Topology()

Review comment:
       I think I see the problem (mismatch number of passed arguments) you are trying to address.  
   
   ```
   "/Users/someUser/.pex/installed_wheels/7b3448a0635a4c114f98f11df06af06d250c297a/kazoo-2.7.0-py2.py3-none-any.whl/kazoo/recipe/watchers.py", line 162, in _log_func_exception
       result = self._func(data, stat, event)
   TypeError: watch_execution_state() takes 2 positional arguments but 3 were given
   ```
   I'm not sure I follow the reasoning of adding `event` to each of the method signatures.  If it is never used, why do we need to pass it at all?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 edited a comment on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
thinker0 edited a comment on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661433596


   @Code0x58 sorry. The last one goes like this.
   
   Other features seem to work fine.
   
   ```
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaMap' object has no attribute 'is_primitive'
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaMap' object has no attribute 'is_primitive'
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaObject' object has no attribute 'is_primitive'
   
   
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaBool' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaBool' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaBool' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] joshfischer1108 commented on a change in pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
joshfischer1108 commented on a change in pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457585129



##########
File path: heron/statemgrs/src/python/zkstatemanager.py
##########
@@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching):
 
     # pylint: disable=unused-variable, unused-argument
     @self.client.DataWatch(path)
-    def watch_topology(data, stats):
+    def watch_topology(data, stats, event):
       """ watch topology """
       if data:
         topology = Topology()

Review comment:
       Ok.  Thanks for clarifying. Makes sense to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661468879


   Thanks again @thinker0! So much for hopes of a dumb upgrade. I'll have to spend some time getting to understand the old and new `javaobj` modules to do a proper migration for `convert_pb_kvs(...)`, which I'll aim to get on tomorrow evening


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661031266


   thanks again @thinker0, I pushed another amendment. The amendment isn't 1-to-1 with the python2 implementation, but it's probably saner. I don't know if it will have a knock on effect down the line 🤞 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-662133945


   an initial look into `convert_pb_kvs(...)` makes it me think it's a pretty hairy thing, so I didn't try to do anything smart and pushed what might get things running. I'll keep having a look through the file to see if I can work out how to clean it up, but I'm not optimistic


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661046491


   @Code0x58 Another
   ```
   [2020-07-20 22:40:03 +0900] [INFO]: Setting topology info for topology: heron-ladp_oa_message_id_report-to-ladp-log-es
   [2020-07-20 22:40:03 +0900] [ERROR]: ==Oops state dump=============
   [2020-07-20 22:40:03 +0900] [ERROR]: References: [[com.typesafe.config.impl.SerializedConfigValue:0x0], <javaobj:com.typesafe.config.impl.SerializedConfigValue>]
   [2020-07-20 22:40:03 +0900] [ERROR]: Stream seeking back at -16 byte (2nd line is an actual position!):
   [2020-07-20 22:40:03 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 221, in readObject
       _, res = self._read_and_exec_opcode(ident=0)
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 256, in _read_and_exec_opcode
       return (opid, handler(ident=ident))
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 420, in do_object
       opcode, obj = self._read_and_exec_opcode(ident=ident+1)
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 255, in _read_and_exec_opcode
       raise RuntimeError("Unknown OpCode in the stream: 0x%x" % opid)
   RuntimeError: Unknown OpCode in the stream: 0x7a
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/tracker.py", line 62, in _convert_java_value
       pobj = javaobj.loads(kv.serialized_value)
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 65, in loads
       return marshaller.readObject()
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 235, in readObject
       self._oops_dump_state()
     File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 545, in _oops_dump_state
       self.object_stream.seek(-16, mode=1)
   TypeError: seek() takes no keyword arguments
   [2020-07-20 22:40:03 +0900] [INFO]: Adding new topology: heron-system-access-es-a-line-me, state_manager: lad
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661050567


   humm, that error looks like it could be a deeper `pyobj` issue, so I should probably stop trying to patch it, and just upgrade `pyobj` for https://github.com/apache/incubator-heron/issues/3575 - I'll be able to give it more attention tonight


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661324466


   CI passed on my machine, so hopefully the switch to the maintained version of `pyobj` sorts things out!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on a change in pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071



##########
File path: heron/statemgrs/src/python/zkstatemanager.py
##########
@@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching):
 
     # pylint: disable=unused-variable, unused-argument
     @self.client.DataWatch(path)
-    def watch_topology(data, stats):
+    def watch_topology(data, stats, event):
       """ watch topology """
       if data:
         topology = Topology()

Review comment:
       I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L155) which might be having some weird interaction with the rest of the codebase - at least we certainly won't see the TypeError now even if the parameter is junk




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661002357


   @Code0x58 
   
   ```
   TypeError: 'float' object cannot be interpreted as an integer
   ```
   ```
   [2020-07-20 21:33:24 +0900] [INFO]: Setting topology info for topology: heron-lass_win_loss_notice_log-to-ladp-log-es
   [2020-07-20 21:33:24 +0900] [ERROR]: ==Oops state dump=============
   [2020-07-20 21:33:24 +0900] [ERROR]: References: [[java.util.HashMap:0x507DAC1], <javaobj:java.util.HashMap>, 'NJ_INSTANCE_PHASE', 'BETA', 'LD_LIBRARY_PATH', '${LD_LIBRARY_PATH}:./lib64:/usr/lib/hadoop/lib/native', 'SENTRY_DSN', 'https://55d2bf90467e4814908d94c10d154229@lad-sentry.line-apps.com/3?sample.rate=0.10', 'SERVER_ENV', 'beta', 'NJ_INSTANCE_ID', 'heron-lass_win_loss_notice_log-to-ladp-log-es', 'SENTRY_ENVIRONMENT']
   [2020-07-20 21:33:24 +0900] [ERROR]: Stream seeking back at -16 byte (2nd line is an actual position!):
   [2020-07-20 21:33:24 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 221, in readObject
       _, res = self._read_and_exec_opcode(ident=0)
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 256, in _read_and_exec_opcode
       return (opid, handler(ident=ident))
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 428, in do_object
       tmp_object = transformer.transform(java_object)
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 625, in transform
       for i in range((len(obj.annotations)-1)/2):
   TypeError: 'float' object cannot be interpreted as an integer
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/tracker.py", line 62, in _convert_java_value
       pobj = javaobj.loads(kv.serialized_value)
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 65, in loads
       return marshaller.readObject()
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 235, in readObject
       self._oops_dump_state()
     File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 545, in _oops_dump_state
       self.object_stream.seek(-16, mode=1)
   TypeError: seek() takes no keyword arguments
   [2020-07-20 21:33:24 +0900] [ERROR]: Caught exception while triggering callback: 'in <string>' requires string as left operand, not int
   [2020-07-20 21:33:24 +0900] [INFO]: Unregister a watch with uid: 0ce59fcd-9c28-473e-94d7-f09aebc29085
   [2020-07-20 21:33:24 +0900] [INFO]: Adding data watch for path: /heron/tmasters/heron-lass_win_loss_notice_log-to-ladp-log-es
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 edited a comment on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 edited a comment on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661013477


   thanks @thinker0, I just pushed a quick amendment that should resolve the `/`→`//` issue. There is still be the separate seek issue which I think is because the underlying (C) stream type doesn't implement seek, but that hopefully only shows itself when when things blow up.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] nicknezis merged pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
nicknezis merged pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on a change in pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071



##########
File path: heron/statemgrs/src/python/zkstatemanager.py
##########
@@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching):
 
     # pylint: disable=unused-variable, unused-argument
     @self.client.DataWatch(path)
-    def watch_topology(data, stats):
+    def watch_topology(data, stats, event):
       """ watch topology """
       if data:
         topology = Topology()

Review comment:
       I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L161) which might be having some weird interaction with the rest of the codebase - at least we certainly won't see the TypeError now even if the parameter is junk




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661013477


   thanks @thinker0, I just pushed a quick amendment that should resolve that issue (`/`→`//`)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-662158049


   +1 
   
   Approved


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 edited a comment on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
thinker0 edited a comment on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661433596


   @Code0x58 sorry. The last one goes like this.
   ```
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaMap' object has no attribute 'is_primitive'
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaMap' object has no attribute 'is_primitive'
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaObject' object has no attribute 'is_primitive'
   
   
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaBool' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaBool' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaBool' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaInt' object has no attribute 'is_primitive'
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on a change in pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071



##########
File path: heron/statemgrs/src/python/zkstatemanager.py
##########
@@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching):
 
     # pylint: disable=unused-variable, unused-argument
     @self.client.DataWatch(path)
-    def watch_topology(data, stats):
+    def watch_topology(data, stats, event):
       """ watch topology """
       if data:
         topology = Topology()

Review comment:
       I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L161) which can obscure the exceptions when they occur within the handler - at least we certainly won't see the TypeError now even if the parameter is junk (no more "exception occurred while handling exception", so that much easier to read)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 commented on a change in pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071



##########
File path: heron/statemgrs/src/python/zkstatemanager.py
##########
@@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching):
 
     # pylint: disable=unused-variable, unused-argument
     @self.client.DataWatch(path)
-    def watch_topology(data, stats):
+    def watch_topology(data, stats, event):
       """ watch topology """
       if data:
         topology = Topology()

Review comment:
       I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L161) which can obscure the exceptions when they occur within the handler - at least we certainly won't see the TypeError now even if the parameter is junk




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] Code0x58 edited a comment on pull request #3574: Decode string when deserializing Java object in python

Posted by GitBox <gi...@apache.org>.
Code0x58 edited a comment on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661013477


   thanks @thinker0, I just pushed a quick amendment that should resolve the `/`→`//` issue, but there'd still be the separate seek issue which I think is because the underlying (C) stream type doesn't implement seek


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.

Posted by GitBox <gi...@apache.org>.
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661433596


   @Code0x58 sorry. The last one goes like this.
   ```
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaMap' object has no attribute 'is_primitive'
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaMap' object has no attribute 'is_primitive'
   [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
   Traceback (most recent call last):
     File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
       if pobj.is_primitive():
   AttributeError: 'JavaObject' object has no attribute 'is_primitive'
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org