You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by GitBox <gi...@apache.org> on 2021/07/09 14:25:29 UTC
[GitHub] [avro] subhashb opened a new pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
subhashb opened a new pull request #1287:
URL: https://github.com/apache/avro/pull/1287
The exception message now includes the field name on which the type exception
was raised.
Closes: AVRO-2720
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Avro Jira](https://issues.apache.org/jira/browse/AVRO/) issues and references them in the PR title.
- https://issues.apache.org/jira/browse/AVRO-2720
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](https://chris.beams.io/posts/git-commit/)":
1. Subject is separated from the body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes how to use it.
NA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] subhashb commented on pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
subhashb commented on pull request #1287:
URL: https://github.com/apache/avro/pull/1287#issuecomment-878708134
@kojiromike Thank you for that. After I incorporated your changes, the problem seems to have got resolved. The 3.6 test seems to have gone into a black hole this time, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
kojiromike commented on pull request #1287:
URL: https://github.com/apache/avro/pull/1287#issuecomment-877625028
Yeah, it could be something about GitHub actions as much as any change to avro. So sure, raise the threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] subhashb commented on pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
subhashb commented on pull request #1287:
URL: https://github.com/apache/avro/pull/1287#issuecomment-877232543
@kojiromike This branch should rebase fine with `master`.
The same timed test `test_minimum_speed` has failed in this PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
kojiromike commented on pull request #1287:
URL: https://github.com/apache/avro/pull/1287#issuecomment-877865387
There's a resource leak in one of the other tests, which creates an http server or something in a thread and then doesn't always close it properly. When that happens, Python warns of a `ResourceWarning`. That warning is sometimes being caught in `catch_warnings` instead of the `IgnoredLogicalType` warning we expecting.
In #708, we introduced [warnings](https://docs.python.org/3/library/warnings.html) so that logical types could be handled better. Testing that change required making sure the right warnings happened. But the [docs say `catch_warnings` is not thread-safe](https://docs.python.org/3/library/warnings.html#warnings.catch_warnings) and we didn't account for that.
What we can do is actually change the global warnings filter. This is also not thread-safe, but it does let us filter only explicit warning types.
```
diff --git a/lang/py/avro/test/test_schema.py b/lang/py/avro/test/test_schema.py
index 1a8e96a50..450674430 100644
--- a/lang/py/avro/test/test_schema.py
+++ b/lang/py/avro/test/test_schema.py
@@ -713,19 +713,21 @@ class SchemaParseTestCase(unittest.TestCase):
# Never hide repeated warnings when running this test case.
warnings.simplefilter("always")
- def parse_valid(self):
+ def parse_valid(self) -> None:
"""Parsing a valid schema should not error, but may contain warnings."""
- with warnings.catch_warnings(record=True) as actual_warnings:
- try:
- self.test_schema.parse()
- except (avro.errors.AvroException, avro.errors.SchemaParseException): # pragma: no coverage
- self.fail(f"Valid schema failed to parse: {self.test_schema!s}")
- actual_messages = [str(wmsg.message) for wmsg in actual_warnings]
- if self.test_schema.warnings:
- expected_messages = [str(w) for w in self.test_schema.warnings]
- self.assertEqual(actual_messages, expected_messages)
- else:
- self.assertEqual(actual_messages, [])
+ test_warnings = self.test_schema.warnings or []
+ try:
+ warnings.filterwarnings(action="error", category=avro.errors.IgnoredLogicalType)
+ self.test_schema.parse()
+ except (avro.errors.IgnoredLogicalType) as e:
+ self.assertIn(type(e), (type(w) for w in test_warnings))
+ self.assertIn(str(e), (str(w) for w in test_warnings))
+ except (avro.errors.AvroException, avro.errors.SchemaParseException): # pragma: no coverage
+ self.fail(f"Valid schema failed to parse: {self.test_schema!s}")
+ else:
+ self.assertEqual([], test_warnings)
+ finally:
+ warnings.filterwarnings(action="default", category=avro.errors.IgnoredLogicalType)
def parse_invalid(self):
"""Parsing an invalid schema should error."""
```
With this change, the `ResourceWarning` would just get messaged, and not raised as an error, but the IgnoredLogicalType would get raised and then caught, and we can test for it.
…at least, I believe this will fix the test issue, but I can't reproduce it locally, so want to give it a try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike merged pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
kojiromike merged pull request #1287:
URL: https://github.com/apache/avro/pull/1287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] subhashb commented on a change in pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
subhashb commented on a change in pull request #1287:
URL: https://github.com/apache/avro/pull/1287#discussion_r667403220
##########
File path: lang/py/avro/test/test_bench.py
##########
@@ -22,6 +22,7 @@
import platform
import random
import string
+import sys
Review comment:
Done. But the saga continues with a schema parser error 😄 , which seems to be transient as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on a change in pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
kojiromike commented on a change in pull request #1287:
URL: https://github.com/apache/avro/pull/1287#discussion_r667372953
##########
File path: lang/py/avro/test/test_bench.py
##########
@@ -22,6 +22,7 @@
import platform
import random
import string
+import sys
Review comment:
This import can be removed. (I'm never sure if everyone can see the code scanner notes, or if this comment is redundant.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on a change in pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
kojiromike commented on a change in pull request #1287:
URL: https://github.com/apache/avro/pull/1287#discussion_r668369912
##########
File path: lang/py/avro/test/test_io.py
##########
@@ -552,7 +552,24 @@ def test_type_exception(self) -> None:
)
)
datum_to_write = {"E": 5, "F": "Bad"}
- self.assertRaises(avro.errors.AvroTypeException, write_datum, datum_to_write, writers_schema)
+ with self.assertRaises(avro.errors.AvroTypeException) as exc:
+ write_datum(datum_to_write, writers_schema)
+ assert str(exc.exception) == 'The datum "Bad" provided for "F" is not an example of the schema "int"'
+
+ def test_type_exception_long(self) -> None:
+ writers_schema = avro.schema.parse(json.dumps({"type": "record", "name": "Test", "fields": [{"name": "foo", "type": "long"}]}))
+ datum_to_write = {"foo": 5.0}
+
+ with self.assertRaises(avro.errors.AvroTypeException) as exc:
+ write_datum(datum_to_write, writers_schema)
+ assert str(exc.exception) == 'The datum "5.0" provided for "foo" is not an example of the schema "long"'
+
+ def test_type_exception_record(self) -> None:
+ writers_schema = avro.schema.parse(json.dumps({"type": "record", "name": "Test", "fields": [{"name": "foo", "type": "long"}]}))
+ datum_to_write = ("foo", 5.0)
+
+ with self.assertRaisesRegex(avro.errors.AvroTypeException, r"The datum \".*\" provided for \".*\" is not an example of the schema [\s\S]*"):
Review comment:
This could be written `r'The datum ".*" provided for ".*" is not an example of the schema [\s\S]*'`
Using single quotes avoids the backslashes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
kojiromike commented on pull request #1287:
URL: https://github.com/apache/avro/pull/1287#issuecomment-877239164
@subhashb if you want, it's OK with me to bump the timing of these tests for python 3.6, specifically. We still should support 3.6, but it's almost at EOL and we can give it a little more performance leeway.
https://github.com/apache/avro/blob/master/lang/py/avro/test/test_bench.py#L54
```
if sys.version_info <= (3, 6): …
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] subhashb commented on pull request #1287: AVRO-2720: Enhance AvroTypeException message to include field name
Posted by GitBox <gi...@apache.org>.
subhashb commented on pull request #1287:
URL: https://github.com/apache/avro/pull/1287#issuecomment-877572235
> @subhashb if you want, it's OK with me to bump the timing of these tests for python 3.6, specifically. We still should support 3.6, but it's almost at EOL and we can give it a little more performance leeway.
@kojiromike
Looks like Pypy 3.7 has the same issue. Do you think we should raise the bar to 5 secs and fix this permanently later? I am unsure of the underlying cause, but I can explore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org