You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Alexander Rukletsov (JIRA)" <ji...@apache.org> on 2016/11/02 11:50:58 UTC
[jira] [Updated] (MESOS-4642) Mesos Agent Json API can dump binary
data from log files out as invalid JSON.
[ https://issues.apache.org/jira/browse/MESOS-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Rukletsov updated MESOS-4642:
---------------------------------------
Summary: Mesos Agent Json API can dump binary data from log files out as invalid JSON. (was: Mesos Agent Json API can dump binary data from log files out as invalid JSON)
> Mesos Agent Json API can dump binary data from log files out as invalid JSON.
> -----------------------------------------------------------------------------
>
> Key: MESOS-4642
> URL: https://issues.apache.org/jira/browse/MESOS-4642
> Project: Mesos
> Issue Type: Bug
> Components: json api, slave
> Affects Versions: 0.27.0
> Reporter: Steven Schlansker
> Priority: Critical
>
> One of our tasks accidentally started logging binary data to stderr. This was not intentional and generally should not happen -- however, it causes severe problems with the Mesos Agent "files/read.json" API, since it gladly dumps this binary data out as invalid JSON.
> {code}
> # hexdump -C /path/to/task/stderr | tail
> 0003d1f0 6f 6e 6e 65 63 74 69 6f 6e 0a 4e 45 54 3a 20 31 |onnection.NET: 1|
> 0003d200 20 6f 6e 72 65 61 64 20 45 4e 4f 45 4e 54 20 32 | onread ENOENT 2|
> 0003d210 39 35 34 35 36 20 32 35 31 20 32 39 35 37 30 37 |95456 251 295707|
> 0003d220 0a 01 00 00 00 00 00 00 ac 57 65 64 2c 20 31 30 |.........Wed, 10|
> 0003d230 20 55 6e 72 65 63 6f 67 6e 69 7a 65 64 20 69 6e | Unrecognized in|
> 0003d240 70 75 74 20 68 65 61 64 65 72 0a |put header.|
> {code}
> {code}
> # curl 'http://agent-host:5051/files/read.json?path=/path/to/task/stderr&offset=220443&length=90000&grep=' | hexdump -C
> 00007970 6e 65 63 74 69 6f 6e 5c 6e 4e 45 54 3a 20 31 20 |nection\nNET: 1 |
> 00007980 6f 6e 72 65 61 64 20 45 4e 4f 45 4e 54 20 32 39 |onread ENOENT 29|
> 00007990 35 34 35 36 20 32 35 31 20 32 39 35 37 30 37 5c |5456 251 295707\|
> 000079a0 6e 5c 75 30 30 30 31 5c 75 30 30 30 30 5c 75 30 |n\u0001\u0000\u0|
> 000079b0 30 30 30 5c 75 30 30 30 30 5c 75 30 30 30 30 5c |000\u0000\u0000\|
> 000079c0 75 30 30 30 30 5c 75 30 30 30 30 ac 57 65 64 2c |u0000\u0000.Wed,|
> 000079d0 20 31 30 20 55 6e 72 65 63 6f 67 6e 69 7a 65 64 | 10 Unrecognized|
> 000079e0 20 69 6e 70 75 74 20 68 65 61 64 65 72 5c 6e 22 | input header\n"|
> 000079f0 2c 22 6f 66 66 73 65 74 22 3a 32 32 30 34 34 33 |,"offset":220443|
> 00007a00 7d |}|
> {code}
> This causes downstream sadness:
> {code}
> ERROR [2016-02-10 18:55:12,303] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 0ee749630f8b26f1
> ! com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xac
> ! at [Source: org.jboss.netty.buffer.ChannelBufferInputStream@6d69ee8; line: 1, column: 31181]
> ! at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3339) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3333) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2360) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2287) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:286) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:29) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:12) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:523) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:381) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1073) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject(SuperSonicBeanDeserializer.java:196) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:142) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:117) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3562) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2648) ~[singularity-0.4.9.jar:0.4.9]
> ! at com.hubspot.singularity.data.SandboxManager.read(SandboxManager.java:97) ~[singularity-0.4.9.jar:0.4.9]
> {code}
> This is extremely similar to https://issues.apache.org/jira/browse/MESOS-3771
> Since this is now the second major issue caused by this, is there any possibility of using a JSON processing library that actually guarantees spec-compliant output? I know we can fix the point problem again here, but it is frustrating that this keeps happening, and I'm sure it will happen again in the future.
> Failing that, maybe we should audit all JSON objects produced to ensure they cannot contain binary data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)