You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flagon.apache.org by "Jyyjy (via GitHub)" <gi...@apache.org> on 2023/05/23 22:30:41 UTC
[GitHub] [flagon-useralejs] Jyyjy opened a new issue, #359: Create testcases to verify userale log data with a schema
Jyyjy opened a new issue, #359:
URL: https://github.com/apache/flagon-useralejs/issues/359
I'm creating this issue based on this PR discussion https://github.com/apache/flagon-useralejs/pull/347.
There is a need for a JSON schema for userale log data, and new testcases that verify logs against said schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flagon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flagon-useralejs] brucearctor commented on issue #359: Create testcases to verify userale log data with a schema
Posted by "brucearctor (via GitHub)" <gi...@apache.org>.
brucearctor commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1591846060
I'm at https://beamsummit.org/ this week --> Let me see what else people here have to say about the differences [ JSON Schema, Proto, Avro, Thrift, FlatBuffers, etc ]. Agreed we probably want a "reference", which translates to the others projected. Whatever that reference/source-of-truth should probably support most of the complex data types. I imagine there isn't always 1:1 mapping across each potential solution that could define schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flagon-useralejs] poorejc commented on issue #359: Create testcases to verify userale log data with a schema
Posted by "poorejc (via GitHub)" <gi...@apache.org>.
poorejc commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1577894660
@Jyyjy I love this idea a lot. My .02c is that we should build the template to be something that end-users can modify and adjust to their preferred schema if they opt to log exclusively with custom logs. We may also see some wonky cases from the browser plugin, which has browser logs. So it may be there there are a few canonical JSON schema's we're dealing with. Notwithstanding, i think this is a very good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create testcases to verify userale log data with a schema (flagon-useralejs)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1681081778
> I have begun to implement a json schema with test cases [here](https://github.com/apache/flagon-useralejs/commit/5fea1aea062068af9db193b85420a58098f1441e).
>
> There is an issue with "load" type logs. Load logs are [created ](https://github.com/apache/flagon-useralejs/blob/9c4cca7cc728ecf00a4b78c826ec5d25e7b8a87c/src/main.js#L70)via `packageCustomLog()`, but the "logType" field is overwritten to be "raw". The schema I've created requires certain fields for raw logs with the expectation that raw logs are created with `packageLog()`.
>
> Instead of loosening the schema to fit raw load logs, I think it makes more sense to make load logs "custom". And perhaps then the raw/custom dichotomy should be renamed to something like event/nonevent? The raw/custom names imply that the base userale script only produces raw logs, and any modifications will only produce custom logs, but that is not the case.
>
> Any thoughts/objections?
Here's the beginnings of what I explained above. I figure these are pretty major changes so they would go into a 3.0 release. I also think it would make sense to rework browser logs so that they use the same functions and fit the same schema.
https://github.com/UMD-ARLIS/incubator-flagon-useralejs/tree/schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
[GitHub] [flagon-useralejs] brucearctor commented on issue #359: Create testcases to verify userale log data with a schema
Posted by "brucearctor (via GitHub)" <gi...@apache.org>.
brucearctor commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1577944586
https://github.com/chrusty/protoc-gen-jsonschema
I'd suggest protobuf as more fundamental [ and can be used to generate JSON Schemas, see the link above ] given how well protobuf plays with Kafka, message queues, and elsewhere. In the data world I operate, JSONSchema is not as prevalent. That said, JSONSchema > nothing, and @Jyyjy is offering to do the work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flagon-useralejs] Jyyjy commented on issue #359: Create testcases to verify userale log data with a schema
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1590313549
I'm assuming that user-ale will add support for protobuf and other data serialization formats. And as different formats are added, their schemas should be added accordingly. There's a variety of schema translation tools, with varying levels of maturity that can minimize the work needed to support these schemas.
I think the question is which schema format acts best as the original source of truth from which other schemas can be derived. I'm not sure what the answer is, but I thought JSON schema would fit because the underlying data is a JS Object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create testcases to verify userale log data with a schema (flagon-useralejs)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1664348799
I have begun to implement a json schema with test cases [here](https://github.com/apache/flagon-useralejs/commit/5fea1aea062068af9db193b85420a58098f1441e).
There is an issue "load" type logs. Load logs are [created ](https://github.com/apache/flagon-useralejs/blob/9c4cca7cc728ecf00a4b78c826ec5d25e7b8a87c/src/main.js#L70)via `packageCustomLog()`, but the "logType" field is overwritten to be "raw". The schema I've created requires certain fields for raw logs with the expectation that raw logs are created with `packageLog()`.
Instead of loosening the schema to fit raw load logs, I think it makes more sense to make load logs "custom". And perhaps then the raw/custom dichotomy should be renamed to something like event/nonevent? The raw/custom names imply that the base userale script only produces raw logs, and any modifications will only produce custom logs, but that is not the case.
Any thoughts/objections?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [I] Create testcases to verify userale log data with a schema (flagon-useralejs)
Posted by "EandrewJones (via GitHub)" <gi...@apache.org>.
EandrewJones commented on issue #359:
URL: https://github.com/apache/flagon-useralejs/issues/359#issuecomment-1912515334
I'm rebumping this discussion since it dovetails with the conversations we had in our stand up today and the associated spate of tickets that came out of it.
I can indeed confirm, some of the changes proposed would be breaking changes and therefore necessitate a new major version. However, without them, all of the schema inconsistencies (esp. between the plugin and package) render down stream analytics on the firehose very tough and unreliable.
I like the notion of a JSON schema because it's closer to the current implementation and requires fewer changes. However, given the work we're doing with TAP which is going to rely heavily on Kafka and Beam, introducing protobuf, as @brucearctor suggests, makes more sense for our medium-term goals.
https://www.npmjs.com/package/proto3-json-serializer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org