You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "intr3p1d (via GitHub)" <gi...@apache.org> on 2024/02/02 02:45:45 UTC
[I] `ISO_8859_1` breaking Hangul in CLP `_logtype` column [pinot]
intr3p1d opened a new issue, #12352:
URL: https://github.com/apache/pinot/issues/12352
# Cause
`clp-ffi-java` [internally use](https://github.com/y-scope/clp-ffi-java/blob/c4a74dbdeb09bd4e7e3d119826dddbe5005ccf53/src/main/java/com/yscope/clp/compressorfrontend/EncodedMessage.java#L30-L36) `StandardCharsets.ISO_8859_1` in `EncodedMessage.getLogTypeAsString();`

(`getDictionaryVarsAsStrings` also)
# Effect
https://github.com/apache/pinot/blob/0a4398634be81cdbbe891b3da249134ef98743e7/pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogRecordExtractor.java#L151-L154
This makes some characters broken in `column_logtype` like this:
`Request processing failed: jakarta.validation.ConstraintViolationException: getAgentsList.from: 0 이상이어야 합니다`
into
`Request processing failed: jakarta.validation.ConstraintViolationException: getAgentsList.from: ì´ìì´ì´ì¼ í©ëë¤`
This is fine after going through the CLPDECODE function, but when dealing with individual `_logtype` columns, these broken strings don't seem appropriate (LIKE searches, etc).
The `clp-ffi-java` library makes all EncodedMessage member variables public. So it would be nice if pinot's `CLPLogMessageDecoder` could handle them (or at least match the other encodings used internally by pinot).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] `ISO_8859_1` breaking UTF-8 in CLP `_logtype` column [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12352:
URL: https://github.com/apache/pinot/issues/12352#issuecomment-5690114962
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] `ISO_8859_1` breaking UTF-8 in CLP `_logtype` column [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12352:
URL: https://github.com/apache/pinot/issues/12352#issuecomment-5234606687
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] `ISO_8859_1` breaking UTF-8 in CLP `_logtype` column [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12352: `ISO_8859_1` breaking UTF-8 in CLP `_logtype` column
URL: https://github.com/apache/pinot/issues/12352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org