You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2022/05/13 20:54:49 UTC
[GitHub] [drill] nielsbasjes opened a new pull request, #2549: DRILL-8225: Update LogParser and Yauaa to support User-Agent Client Hints
nielsbasjes opened a new pull request, #2549:
URL: https://github.com/apache/drill/pull/2549
# [DRILL-8225](https://issues.apache.org/jira/browse/DRILL-8225): Update LogParser and Yauaa to support User-Agent Client Hints
## Description
In order to support logging, parsing and analyzing the User-Agent Client Hints it was needed to update the LogParser to fix a parsing bug and to update Yauaa to support actually analyzing them.
See https://yauaa.basjes.nl/using/clienthints/
## Documentation
The change includes the updated Readme for the UDFs.
## Testing
The change includes the original tests and additional tests for the new functionality.
For some of the existing tests the readability of the test has been improved.
For specific edge cases (like passing a null value and empty string) the parse results have changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2549: DRILL-8225: Update LogParser and Yauaa to support User-Agent Client Hints
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2549:
URL: https://github.com/apache/drill/pull/2549#discussion_r875447108
##########
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java:
##########
@@ -168,4 +241,185 @@ public void testEmptyUAString() throws Exception {
.baselineValues("Hacker")
.go();
}
+
+ @Test
+ public void testClientHints() throws Exception {
+ String query =
+ "SELECT " +
+ " t1.ua.DeviceClass AS DeviceClass,\n" +
+ " t1.ua.DeviceName AS DeviceName,\n" +
+ " t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ " t1.ua.DeviceCpu AS DeviceCpu,\n" +
+ " t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ " t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ " t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ " t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ " t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ " t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ " t1.ua.OperatingSystemNameVersionMajor AS OperatingSystemNameVersionMajor,\n" +
+ " t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ " t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ " t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ " t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ " t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ " t1.ua.LayoutEngineNameVersionMajor AS LayoutEngineNameVersionMajor,\n" +
+ " t1.ua.AgentClass AS AgentClass,\n" +
+ " t1.ua.AgentName AS AgentName,\n" +
+ " t1.ua.AgentVersion AS AgentVersion,\n" +
+ " t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ " t1.ua.AgentNameVersion AS AgentNameVersion,\n" +
+ " t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor\n" +
+ "FROM (" +
+ " SELECT" +
+ " parse_user_agent(" +
+ " 'User-Agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36'," +
+ " 'Sec-Ch-Ua', '\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"'," +
+ " 'Sec-Ch-Ua-Arch', '\"x86\"'," +
+ " 'Sec-Ch-Ua-Bitness', '\"64\"'," +
+ " 'Sec-Ch-Ua-Full-Version', '\"100.0.4896.127\"'," +
+ " 'Sec-Ch-Ua-Full-Version-List', '\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"100.0.4896.127\", \"Google Chrome\";v=\"100.0.4896.127\"'," +
+ " 'Sec-Ch-Ua-Mobile', '?0'," +
+ " 'Sec-Ch-Ua-Model', '\"\"'," +
+ " 'Sec-Ch-Ua-Platform', '\"Linux\"'," +
+ " 'Sec-Ch-Ua-Platform-Version', '\"5.13.0\"'," +
+ " 'Sec-Ch-Ua-Wow64', '?0'" +
+ " ) AS ua " +
+ " FROM (values(1))" +
+ ") AS t1";
+
+ testBuilder()
+ .sqlQuery(query)
+ .unOrdered()
+ .baselineRecords(
+ Collections.singletonList(// Singleton list because we expect 1 record
+ expectations(
+ "DeviceClass", "Desktop",
+ "DeviceName", "Linux Desktop",
+ "DeviceBrand", "Unknown",
+ "DeviceCpu", "Intel x86_64",
+ "DeviceCpuBits", "64",
+ "OperatingSystemClass", "Desktop",
+ "OperatingSystemName", "Linux",
+ "OperatingSystemVersion", "5.13.0",
+ "OperatingSystemVersionMajor", "5",
+ "OperatingSystemNameVersion", "Linux 5.13.0",
+ "OperatingSystemNameVersionMajor", "Linux 5",
+ "LayoutEngineClass", "Browser",
+ "LayoutEngineName", "Blink",
+ "LayoutEngineVersion", "100.0",
+ "LayoutEngineVersionMajor", "100",
+ "LayoutEngineNameVersion", "Blink 100.0",
+ "LayoutEngineNameVersionMajor", "Blink 100",
+ "AgentClass", "Browser",
+ "AgentName", "Chrome",
+ "AgentVersion", "100.0.4896.127",
+ "AgentVersionMajor", "100",
+ "AgentNameVersion", "Chrome 100.0.4896.127",
+ "AgentNameVersionMajor", "Chrome 100"
+ )
+ )
+ )
+ .go();
+ }
+
+ // ====================================================================
+
+ @Test
+ public void testEmptyClientHints() throws Exception {
+ String query =
+ "SELECT " +
+ " t1.ua.DeviceClass AS DeviceClass,\n" +
+ " t1.ua.DeviceName AS DeviceName,\n" +
+ " t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ " t1.ua.DeviceCpu AS DeviceCpu,\n" +
+ " t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ " t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ " t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ " t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ " t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ " t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ " t1.ua.OperatingSystemNameVersionMajor AS OperatingSystemNameVersionMajor,\n" +
+ " t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ " t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ " t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ " t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ " t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ " t1.ua.LayoutEngineNameVersionMajor AS LayoutEngineNameVersionMajor,\n" +
+ " t1.ua.AgentClass AS AgentClass,\n" +
+ " t1.ua.AgentName AS AgentName,\n" +
+ " t1.ua.AgentVersion AS AgentVersion,\n" +
+ " t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ " t1.ua.AgentNameVersion AS AgentNameVersion,\n" +
+ " t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor\n" +
+ "FROM (" +
+ " SELECT" +
+ " parse_user_agent(" +
+ // NOTE: Here we do NOT say "User-Agent" --> It is just the first one in the list.
+ " 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36'," +
+ " 'Sec-Ch-Ua', ''," +
+ " 'Sec-Ch-Ua-Arch', ''," +
+ " 'Sec-Ch-Ua-Bitness', ''," +
+ " 'Sec-Ch-Ua-Full-Version', ''," +
+ " 'Sec-Ch-Ua-Full-Version-List', ''," +
+ " 'Sec-Ch-Ua-Mobile', ''," +
+ " 'Sec-Ch-Ua-Model', ''," +
+ " 'Sec-Ch-Ua-Platform', ''," +
+ " 'Sec-Ch-Ua-Platform-Version', ''," +
+ " 'Sec-Ch-Ua-Wow64', ''" +
+ " ) AS ua " +
+ " FROM (values(1))" +
+ ") AS t1";
+
+ testBuilder()
+ .sqlQuery(query)
+ .unOrdered()
+ .baselineRecords(
+ Collections.singletonList(// Singleton list because we expect 1 record
+ expectations(
+ "DeviceClass", "Desktop",
+ "DeviceName", "Linux Desktop",
+ "DeviceBrand", "Unknown",
+ "DeviceCpu", "Intel x86_64",
+ "DeviceCpuBits", "64",
+ "OperatingSystemClass", "Desktop",
+ "OperatingSystemName", "Linux",
+ "OperatingSystemVersion", "??",
+ "OperatingSystemVersionMajor", "??",
+ "OperatingSystemNameVersion", "Linux ??",
+ "OperatingSystemNameVersionMajor", "Linux ??",
+ "LayoutEngineClass", "Browser",
+ "LayoutEngineName", "Blink",
+ "LayoutEngineVersion", "100.0",
+ "LayoutEngineVersionMajor", "100",
+ "LayoutEngineNameVersion", "Blink 100.0",
+ "LayoutEngineNameVersionMajor", "Blink 100",
+ "AgentClass", "Browser",
+ "AgentName", "Chrome",
+ "AgentVersion", "100.0.4896.127",
+ "AgentVersionMajor", "100",
+ "AgentNameVersion", "Chrome 100.0.4896.127",
+ "AgentNameVersionMajor", "Chrome 100"
+ )
+ )
+ )
+ .go();
+ }
+
+ /**
+ * Converts a more readable list of keys and values into what the ClusterTest supports.
+ * @param strings List of ["key", "value"]
+ * @return A Map of the same keys and values that is in the right format.
+ */
+ private Map<String, Object> expectations(String... strings) {
Review Comment:
This is a nice function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre merged pull request #2549: DRILL-8225: Update LogParser and Yauaa to support User-Agent Client Hints
Posted by GitBox <gi...@apache.org>.
cgivre merged PR #2549:
URL: https://github.com/apache/drill/pull/2549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org