You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/01/12 07:27:26 UTC
[GitHub] [incubator-inlong] yunqingmoswu opened a new pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
yunqingmoswu opened a new pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147
### Title Name: [INLONG-2051] audit query support mysql and elasticsearch
[Feature] [InLong audit] Audit API #2051
### Motivation
*Explain here the context, and why you're making that change. What is the problem you're trying to solve.*
### Modifications
*Describe the modifications you've done.*
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] yunqingmoswu commented on a change in pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on a change in pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147#discussion_r786390517
##########
File path: inlong-manager/manager-web/src/main/resources/application-prod.properties
##########
@@ -44,24 +41,31 @@ spring.datasource.druid.testOnReturn=false
spring.datasource.druid.filters=stat,wall
# Open the mergeSql function through the connectProperties property, Slow SQL records
spring.datasource.druid.connectionProperties=druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000
-
# Manager address of Tube cluster, used to create Topic
cluster.tube.manager=http://127.0.0.1:8081
# Master address, used to manage Tube broker
cluster.tube.master=127.0.0.1:8000,127.0.0.1:8010
# Tube cluster ID
cluster.tube.clusterId=1
-
# Push configuration to the path on ZooKeeper
cluster.zk.url=127.0.0.1:2181
cluster.zk.root=inlong_hive
-
# Application name in Sort
sort.appName=inlong_app
-
# Pulsar admin URL
pulsar.adminUrl=http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080
# Pulsar broker address
pulsar.serviceUrl=pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650
# Default tenant of Pulsar
pulsar.defaultTenant=public
+# Elasticsearch config
Review comment:
It has done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] doleyzi commented on a change in pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
doleyzi commented on a change in pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147#discussion_r784692321
##########
File path: inlong-manager/manager-web/src/test/java/org/apache/inlong/manager/service/core/AuditServiceTest.java
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.core;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.inlong.manager.common.enums.AuditQuerySource;
+import org.apache.inlong.manager.common.pojo.audit.AuditRequest;
+import org.apache.inlong.manager.common.pojo.audit.AuditVO;
+import org.apache.inlong.manager.common.util.JsonUtils;
+import org.apache.inlong.manager.web.ServiceBaseTest;
+import org.junit.Test;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.test.context.TestComponent;
+
+@TestComponent
+public class AuditServiceTest extends ServiceBaseTest {
+
+ @Autowired
+ private AuditService auditService;
+
+ @Test
+ public void testQueryFromMySQL() throws IOException {
+ AuditRequest request = new AuditRequest();
+ request.setAuditId("3");
+ request.setInlongGroupId("g1");
+ request.setInlongStreamId("s1");
+ request.setDt("2022-01-01");
+ List<AuditVO> result = new ArrayList<>();
+ AuditVO auditVO = new AuditVO();
+ auditVO.setCount(123L);
+ auditVO.setLogTs("2022-01-01 00:00:00");
+ result.add(auditVO);
+ System.out.println(JsonUtils.toJson(result));
Review comment:
It is recommended to use assertEquals for unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] healchow merged pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
healchow merged pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] codecov-commenter edited a comment on pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147#issuecomment-1012784457
# [Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2147](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (691685b) into [master](https://codecov.io/gh/apache/incubator-inlong/commit/2c74fe011617bd0810608692b1f7e2d6077bf981?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2c74fe0) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #2147 +/- ##
=========================================
Coverage 12.25% 12.26%
- Complexity 1157 1159 +2
=========================================
Files 413 413
Lines 35243 35243
Branches 5544 5544
=========================================
+ Hits 4318 4321 +3
+ Misses 30153 30151 -2
+ Partials 772 771 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../tubemq/corebase/policies/FlowCtrlRuleHandler.java](https://codecov.io/gh/apache/incubator-inlong/pull/2147/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9wb2xpY2llcy9GbG93Q3RybFJ1bGVIYW5kbGVyLmphdmE=) | `34.51% <0.00%> (+0.44%)` | :arrow_up: |
| [.../inlong/tubemq/corebase/policies/FlowCtrlItem.java](https://codecov.io/gh/apache/incubator-inlong/pull/2147/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9wb2xpY2llcy9GbG93Q3RybEl0ZW0uamF2YQ==) | `40.00% <0.00%> (+1.11%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2c74fe0...691685b](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] dockerzhang commented on a change in pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
dockerzhang commented on a change in pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147#discussion_r785590058
##########
File path: inlong-manager/manager-web/src/main/resources/application-dev.properties
##########
@@ -45,24 +42,31 @@ spring.datasource.druid.testOnReturn=false
spring.datasource.druid.filters=stat,wall
# Open the mergeSql function through the connectProperties property, Slow SQL records
spring.datasource.druid.connectionProperties=druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000
-
# Manager address of Tube cluster, used to create Topic
cluster.tube.manager=http://127.0.0.1:8081
# Master address, used to manage Tube broker
cluster.tube.master=127.0.0.1:8000,127.0.0.1:8010
# Tube cluster ID
cluster.tube.clusterId=1
-
# Push configuration to the path on ZooKeeper
cluster.zk.url=127.0.0.1:2181
cluster.zk.root=inlong_hive
-
# Application name in Sort
sort.appName=inlong_app
-
# Pulsar admin URL
pulsar.adminUrl=http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080
# Pulsar broker address
pulsar.serviceUrl=pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650
# Default tenant of Pulsar
pulsar.defaultTenant=public
+# Elasticsearch config
Review comment:
how to distinguish using MySQL or ES?
##########
File path: inlong-manager/manager-web/src/main/resources/application-prod.properties
##########
@@ -44,24 +41,31 @@ spring.datasource.druid.testOnReturn=false
spring.datasource.druid.filters=stat,wall
# Open the mergeSql function through the connectProperties property, Slow SQL records
spring.datasource.druid.connectionProperties=druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000
-
# Manager address of Tube cluster, used to create Topic
cluster.tube.manager=http://127.0.0.1:8081
# Master address, used to manage Tube broker
cluster.tube.master=127.0.0.1:8000,127.0.0.1:8010
# Tube cluster ID
cluster.tube.clusterId=1
-
# Push configuration to the path on ZooKeeper
cluster.zk.url=127.0.0.1:2181
cluster.zk.root=inlong_hive
-
# Application name in Sort
sort.appName=inlong_app
-
# Pulsar admin URL
pulsar.adminUrl=http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080
# Pulsar broker address
pulsar.serviceUrl=pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650
# Default tenant of Pulsar
pulsar.defaultTenant=public
+# Elasticsearch config
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] yunqingmoswu commented on a change in pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on a change in pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147#discussion_r786394154
##########
File path: inlong-manager/manager-web/src/main/resources/application-prod.properties
##########
@@ -44,24 +41,31 @@ spring.datasource.druid.testOnReturn=false
spring.datasource.druid.filters=stat,wall
# Open the mergeSql function through the connectProperties property, Slow SQL records
spring.datasource.druid.connectionProperties=druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000
-
# Manager address of Tube cluster, used to create Topic
cluster.tube.manager=http://127.0.0.1:8081
# Master address, used to manage Tube broker
cluster.tube.master=127.0.0.1:8000,127.0.0.1:8010
# Tube cluster ID
cluster.tube.clusterId=1
-
# Push configuration to the path on ZooKeeper
cluster.zk.url=127.0.0.1:2181
cluster.zk.root=inlong_hive
-
# Application name in Sort
sort.appName=inlong_app
-
# Pulsar admin URL
pulsar.adminUrl=http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080
# Pulsar broker address
pulsar.serviceUrl=pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650
# Default tenant of Pulsar
pulsar.defaultTenant=public
+# Elasticsearch config
Review comment:
It has done by add config item,The following are the specific configuration item: Audit query source that decide what data source to query, currently only supports [MYSQL|ELASTICSEARCH]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] codecov-commenter commented on pull request #2147: [INLONG-2051] audit query support mysql and elasticsearch
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #2147:
URL: https://github.com/apache/incubator-inlong/pull/2147#issuecomment-1012784457
# [Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2147](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f439ab8) into [master](https://codecov.io/gh/apache/incubator-inlong/commit/2c74fe011617bd0810608692b1f7e2d6077bf981?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2c74fe0) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #2147 +/- ##
============================================
+ Coverage 12.25% 12.27% +0.02%
- Complexity 1157 1159 +2
============================================
Files 413 413
Lines 35243 35243
Branches 5544 5544
============================================
+ Hits 4318 4327 +9
+ Misses 30153 30147 -6
+ Partials 772 769 -3
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../producer/qltystats/DefaultBrokerRcvQltyStats.java](https://codecov.io/gh/apache/incubator-inlong/pull/2147/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9pbmxvbmcvdHViZW1xL2NsaWVudC9wcm9kdWNlci9xbHR5c3RhdHMvRGVmYXVsdEJyb2tlclJjdlFsdHlTdGF0cy5qYXZh) | `46.09% <0.00%> (+0.78%)` | :arrow_up: |
| [.../java/org/apache/flume/sink/tubemq/TubemqSink.java](https://codecov.io/gh/apache/incubator-inlong/pull/2147/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29ubmVjdG9ycy90dWJlbXEtY29ubmVjdG9yLWZsdW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9mbHVtZS9zaW5rL3R1YmVtcS9UdWJlbXFTaW5rLmphdmE=) | `55.42% <0.00%> (+4.00%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2c74fe0...f439ab8](https://codecov.io/gh/apache/incubator-inlong/pull/2147?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org