You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hailin0 (via GitHub)" <gi...@apache.org> on 2023/04/06 14:36:21 UTC
[GitHub] [doris] hailin0 opened a new pull request, #18448: [feature](extension) support beats output to doris plugin
hailin0 opened a new pull request, #18448:
URL: https://github.com/apache/doris/pull/18448
# Proposed changes
Issue Number: close #xxx
## Problem summary
This is an output implementation of [elastic beats](https://github.com/elastic/beats) for support [Filebeat](https://github.com/elastic/beats/tree/master/filebeat), [Metricbeat](https://github.com/elastic/beats/tree/master/metricbeat), [Packetbeat](https://github.com/elastic/beats/tree/master/packetbeat), [Winlogbeat](https://github.com/elastic/beats/tree/master/winlogbeat), [Auditbeat](https://github.com/elastic/beats/tree/master/auditbeat), [Heartbeat](https://github.com/elastic/beats/tree/master/heartbeat) to [Apache Doris](https://github.com/apache/doris).
This module is used to output data to Doris for elastic beats, use the HTTP protocol to interact with the Doris FE Http interface, and import data through Doris's stream load.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [x] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1500753675
Hi @hailin0 Thanks for your contribution.
There are some issues:
1. We prefer using 4 spaces instead tab for intent, could you please change it?
2. How to run `extension/beats/doris/doris_integration_test.go`? Please give an example so that I can add it to regression test.
3. `extension/beats/go.mod ` missing license header, please add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] CalvinKirs commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1501676552
> > Hi @hailin0 Thanks for your contribution. There are some issues:↳
> >
> > 1. We prefer using 4 spaces instead tab for intent, could you please change it?
> > 2. How to run `extension/beats/doris/doris_integration_test.go`? Please give an example so that I can add it to regression test.
> > 3. `extension/beats/go.mod ` missing license header, please add it.
you should update this file https://github.com/apache/doris/blob/master/.licenserc.yaml#L6,
Then we will skip the checking of these two files.
>
> I see logstash plugin, how does he do the test↳
>
> https://github.com/apache/doris/tree/master/extension/logstash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1505126107
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #18448: [feature](extension) support beats output to doris
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #18448:
URL: https://github.com/apache/doris/pull/18448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1503664716
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 commented on pull request #18448: [feature](extension) support beats output to doris plugin
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1499172348
@morningman
Please help to review, Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1501798266
> > > Hi @hailin0 Thanks for your contribution. There are some issues:↳
> > >
> > > 1. We prefer using 4 spaces instead tab for intent, could you please change it?
> > > 2. How to run `extension/beats/doris/doris_integration_test.go`? Please give an example so that I can add it to regression test.
> > > 3. `extension/beats/go.mod ` missing license header, please add it.
>
> you should update this file https://github.com/apache/doris/blob/master/.licenserc.yaml#L6, Then we will skip the checking of these two files.
>
> > I see logstash plugin, how does he do the test↳
> > https://github.com/apache/doris/tree/master/extension/logstash
fixed.
please recheck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1503670016
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1500817253
> Hi @hailin0 Thanks for your contribution. There are some issues:
>
> 1. We prefer using 4 spaces instead tab for intent, could you please change it?
> 2. How to run `extension/beats/doris/doris_integration_test.go`? Please give an example so that I can add it to regression test.
> 3. `extension/beats/go.mod ` missing license header, please add it.
1. fixed
2. `doris_integration_test.go` depends on `doris-server` to started, is there any solution?
3. `go.mod` and `go.sum` cannot add file headers，reference [1](https://github.com/apache/dubbo-go/blob/main/go.mod) [2](https://github.com/apache/skywalking-satellite/blob/main/go.mod)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1500899000
> Hi @hailin0 Thanks for your contribution. There are some issues:
>
> 1. We prefer using 4 spaces instead tab for intent, could you please change it?
> 2. How to run `extension/beats/doris/doris_integration_test.go`? Please give an example so that I can add it to regression test.
> 3. `extension/beats/go.mod ` missing license header, please add it.
I see logstash plugin, how does he do the test
https://github.com/apache/doris/tree/master/extension/logstash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 commented on pull request #18448: [feature](extension) support beats output to doris plugin
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1499203549
<img width="1092" alt="image" src="https://user-images.githubusercontent.com/14371345/230417506-201fd254-0f06-47a9-9424-1f8f168edaea.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1503670148
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1503870540
> hi @hailin0 Could you give a brief command line to run `extension/beats/doris/doris_integration_test.go`, suppose that there is already Doris cluster?
>
> I will add this to the regression test later. And this PR LTGM
hi @morningman
reference
1. Init doris tables
```sql
create database test_beats;
CREATE TABLE `test_beats`.`test_beats_csv` (
`id` bigint(20),
`name` varchar(100),
`timestamp` varchar(100),
`type` varchar(100)
)
UNIQUE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_num" = "1"
);
CREATE TABLE `test_beats`.`test_beats_json` (
`id` bigint(20),
`name` varchar(100)
)
UNIQUE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_num" = "1"
);
CREATE TABLE `test_beats`.`test_beats_codec` (
`id` bigint(20),
`name` varchar(100),
`timestamp` varchar(100),
`type` varchar(100)
)
UNIQUE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_num" = "1"
)
```
2. Run golang testcase
``` shell
cd doris/extension/beats/doris/
go test -fenodes="http://localhost:8030" -user="root" -password="root" -database="test_beats"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hailin0 closed pull request #18448: [feature](extension) support beats output to doris
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 closed pull request #18448: [feature](extension) support beats output to doris
URL: https://github.com/apache/doris/pull/18448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18448: [feature](extension) support beats output to doris
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18448:
URL: https://github.com/apache/doris/pull/18448#issuecomment-1505127295
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org