You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/19 07:29:22 UTC
[GitHub] [incubator-doris] zjd-melo opened a new issue #5263: How to use logstash sync mysql data to doris
zjd-melo opened a new issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763393721
By remove the stdout part, this error happend.
```[2021-01-20T15:15:17,534][WARN ][logstash.outputs.doris][main] Failed to flush outgoing items
{:outgoing_count=>1, :exception=>"TypeError", :backtrace=>["org/jruby/RubyString.java:2684:in `<<'",
"/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:179:in `block in flush'",
"org/jruby/RubyArray.java:1800:in `each'",
"/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:178:in `flush'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'",
"org/jruby/RubyHash.java:1417:in `each'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:112:in `block in buffer_initialize'",
"org/jruby/RubyKernel.java:1425:in `loop'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:110:in `block in buffer_initialize'"]}```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381571
@francisoliverlee
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] francisoliverlee commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763375496
@zjd-melo
check this doc from Elastic, it seems ok for jdbc event.
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo removed a comment on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo removed a comment on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381571
@francisoliverlee
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763357019
Thank you for your replay, does this plugin receive events from jdbc input?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo removed a comment on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo removed a comment on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381680
@francisoliverlee
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381680
@francisoliverlee
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763386672
@francisoliverlee
```input {
jdbc {
jdbc_connection_string => "jdbc:mysql://192.168.10.47:3306/micros_policy"
jdbc_user => "root"
jdbc_password => "sqlpwd"
jdbc_default_timezone => "Asia/Shanghai"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_fetch_size => 5000
schedule => "* * * * * Asia/Shanghai"
record_last_run => true
tracking_column => "last_modified"
tracking_column_type => "timestamp"
last_run_metadata_path => "./last_updated.db"
use_column_value => true
clean_run => false
statement => "select id, model_name,'\n' as newline from policy_newmodelingconfig where last_modified > :sql_last_value"
}
}
filter {
mutate {
remove_field => ["@version", "@timestamp"]
}
}
output {
stdout {
codec => plain {
format => "%{[id]},%{[model_name]}%{[newline]}"
}
}
doris {
codec => plain {
format => "%{[id]},%{[model_name]}%{[newline]}"
}
timezone => "Asia/Shanghai"
http_hosts => ["http://localhost:8030"]
user => "root"
password => ""
db => "test"
table => "test"
label_prefix => "doris"
column_separator => ","
}
}
```
With this conf i got this.
```
3,m0000001
[2021-01-20T14:59:01,145][WARN ][logstash.outputs.doris ][main] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"TypeError", :backtrace=>["org/jruby/RubyString.java:2684:in `<<'",
"/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:179:in `block in flush'",
"org/jruby/RubyArray.java:1800:in `each'", "/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:178:in `flush'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1417:in `each'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'",
"/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'",
"/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:171:in `receive'",
"/logstash-7.4.2/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1800:in `each'",
"/logstash-7.4.2/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'",
"org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'",
"/logstash-7.4.2/logstash-core/lib/logstash/java_pipeline.rb:243:in `block in start_workers'"]}
```
I don't know why this happen, should i format the event messsgae as csv?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] francisoliverlee commented on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763282917
@zjd-melo
http://doris.apache.org/master/zh-CN/extending-doris/logstash.html#%E5%AE%89%E8%A3%85%E5%92%8C%E7%BC%96%E8%AF%91
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] francisoliverlee edited a comment on issue #5263: How to use logstash sync mysql data to doris
Posted by GitBox <gi...@apache.org>.
francisoliverlee edited a comment on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763375496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org