You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/19 07:29:22 UTC

[GitHub] [incubator-doris] zjd-melo opened a new issue #5263: How to use logstash sync mysql data to doris

zjd-melo opened a new issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763393721


   By remove the stdout part, this error happend.
   ```[2021-01-20T15:15:17,534][WARN ][logstash.outputs.doris][main] Failed to flush outgoing items 
   {:outgoing_count=>1, :exception=>"TypeError", :backtrace=>["org/jruby/RubyString.java:2684:in `<<'", 
   "/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:179:in `block in flush'", 
   "org/jruby/RubyArray.java:1800:in `each'", 
   "/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:178:in `flush'", 
   "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", 
   "org/jruby/RubyHash.java:1417:in `each'", 
   "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", 
   "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:112:in `block in buffer_initialize'", 
   "org/jruby/RubyKernel.java:1425:in `loop'", 
   "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:110:in `block in buffer_initialize'"]}```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381571


   @francisoliverlee 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] francisoliverlee commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763375496


   @zjd-melo 
   check this doc from Elastic, it seems ok for jdbc event.
   https://www.elastic.co/guide/en/logstash/current/input-plugins.html


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo removed a comment on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo removed a comment on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381571


   @francisoliverlee 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763357019


   Thank you for your replay, does this plugin receive events from jdbc input?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo removed a comment on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo removed a comment on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381680


   @francisoliverlee 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763381680


   @francisoliverlee 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zjd-melo commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
zjd-melo commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763386672


    @francisoliverlee 
    ```input {
     jdbc {
       jdbc_connection_string => "jdbc:mysql://192.168.10.47:3306/micros_policy"
       jdbc_user => "root"
       jdbc_password => "sqlpwd"
       jdbc_default_timezone => "Asia/Shanghai"
       jdbc_driver_class => "com.mysql.jdbc.Driver"
       jdbc_fetch_size => 5000
   
       schedule => "* * * * * Asia/Shanghai"
   
       record_last_run => true
       tracking_column => "last_modified"
       tracking_column_type => "timestamp"
       last_run_metadata_path => "./last_updated.db"
       use_column_value => true
       clean_run => false
   
       statement => "select id, model_name,'\n' as newline  from policy_newmodelingconfig  where last_modified > :sql_last_value"
     }
   }
   
   filter {
     mutate {
       remove_field => ["@version", "@timestamp"]
     }
   }
   
   output {
     stdout {
       codec => plain {
         format => "%{[id]},%{[model_name]}%{[newline]}"
       }
     }
   
     doris {
       codec => plain {
         format => "%{[id]},%{[model_name]}%{[newline]}"
       }
       timezone => "Asia/Shanghai"
   
       http_hosts => ["http://localhost:8030"]
       user => "root"
       password => ""
       db => "test"
       table => "test"
       label_prefix => "doris"
       column_separator => ","
     }
   }
   ```
   With this conf i got this.
   ```
   3,m0000001
   [2021-01-20T14:59:01,145][WARN ][logstash.outputs.doris   ][main] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"TypeError", :backtrace=>["org/jruby/RubyString.java:2684:in `<<'",
     "/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:179:in `block in flush'",
     "org/jruby/RubyArray.java:1800:in `each'", "/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:178:in `flush'",
     "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1417:in `each'",
     "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'",
     "/logstash-7.4.2/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'",
     "/logstash-7.4.2/vendor/local_gems/b61803dd/logstash-output-doris-0.1.0/lib/logstash/outputs/doris.rb:171:in `receive'",
     "/logstash-7.4.2/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1800:in `each'",
     "/logstash-7.4.2/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'",
     "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'",
     "/logstash-7.4.2/logstash-core/lib/logstash/java_pipeline.rb:243:in `block in start_workers'"]}
   ```
   
   I don't know why this happen, should i format the event messsgae as csv?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] francisoliverlee commented on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763282917


   @zjd-melo 
   http://doris.apache.org/master/zh-CN/extending-doris/logstash.html#%E5%AE%89%E8%A3%85%E5%92%8C%E7%BC%96%E8%AF%91


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] francisoliverlee edited a comment on issue #5263: How to use logstash sync mysql data to doris

Posted by GitBox <gi...@apache.org>.
francisoliverlee edited a comment on issue #5263:
URL: https://github.com/apache/incubator-doris/issues/5263#issuecomment-763375496






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org