You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by "Eugene (JIRA)" <ji...@apache.org> on 2019/03/13 09:04:00 UTC
[jira] [Commented] (GRIFFIN-238) try to write metric into
elasticsearch, but fail to call post api
[ https://issues.apache.org/jira/browse/GRIFFIN-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16791468#comment-16791468 ]
Eugene commented on GRIFFIN-238:
--------------------------------
root cause:
this error is led by limit of scalaj.http implementation.
scalaj.http only allows HttpURLConnection to call remote web api, but on spark cluster condition, no HttpURLConnection returns instead of class org.apache.hadoop.fs.FsUrlConnection. so rest post api never really calls out.
private def doConnection[T](
parser: (Int, Map[String, IndexedSeq[String]], InputStream) => T,
urlToFetch: URL,
connectFunc: (HttpRequest, HttpURLConnection) => Unit
): HttpResponse[T] = {
proxyConfig.map(urlToFetch.openConnection).getOrElse(urlToFetch.openConnection) match {
case conn: HttpURLConnection =>
resolution:
use apache httpcomponent instead of scalaj.http on spark jobs
> try to write metric into elasticsearch, but fail to call post api
> -----------------------------------------------------------------
>
> Key: GRIFFIN-238
> URL: https://issues.apache.org/jira/browse/GRIFFIN-238
> Project: Griffin (Incubating)
> Issue Type: Bug
> Components: accuracy-batch
> Affects Versions: 0.4.0
> Reporter: Eugene
> Assignee: Eugene
> Priority: Major
> Fix For: 0.5.0
>
> Attachments: ubuntu_41972
>
>
> when a batch job was scheduled in spark cluster, the last step is saving metric into ES sink. but we hit exception like below
>
> 19/03/05 16:22:03 WARN sink.SinkTaskRunner$: task 1551769200000 fails [ using time 0 ms ] : org.apache.hadoop.fs.FsUrlConnection:http://127.0.0.1:9200/griffin/accuracy (of class org.apache.hadoop.fs.FsUrlConnection)
> 19/03/05 16:22:03 ERROR sink.SinkTaskRunner$: task fails: task 1551769200000 retry ends but fails
> scala.MatchError: org.apache.hadoop.fs.FsUrlConnection:http://127.0.0.1:9200/griffin/accuracy (of class org.apache.hadoop.fs.FsUrlConnection)
> at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:343)
> at scalaj.http.HttpRequest.exec(Http.scala:335)
> at scalaj.http.HttpRequest.asString(Http.scala:455)
> at org.apache.griffin.measure.utils.HttpUtil$.httpRequest(HttpUtil.scala:49)
> at org.apache.griffin.measure.sink.ElasticSearchSink$$anonfun$org$apache$griffin$measure$sink$ElasticSearchSink$$func$1$1.apply$mcZ$sp(ElasticSearchSink.scala:67)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)