You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/07/21 08:31:17 UTC
[GitHub] [incubator-doris] eyewire opened a new issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
eyewire opened a new issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288
**Describe the bug**
spark-doris-connector connect doris http-server-v2 failed
**To Reproduce**
use spark-doris-connector read from doris, http-server-v2 is set true, cannot read anything.but when we set to http-server-v1 is ok.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
**Additional context**
please fix this problem, because we also want to use flink-doris-connector to connect doris, but flink-doris-connector cannot use in http-server-v1, so we met a huge problem. please help~ thx~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] eyewire edited a comment on issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
Posted by GitBox <gi...@apache.org>.
eyewire edited a comment on issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288#issuecomment-884761403
> Can you show code or detail log ? it work normally for me
we upgrade our doris version ,but spark-doris-connector get error message:
21/07/22 08:57:46 ERROR SchemaUtils: Doris FE's response cannot map to schema. res: {"msg":"Unauthorized","code":401,"data":"Need auth information.","count":0}
org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "msg" (Class org.apache.doris.spark.rest.models.Schema), not marked as ignorable
at [Source: java.io.StringReader@55d9b8f0; line: 1, column: 9] (through reference chain: org.apache.doris.spark.rest.models.Schema["msg"])
at org.codehaus.jackson.map.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:53)
at org.codehaus.jackson.map.deser.StdDeserializationContext.unknownFieldException(StdDeserializationContext.java:267)
at org.codehaus.jackson.map.deser.std.StdDeserializer.reportUnknownProperty(StdDeserializer.java:673)
at org.codehaus.jackson.map.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:659)
at org.codehaus.jackson.map.deser.BeanDeserializer.handleUnknownProperty(BeanDeserializer.java:1365)
at org.codehaus.jackson.map.deser.BeanDeserializer._handleUnknown(BeanDeserializer.java:725)
at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:703)
at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
at org.apache.doris.spark.rest.RestService.parseSchema(RestService.java:239)
at org.apache.doris.spark.rest.RestService.getSchema(RestService.java:223)
at org.apache.doris.spark.sql.SchemaUtils$.discoverSchemaFromFe(SchemaUtils.scala:51)
at org.apache.doris.spark.sql.SchemaUtils$.discoverSchema(SchemaUtils.scala:41)
at org.apache.doris.spark.sql.DorisRelation.lazySchema$lzycompute(DorisRelation.scala:48)
at org.apache.doris.spark.sql.DorisRelation.lazySchema(DorisRelation.scala:48)
at org.apache.doris.spark.sql.DorisRelation.schema(DorisRelation.scala:52)
at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:40)
at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:389)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
at com.baidu.crowdtest.markdw.dbhelper.DbUtil.getDataFrame(DbUtil.java:128)
at com.baidu.crowdtest.markdw.etl.step.stage.support.dev.generateStartDataFrame(dev.java:29)
at com.baidu.crowdtest.markdw.etl.step.BaseSparkStep.runSparkStep(BaseSparkStep.java:242)
at com.baidu.crowdtest.markdw.MainEntrance.main(MainEntrance.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
code is ：
`spark.read().format("doris")
.option("user", "XXX")
.option("password", "XXXX")
.option("doris.table.identifier", "data_warehouse.ods_mark_task_config")
.option("doris.fenodes", "XXXXXX:8030")
// .option("doris.filter.query", "pro_id > 4000000 and pro_id < 4001000")
.load();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] JNSimba commented on issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
Posted by GitBox <gi...@apache.org>.
JNSimba commented on issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288#issuecomment-884724400
Can you show code or detail log ? it work normally for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
Posted by GitBox <gi...@apache.org>.
hf200012 commented on issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288#issuecomment-884727481
https://github.com/apache/incubator-doris/pull/5788
Use doris-0.14.12.4 version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] JNSimba commented on issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
Posted by GitBox <gi...@apache.org>.
JNSimba commented on issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288#issuecomment-884814882
your spark-doris-connector jar may be need update . #5366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] eyewire edited a comment on issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
Posted by GitBox <gi...@apache.org>.
eyewire edited a comment on issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288#issuecomment-884761403
> Can you show code or detail log ? it work normally for me
we upgrade our doris version ,but spark-doris-connector get error message:
21/07/22 08:52:35 ERROR SchemaUtils: Doris FE's response cannot map to schema. res: {"msg":"Unauthorized","code":401,"data":"Need auth information.","count":0}
code is ：
`spark.read().format("doris")
.option("user", "XXX")
.option("password", "XXXX")
.option("doris.table.identifier", "data_warehouse.ods_mark_task_config")
.option("doris.fenodes", "XXXXXX:8030")
// .option("doris.filter.query", "pro_id > 4000000 and pro_id < 4001000")
.load();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] eyewire commented on issue #6288: spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok.
Posted by GitBox <gi...@apache.org>.
eyewire commented on issue #6288:
URL: https://github.com/apache/incubator-doris/issues/6288#issuecomment-884761403
> Can you show code or detail log ? it work normally for me
21/07/22 08:52:35 ERROR SchemaUtils: Doris FE's response cannot map to schema. res: {"msg":"Unauthorized","code":401,"data":"Need auth information.","count":0}
code is ：
`spark.read().format("doris")
.option("user", "XXX")
.option("password", "XXXX")
.option("doris.table.identifier", "data_warehouse.ods_mark_task_config")
.option("doris.fenodes", "XXXXXX:8030")
// .option("doris.filter.query", "pro_id > 4000000 and pro_id < 4001000")
.load();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org