You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zhengshubin (via GitHub)" <gi...@apache.org> on 2025/04/11 06:15:38 UTC

[I] [Bug] java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.toArrowSchema [doris-spark-connector]

zhengshubin opened a new issue, #308:
URL: https://github.com/apache/doris-spark-connector/issues/308

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   spark 3.5
   spark-doris-connector 25.0.1
   
   ### What's Wrong?
   
   java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.
   toArrowSchema(Lorg/apache/spark/sql/types/StructType;Ljava/lang/String;)Lorg/apache/doris/shaded/org/apache/arrow/vector/types/pojo/Schema;
   org.apache.doris.spark.client.write.StreamLoadProcessor.toArrowFormat(StreamLoadProcessor.java:55)\n\tat 
    org.apache.doris.spark.client.write.AbstractStreamLoadProcessor.stop(AbstractStreamLoadProcessor.java:162)\n\tat 
    org.apache.doris.spark.write.DorisDataWriter.commit(DorisDataWriter.scala:73)\n\tat
   
   ### What You Expected?
   
   .
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.toArrowSchema [doris-spark-connector]

Posted by "zhijunn (via GitHub)" <gi...@apache.org>.
zhijunn commented on issue #308:
URL: https://github.com/apache/doris-spark-connector/issues/308#issuecomment-2798748643

   When set "doris.sink.properties.format"="arrow"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.toArrowSchema [doris-spark-connector]

Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on issue #308:
URL: https://github.com/apache/doris-spark-connector/issues/308#issuecomment-2800821546

   please use csv or json format


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.toArrowSchema [doris-spark-connector]

Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on issue #308:
URL: https://github.com/apache/doris-spark-connector/issues/308#issuecomment-2864867207

    This  pr may solve your problem https://github.com/apache/doris-spark-connector/pull/315


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.toArrowSchema [doris-spark-connector]

Posted by "zhijunn (via GitHub)" <gi...@apache.org>.
zhijunn commented on issue #308:
URL: https://github.com/apache/doris-spark-connector/issues/308#issuecomment-2798749405

   > For spark 2.4, has similar problem. It seems that Doris copied Spark's ArrowUtils codes but replaced the ArrowUtils.create() and somewhere other method's parameter class with the shaed class of Arrow, causing the method conflict with Spark.
   
   When set "doris.sink.properties.format"="arrow"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils.toArrowSchema [doris-spark-connector]

Posted by "zhijunn (via GitHub)" <gi...@apache.org>.
zhijunn commented on issue #308:
URL: https://github.com/apache/doris-spark-connector/issues/308#issuecomment-2798747090

   For spark 2.4, has similar problem. It seems that Doris copied Spark's ArrowUtils codes but replaced the ArrowUtils.create() and 
   somewhere other method's parameter class with shaed class, causing the class conflict with Spark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org