You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Karthick Nk <kc...@gmail.com> on 2023/05/10 18:01:20 UTC
Error while merge in delta table
Hi,
I am trying to merge daaframe with delta table in databricks, but i am
getting error, i have attached the code nippet and error message for
reference below,
code:
[image: image.png]
error:
[image: image.png]
Thanks
Re: Error while merge in delta table
Posted by Farhan Misarwala <fa...@gmail.com>.
Hi Karthick,
If you have confirmed that the incompatibility between Delta and spark
versions is not the case, then I would say the same what Jacek said
earlier, there’s not enough “data” here.
To further comment on it, we would need to know more on how you are
structuring your multi threaded PySpark app which writes in parallel.
It would help if you can reproduce this and give us a shareable code
snippet here.
All the best,
Farhan.
On Fri, May 12, 2023 at 10:17 AM Karthick Nk <kc...@gmail.com> wrote:
> Hi Farhan,
> Thank you for your response, I am using databricks with 11.3x-scala2.12.
>
> Here I am overwriting all the tables in the same database in concurrent
> thread, But when I do in the iterative manner it is working fine,
> For Example, i am having 200 tables in same database, i am overwriting the
> tables in a concurrent manner, are this is the issue(so we have any
> constraint for it)
>
> For this kind of run time how we can usually identify the root cause of
> it?????
>
>
> On Thu, May 11, 2023 at 9:37 PM Farhan Misarwala <
> farhan.misarwala@gmail.com> wrote:
>
>> Hi Karthick,
>>
>> I think I have seen this before and this probably could be because of an
>> incompatibility between your spark and delta versions.
>>
>> Or an incompatibility between the delta version you are using now vs the
>> one you used earlier on the existing table you are merging with.
>>
>> Let me know if this is the case.
>>
>> Thanks,
>> Farhan.
>>
>> On Thu, May 11, 2023 at 2:54 PM Jacek Laskowski <ja...@japila.pl> wrote:
>>
>>> Hi Karthick,
>>>
>>> Sorry to say it but there's not enough "data" to help you. There should
>>> be something more above or below this exception snippet you posted that
>>> could pinpoint the root cause.
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> "The Internals Of" Online Books <https://books.japila.pl/>
>>> Follow me on https://twitter.com/jaceklaskowski
>>>
>>> <https://twitter.com/jaceklaskowski>
>>>
>>>
>>> On Wed, May 10, 2023 at 8:03 PM Karthick Nk <kc...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to merge daaframe with delta table in databricks, but i am
>>>> getting error, i have attached the code nippet and error message for
>>>> reference below,
>>>>
>>>> code:
>>>> [image: image.png]
>>>>
>>>> error:
>>>>
>>>> [image: image.png]
>>>>
>>>> Thanks
>>>>
>>>
Re: Error while merge in delta table
Posted by Karthick Nk <kc...@gmail.com>.
Hi Farhan,
Thank you for your response, I am using databricks with 11.3x-scala2.12.
Here I am overwriting all the tables in the same database in concurrent
thread, But when I do in the iterative manner it is working fine,
For Example, i am having 200 tables in same database, i am overwriting the
tables in a concurrent manner, are this is the issue(so we have any
constraint for it)
For this kind of run time how we can usually identify the root cause of
it?????
On Thu, May 11, 2023 at 9:37 PM Farhan Misarwala <fa...@gmail.com>
wrote:
> Hi Karthick,
>
> I think I have seen this before and this probably could be because of an
> incompatibility between your spark and delta versions.
>
> Or an incompatibility between the delta version you are using now vs the
> one you used earlier on the existing table you are merging with.
>
> Let me know if this is the case.
>
> Thanks,
> Farhan.
>
> On Thu, May 11, 2023 at 2:54 PM Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi Karthick,
>>
>> Sorry to say it but there's not enough "data" to help you. There should
>> be something more above or below this exception snippet you posted that
>> could pinpoint the root cause.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> "The Internals Of" Online Books <https://books.japila.pl/>
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> <https://twitter.com/jaceklaskowski>
>>
>>
>> On Wed, May 10, 2023 at 8:03 PM Karthick Nk <kc...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to merge daaframe with delta table in databricks, but i am
>>> getting error, i have attached the code nippet and error message for
>>> reference below,
>>>
>>> code:
>>> [image: image.png]
>>>
>>> error:
>>>
>>> [image: image.png]
>>>
>>> Thanks
>>>
>>
Re: Error while merge in delta table
Posted by Farhan Misarwala <fa...@gmail.com>.
Hi Karthick,
I think I have seen this before and this probably could be because of an
incompatibility between your spark and delta versions.
Or an incompatibility between the delta version you are using now vs the
one you used earlier on the existing table you are merging with.
Let me know if this is the case.
Thanks,
Farhan.
On Thu, May 11, 2023 at 2:54 PM Jacek Laskowski <ja...@japila.pl> wrote:
> Hi Karthick,
>
> Sorry to say it but there's not enough "data" to help you. There should be
> something more above or below this exception snippet you posted that could
> pinpoint the root cause.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
> <https://twitter.com/jaceklaskowski>
>
>
> On Wed, May 10, 2023 at 8:03 PM Karthick Nk <kc...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to merge daaframe with delta table in databricks, but i am
>> getting error, i have attached the code nippet and error message for
>> reference below,
>>
>> code:
>> [image: image.png]
>>
>> error:
>>
>> [image: image.png]
>>
>> Thanks
>>
>
Re: Error while merge in delta table
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Karthick,
Sorry to say it but there's not enough "data" to help you. There should be
something more above or below this exception snippet you posted that could
pinpoint the root cause.
Pozdrawiam,
Jacek Laskowski
----
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
On Wed, May 10, 2023 at 8:03 PM Karthick Nk <kc...@gmail.com> wrote:
> Hi,
>
> I am trying to merge daaframe with delta table in databricks, but i am
> getting error, i have attached the code nippet and error message for
> reference below,
>
> code:
> [image: image.png]
>
> error:
>
> [image: image.png]
>
> Thanks
>