You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "hiliuxg (via GitHub)" <gi...@apache.org> on 2023/04/27 09:12:48 UTC
[GitHub] [incubator-paimon] hiliuxg opened a new issue, #1044: [Bug] lost data when using spark load hive data to paimon
hiliuxg opened a new issue, #1044:
URL: https://github.com/apache/incubator-paimon/issues/1044
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
paimon-0.4-SNAPSHOT
### Compute Engine
paimon-spark-3.2-0.4-20230427.002025-43
### Minimal reproduce step
### create paimon table
```
CREATE TABLE paimon.fx_paimon.t_consume_info_f (
dt string,
consumeid string,
addtime string,
actionid string,
giftid string,
giftname string,
giftnum string,
coin string,
userid string,
senderkugouid string,
recuserid string,
reckugouid string,
isvmc string,
money string,
recbeannum string,
oldmoney string,
newmoney string,
oldcoin string,
newcoin string,
pid string,
senderroomid string,
senderdepartment string,
senderproductid string,
senderminorproductid string,
senderhardwareplatform string,
senderchannelid string,
sendersubchannelid string,
receiverdepartment string,
accountchangetype string,
price string,
createtime string,
updatetime string,
bizordertime string,
status string,
exchange string,
std_plat string,
deviceid string,
appversion string,
areacode string,
timezone string
) tblproperties (
'primary-key' = 'consumeid',
'write-mode' = 'change-log',
'bucket' = '100',
'bucket-key' = 'consumeid',
'snapshot.time-retained' = '1d',
'snapshot.num-retained.min' = '2',
'snapshot.num-retained.max' = '5'
)
```
### insert into paimon from hive table
```
insert into paimon.fx_paimon.t_consume_info_f
select
`dt` varchar,
ifnull(`consumeid`, 'other') varchar,
`addtime` varchar,
`actionid` varchar,
`giftid` varchar,
`giftname` varchar,
`giftnum` varchar,
`coin` varchar,
`userid` varchar,
`senderkugouid` varchar,
`recuserid` varchar,
`reckugouid` varchar,
`isvmc` varchar,
`money` varchar,
`recbeannum` varchar,
`oldmoney` varchar,
`newmoney` varchar,
`oldcoin` varchar,
`newcoin` varchar,
`pid` varchar,
`senderroomid` varchar,
`senderdepartment` varchar,
`senderproductid` varchar,
`senderminorproductid` varchar,
`senderhardwareplatform` varchar,
`senderchannelid` varchar,
`sendersubchannelid` varchar,
`receiverdepartment` varchar,
`accountchangetype` varchar,
`price` varchar,
`createtime` varchar,
`updatetime` varchar,
`bizordertime` varchar,
`status` varchar,
`exchange` varchar,
`std_plat` varchar,
`deviceid` varchar,
`appversion` varchar,
`areacode` varchar,
`timezone` varchar
from
fx_dcl.canal_t_consume_part
where dt >= '2023-01-31' and dt < '2023-04-27'
```
### select count data
```
select
dt ,
count(1) as cnt
from paimon.fx_paimon.t_consume_info_f
group by dt
order by dt
```
### What doesn't meet your expectations?
On January 31, there is one less data than the original hive table , and other days are the same .
### Anything else?
When I only load the day of January 31, the data on both sides are consistent
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] hiliuxg closed issue #1044: [Bug] lost data when using spark load hive data to paimon
Posted by "hiliuxg (via GitHub)" <gi...@apache.org>.
hiliuxg closed issue #1044: [Bug] lost data when using spark load hive data to paimon
URL: https://github.com/apache/incubator-paimon/issues/1044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] hiliuxg commented on issue #1044: [Bug] lost data when using spark load hive data to paimon
Posted by "hiliuxg (via GitHub)" <gi...@apache.org>.
hiliuxg commented on issue #1044:
URL: https://github.com/apache/incubator-paimon/issues/1044#issuecomment-1535988639
It's not that Paimon has a bug, it's that there's a problem with my test dataset, and I close it, sorry to bother.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org