You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "young138120 (via GitHub)" <gi...@apache.org> on 2023/12/14 06:30:31 UTC
[I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
young138120 opened a new issue, #10327:
URL: https://github.com/apache/hudi/issues/10327
**Describe the problem you faced**
When I use Flink to write into Hudi and wait for the compaction to complete, I find that I can't retrieve data when querying a specific partition. However, the data is available when performing a full table query. Then, when I query the partition again, the data appears. What could be the cause of this? I have checked the commit and compaction in .hoodie, and everything seems normal, as well as the parquet files under the partition.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
0.9.0
* Spark version :
3.1.1
* Hive version :
3.1.0
* Hadoop version :
3.1.1
* Storage (HDFS/S3/GCS..) :
HDFS
* Running on Docker? (yes/no) :
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859503906
> so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the hudi table from Hive catalog).
I've tried this with the same result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 closed issue #10327: [SUPPORT] Flink to hudi ，but the partitioned table query is not expected
URL: https://github.com/apache/hudi/issues/10327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1857329342
what catalog did you use, is the meta sync successful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1862056072
Can you ensure the data set been successfully committed into the table, you can check the `.hoodie/` directory to see if there are completed metadata files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1857225825
> did you try `refresh table` cmd?
yeah， i already to try it ,but it isn't work for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1862411283
> Can you ensure the data set been successfully committed into the table, you can check the `.hoodie/` directory to see if there are completed metadata files.
I am sure and tried again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859809742
> > so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the hudi table from Hive catalog).
>
> can you give me a sample that sync meta to hive catalog , i try to add configration options like this ,and get the exception  
>
> I can't get information from the Internet about the exception
I forget copy hive-site.xml to resource fold , it's my fault
I run the flink job on test env now ,and try to query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1857373373
> what catalog did you use, is the meta sync successful?
I use hudi-flink1.15-bundle-0.14.0 version jar
and init sink like this
I find the source code that all of the configration is default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1863802200
> how you defines the table in spark sql?
what's the mean defines the table ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1857216748
did you try `refresh table` cmd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859507966
> so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the hudi table from Hive catalog).
can you give me a sample that sync meta to hive catalog , i try to add configration options like this ,and get the exception
I can't get information from the Internet about the exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1863801300
how you defines the table in spark sql?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1855669949
Not sure whether it is because Spark session caches the partition info of the table, did you try other Hudi releases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859860650
> > > so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the hudi table from Hive catalog).
> >
> >
> > can you give me a sample that sync meta to hive catalog , i try to add configration options like this ,and get the exception  
> > I can't get information from the Internet about the exception
>
> I forget copy hive-site.xml to resource fold , it's my fault I run the flink job on test env now ,and try to query
Still can't query result~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859489273
so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the hudi table from Hive catalog).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1857089655
> Not sure whether it is because Spark session caches the partition info of the table, did you try other Hudi releases?
spark session is newly created，so it has not cache .
I can not upgrade the pro env version , can i force refresh the newly info to query ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink to hudi ，but the partitioned table query is not expected [hudi]
Posted by "young138120 (via GitHub)" <gi...@apache.org>.
young138120 commented on issue #10327:
URL: https://github.com/apache/hudi/issues/10327#issuecomment-1870738635
It should be caused by different versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org