You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "juanAmayaRamirez (via GitHub)" <gi...@apache.org> on 2023/05/03 22:11:06 UTC

[GitHub] [hudi] juanAmayaRamirez commented on issue #7879: [Bug] Hudi AWS Connector Throws Error on Hive Sync with Glue

juanAmayaRamirez commented on issue #7879:
URL: https://github.com/apache/hudi/issues/7879#issuecomment-1533818465

   Hi @soumilshah1995 just here to ask what the issue was.
   I am having a similar issue with lake formation that I can't get to figure out when trying to read a Hudi table from Data catalog. Can this be related? If not, do you have any suggestions?
   
   Glue config:
   Glue 4.0
   Job Parameters:
   --datalake-formats hudi
   --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false
   
   Code:
   ```
   spark = SparkSession.builder.config('spark.serializer','org.apache.spark.serializer.KryoSerializer')\
       .config('spark.sql.hive.convertMetastoreParquet', 'false')\
       .config("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")\
       .config("spark.sql.avro.datetimeRebaseModeInWrite", "CORRECTED")\
       .getOrCreate()
   
   glueContext = GlueContext(spark.sparkContext)
   job = Job(glueContext)
   job.init(args['JOB_NAME'], args)
   logger = glueContext.get_logger()
   
   dataFrame = glueContext.create_dynamic_frame_from_catalog(
       database = "my_db",
       table_name = "my_table"
   )
   ```
   Error:
   2023-05-03 21:18:58,045 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(77)): Error from Python:Traceback (most recent call last):
     File "/tmp/read hudi without connector.py", line 37, in <module>
       dataFrame = glueContext.create_dynamic_frame_from_catalog(
     File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 188, in create_dynamic_frame_from_catalog
       return source.getFrame(**kwargs)
     File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
       jframe = self._jsource.getDynamicFrame()
     File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
       return_value = get_return_value(
     File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
       return f(*a, **kw)
     File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
       raise Py4JJavaError(
   py4j.protocol.Py4JJavaError: An error occurred while calling o108.getDynamicFrame.
   : java.lang.UnsupportedOperationException: Reads and writes using Lake Formation permissions are not supported for hudi tables.
   	at com.amazonaws.services.glue.GlueUtility$.checkDataLakeFormatAndLakeFormation(GlueUtility.scala:54)
   	at com.amazonaws.services.glue.SparkSQLDataSource.getDynamicFrame(DataSource.scala:760)
   	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
   	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
   	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
   	at py4j.Gateway.invoke(Gateway.java:282)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
   	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
   	at java.lang.Thread.run(Thread.java:750)
   ![image](https://user-images.githubusercontent.com/97113713/236059664-8e4fc3ac-58ab-45a8-b6ba-8e648998e52f.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org