You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bhavya-ganatra (via GitHub)" <gi...@apache.org> on 2026/02/09 09:49:46 UTC
[I] Spark catalog refresh uses default database's table rather than provided database's table [hudi]
bhavya-ganatra opened a new issue, #18139:
URL: https://github.com/apache/hudi/issues/18139
### Bug Description
**What happened:**
I am using Hudi with AWS Glue sync. I am using below AWS Glue sync related configurations:
`
options.put("hoodie.datasource.hive_sync.enable", "true");
options.put("hoodie.datasource.hive_sync.mode", "hms");
options.put("hoodie.datasource.hive_sync.database", glueDatabase);
options.put("hoodie.datasource.hive_sync.table", tableName);
options.put("hoodie.datasource.hive_sync.partition_fields", getPartitionKeyFieldName(config));
options.put("hoodie.datasource.hive_sync.partition_extractor_class",
"org.apache.hudi.hive.MultiPartKeysValueExtractor");
options.put("hoodie.datasource.hive_sync.use_jdbc", "false");
options.put("hoodie.datasource.meta.sync.enable", "true");
options.put("hoodie.datasource.meta_sync.condition.sync", "true");
options.put("hoodie.meta.sync.client.tool.class", "org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool");
options.put("hoodie.datasource.hive_sync.auto_create_database", "true");
options.put("hoodie.datasource.hive_sync.support_timestamp", "true");
options.put("hoodie.datasource.hive_sync.create_managed_table", "false");
options.put("hoodie.datasource.hive_sync.skip_ro_suffix", "false");
`
I am passing glueDatabase = `staging_lh_dynamodb_tenant_123` and table: `content` which correspondence to s3 path: `s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content`
There is also another database in Glue, name: "default", table: `content` which correspondence to s3 path:
`s3a://bucket-perf-scale/lakehouse/dd/data/51/content`.
When I had ingested few upserts via my spark job, Hudi write was successful, on bucket-staging. But, after that when it tried to sync with AWS Glue, it was using "default" database's content table's s3 path: bucket-perf-scale.
Since, we have provided IAM roles such that service writing to one bucket doesn't not have access to any other buckets, hence it was throwing expcetion related to permission issue, which is expected.
But, I wanted to know why it was trying to use default database's content table. Even though I had passed database name with `hoodie.datasource.hive_sync.database` property.
Based on stacktrace, I found below code block in spark library, it seems to be using only tableName, so that might use "default" database. But, I am not sure if this is exact culpirt, i might be missing some configuration.
`
Potential cause -
// Spark library function - passes tableName only
public void refreshTable(final String tableName) {
LogicalPlan relation = this.org$apache$spark$sql$internal$CatalogImpl$$sparkSession.table(tableName).queryExecution().analyzed();
relation.refresh();
this.invalidateCache$1(relation);
this.org$apache$spark$sql$internal$CatalogImpl$$sparkSession.sharedState().cacheManager().recacheByPlan(this.org$apache$spark$sql$internal$CatalogImpl$$sparkSession, relation);
}
`
Note that: My writer is writing to multiple different s3 locations/tables. So, it is not possible to set database name as part of spark level configurations. I need some dynamic config setting which are applicable only during writes.
**What you expected:**
If database name is passed as hoodie.datasource.hive_sync.database, it shouldn't use default database. It should refer to this property.
**Steps to reproduce:**
1. Enable Inline Glue sync in writer configuration
2. Ingest some data so glue database gets created automatically.
3. Create table with the same name, ex. content, under default database.
4. Ingest few more data so Glue sync can get triggered.
5. Validate that write is successful, but glue sync might fail due to the same table under default database.
[Note, table can be created earlier as well. i.e. step 2 can be avoided]
Note that, this Glue table - staging_lh_dynamodb_tenant_123 was created automatically by Glue itself. "default" database -> content table - this was created by some manual work.
### Environment
**Hudi version: 1.1.0**
**Query engine: Spark 3.5.6** (Spark/Flink/Trino etc)
**Relevant configs: **
Below are all the hudi options provided during write:
`Hudi options specified: {hoodie.write.lock.dynamodb.region=us-west-2, hoodie.metrics.pushgateway.report.labels=cluster:staging,namespace:lh-dynamodb, hoodie.datasource.hive_sync.partition_fields=, hoodie.index.type=BUCKET, hoodie.clean.automatic=false, hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider, hoodie.compact.inline=false, hoodie.datasource.write.recordkey.field=cid, hoodie.datasource.hive_sync.create_managed_table=false, hoodie.metadata.enable=true, hoodie.datasource.write.table.type=MERGE_ON_READ, hoodie.datasource.write.keygenerator.type=NON_PARTITION, hoodie.parquet.small.file.limit=134217728, hoodie.write.lock.dynamodb.partition_key=content-123, hoodie.cleaner.commits.retained=2, hoodie.datasource.meta.sync.enable=true, hoodie.write.lock.dynamodb.table=hudi-lock-staging-lh-dynamodb-table, hoodie.write.lock.wait_time_ms=60000, hoodie.memory.spillable.map.path=/tmp/, hoodie.write.record.merge.mode=CUSTOM, hoodie.table.cdc.enabled
=true, hoodie.datasource.hive_sync.use_jdbc=false, hoodie.delete.shuffle.parallelism=4, hoodie.table.name=content, hoodie.table.log.file.format=parquet, hoodie.upsert.shuffle.parallelism=4, hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool, hoodie.datasource.hive_sync.skip_ro_suffix=false, hoodie.datasource.write.precombine.field=processing_timestamp, hoodie.insert.shuffle.parallelism=4, hoodie.cleaner.policy.failed.writes=LAZY, hoodie.datasource.compaction.async.enable=false, hoodie.datasource.write.operation=upsert, hoodie.logfile.data.block.format=parquet, hoodie.datasource.hive_sync.table=content, hoodie.index.bucket.engine=SIMPLE, hoodie.datasource.meta_sync.condition.sync=true, hoodie.metadata.index.column.stats.enable=false, hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.MultiPartKeysValueExtractor, hoodie.write.record.merge.custom.implementation.classes=com.xyz.lhcommons.hudi.merger.CustomHoodieSparkRecordMerger, h
oodie.datasource.hive_sync.auto_create_database=true, hoodie.datasource.hive_sync.mode=hms, hoodie.parquet.max.file.size=268435456, hoodie.datasource.hive_sync.support_timestamp=true, hoodie.datasource.write.hive_style_partitioning=true, hoodie.datasource.hive_sync.enable=true, hoodie.bulkinsert.shuffle.parallelism=4, hoodie.cleaner.policy=KEEP_LATEST_COMMITS, hoodie.metrics.pushgateway.job.name=lhwriter, hoodie.write.lock.dynamodb.endpoint_url=dynamodb.us-west-2.amazonaws.com, hoodie.table.cdc.supplemental.logging=data_before_after, hoodie.write.concurrency.mode=optimistic_concurrency_control, hoodie.memory.merge.fraction=0.2, hoodie.datasource.write.partitionpath.field=, hoodie.bucket.index.num.buckets=17, hoodie.write.record.merge.strategy.id=749c702e-ffa2-43da-ab2e-9c43d5080754, hoodie.datasource.hive_sync.database=staging_lh_dynamodb_tenant_123}`
### Logs and Stack Trace
Below is full exception trace and few relevant logs:
`
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Last commit time synced was found to be 20260204194748073, last commit completion time is found to be 20260204194759976
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Sync all partitions given the last commit time synced is empty or before the start of the active timeline. Listing all partitions in s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content, file system: S3AFileSystem{uri=s3a://bucket-staging, workingDir=s3a://bucket-staging/user/hadoop, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, performanceFlags={}, OpenFileSupport{changePolicy=ETagChangeDetectionPolicy mode=Server, defaultReadAhead=65536, defaultBufferSize=65536, defaultAsyncDrainThreshold=16000, defaultInputPolicy=default}, blockSize=33554432, multiPartThreshold=134217728, s3EncryptionAlgorithm='NONE', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@71dfb684, auditManager=Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='424ad7
b7-e6cb-408b-9eaf-ecfd7b7d8caa', headerEnabled=true, rejectOutOfSpan=false, isMultipartUploadEnabled=true}}, authoritativePath=[], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=224, available=224, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@70038363[Running, pool size = 59, active threads = 0, queued tasks = 0, completed tasks = 129], credentials=AWSCredentialProviderList name=; refcount= 1; size=1: [DefaultCredentialsProvider(providerChain=LazyAwsCredentialsProvider(delegate=Lazy(value=AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]))))] last provider: DefaultCredentialsProvider(providerChain=LazyAwsCredentialsProvider(deleg
ate=Lazy(value=AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()])))), delegation tokens=disabled, DirectoryMarkerRetention{policy='keep'}, instrumentation {S3AInstrumentation{}}, ClientSideEncryption=false}
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Sync complete for content_ro
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Trying to sync hoodie table content_rt with base path s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content of type MERGE_ON_READ
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - No Schema difference for content_rt.
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Last commit time synced was found to be 20260204194748073, last commit completion time is found to be 20260204194759976
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Sync all partitions given the last commit time synced is empty or before the start of the active timeline. Listing all partitions in s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content, file system: S3AFileSystem{uri=s3a://bucket-staging, workingDir=s3a://bucket-staging/user/hadoop, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, performanceFlags={}, OpenFileSupport{changePolicy=ETagChangeDetectionPolicy mode=Server, defaultReadAhead=65536, defaultBufferSize=65536, defaultAsyncDrainThreshold=16000, defaultInputPolicy=default}, blockSize=33554432, multiPartThreshold=134217728, s3EncryptionAlgorithm='NONE', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@71dfb684, auditManager=Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='424ad7
b7-e6cb-408b-9eaf-ecfd7b7d8caa', headerEnabled=true, rejectOutOfSpan=false, isMultipartUploadEnabled=true}}, authoritativePath=[], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=224, available=224, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@70038363[Running, pool size = 59, active threads = 0, queued tasks = 0, completed tasks = 129], credentials=AWSCredentialProviderList name=; refcount= 1; size=1: [DefaultCredentialsProvider(providerChain=LazyAwsCredentialsProvider(delegate=Lazy(value=AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]))))] last provider: DefaultCredentialsProvider(providerChain=LazyAwsCredentialsProvider(deleg
ate=Lazy(value=AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()])))), delegation tokens=disabled, DirectoryMarkerRetention{policy='keep'}, instrumentation {S3AInstrumentation{}}, ClientSideEncryption=false}
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Sync complete for content_rt
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Trying to sync hoodie table content with base path s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content of type MERGE_ON_READ
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - No Schema difference for content.
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Last commit time synced was found to be 20260204194748073, last commit completion time is found to be 20260204194759976
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Sync all partitions given the last commit time synced is empty or before the start of the active timeline. Listing all partitions in s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content, file system: S3AFileSystem{uri=s3a://bucket-staging, workingDir=s3a://bucket-staging/user/hadoop, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, performanceFlags={}, OpenFileSupport{changePolicy=ETagChangeDetectionPolicy mode=Server, defaultReadAhead=65536, defaultBufferSize=65536, defaultAsyncDrainThreshold=16000, defaultInputPolicy=default}, blockSize=33554432, multiPartThreshold=134217728, s3EncryptionAlgorithm='NONE', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@71dfb684, auditManager=Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='424ad7
b7-e6cb-408b-9eaf-ecfd7b7d8caa', headerEnabled=true, rejectOutOfSpan=false, isMultipartUploadEnabled=true}}, authoritativePath=[], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=224, available=224, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@70038363[Running, pool size = 59, active threads = 0, queued tasks = 0, completed tasks = 129], credentials=AWSCredentialProviderList name=; refcount= 1; size=1: [DefaultCredentialsProvider(providerChain=LazyAwsCredentialsProvider(delegate=Lazy(value=AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]))))] last provider: DefaultCredentialsProvider(providerChain=LazyAwsCredentialsProvider(deleg
ate=Lazy(value=AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()])))), delegation tokens=disabled, DirectoryMarkerRetention{policy='keep'}, instrumentation {S3AInstrumentation{}}, ClientSideEncryption=false}
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.hive.HiveSyncTool - Sync complete for content
2026-02-05 07:19:39,959 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:79) [] - Getting current Glue AuditContext
2026-02-05 07:19:39,959 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:111) [] - No AuditContext is present
2026-02-05 07:19:40,117 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:79) [] - Getting current Glue AuditContext
2026-02-05 07:19:40,118 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:111) [] - No AuditContext is present
2026-02-05 07:19:40,178 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:79) [] - Getting current Glue AuditContext
2026-02-05 07:19:40,184 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:111) [] - No AuditContext is present
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.DataSourceUtils - Getting table path..
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.common.util.TablePathUtils - Getting table path from path : s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.DefaultSource - Obtained hudi table path: s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from s3a://bucket-staging/lakehouse/lh-dynamodb/data/123/content/.hoodie/hoodie.properties
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.DefaultSource - Is bootstrapped table => false, tableType is: MERGE_ON_READ, queryType is: read_optimized
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Closing Log file reader .00000000-1248-4c23-88cb-b647eac38862-0_20260205071918459.log.1_0-62-479
2026-02-05 07:19:40,685 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:79) [] - Getting current Glue AuditContext
2026-02-05 07:19:40,686 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:111) [] - No AuditContext is present
2026-02-05 07:19:40,780 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:79) [] - Getting current Glue AuditContext
2026-02-05 07:19:40,782 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:111) [] - No AuditContext is present
2026-02-05 07:19:40,835 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:79) [] - Getting current Glue AuditContext
2026-02-05 07:19:40,837 INFO [stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] (AuditContextUtil.java:111) [] - No AuditContext is present
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.HoodieSparkSqlWriterInternal - Config.inlineCompactionEnabled ? false
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.HoodieSparkSqlWriterInternal - Config.asyncClusteringEnabled ? false
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.client.BaseHoodieClient - Stopping Timeline service !!
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.common.util.collection.ExternalSpillableMap - KeyBasedFileGroupRecordBuffer : Total entries in InMemory map 2, with average record size as 3672, currentInMemoryMapSize 7344. No entries were spilled to disk.
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.client.embedded.EmbeddedTimelineService - Closing Timeline server
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.timeline.service.TimelineService - Closing Timeline Service with port 33855
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO io.javalin.Javalin - Stopping Javalin ...
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO io.javalin.Javalin - Javalin has stopped
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.timeline.service.TimelineService - Closed Timeline Service with port 33855
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.client.embedded.EmbeddedTimelineService - Closed Timeline server
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.client.transaction.TransactionManager - Transaction manager closed
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.client.transaction.TransactionManager - Transaction manager closed
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.hudi.metrics.Metrics - Stopping the metrics reporter...
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] INFO org.apache.spark.sql.execution.SQLExecution - Skipped SparkListenerSQLExecutionObfuscatedInfo event due to NON_EMPTY_ERROR.
[stream execution thread for [id = 82a0bf9b-64bc-45c4-9072-35f2e5acd3ff, runId = 50dc2060-a949-4d8e-bbce-a6794b24f247]] ERROR com.xyz.lhwriter.writer.HudiDataWriter - Error writing to Hudi for tenant 123 entity content: java.nio.file.AccessDeniedException: s3a://bucket-perf-scale/lakehouse/dd/data/51/content/.hoodie: getFileStatus on s3a://bucket-perf-scale/lakehouse/dd/data/51/content/.hoodie: software.amazon.awssdk.services.s3.model.S3Exception: User: arn:aws:sts::744500168534:assumed-role/staging_emr_irsa/aws-sdk-java-1770275951000 is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::bucket-perf-scale" because no identity-based policy allows the s3:ListBucket action (Service: S3, Status Code: 403, Request ID: 98W8TAS2YDPVBCER, Extended Request ID: IoNmWcgGoL36lqev2rjxSeNYnjtxB/zYbmkX04Mzraco9JXGqMzfflLr+uwBPSizRS94YbOzT6I=) (SDK Attempt Count: 1):AccessDenied
java.util.concurrent.ExecutionException: java.nio.file.AccessDeniedException: s3a://bucket-perf-scale/lakehouse/dd/data/51/content/.hoodie: getFileStatus on s3a://bucket-perf-scale/lakehouse/dd/data/51/content/.hoodie: software.amazon.awssdk.services.s3.model.S3Exception: User: arn:aws:sts::744500168534:assumed-role/staging_emr_irsa/aws-sdk-java-1770275951000 is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::bucket-perf-scale" because no identity-based policy allows the s3:ListBucket action (Service: S3, Status Code: 403, Request ID: 98W8TAS2YDPVBCER, Extended Request ID: IoNmWcgGoL36lqev2rjxSeNYnjtxB/zYbmkX04Mzraco9JXGqMzfflLr+uwBPSizRS94YbOzT6I=) (SDK Attempt Count: 1):AccessDenied
at org.sparkproject.guava.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
at org.sparkproject.guava.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)
at org.sparkproject.guava.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at org.sparkproject.guava.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
at org.sparkproject.guava.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)
at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)
at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2257)
at org.sparkproject.guava.cache.LocalCache.get(LocalCache.java:4000)
at org.sparkproject.guava.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getCachedPlan(SessionCatalog.scala:215)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.org$apache$spark$sql$execution$datasources$FindDataSourceTable$$readDataSourceTable(DataSourceStrategy.scala:277)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable$$anonfun$apply$2.applyOrElse(DataSourceStrategy.scala:353)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable$$anonfun$apply$2.applyOrElse(DataSourceStrategy.scala:330)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:205)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:77)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:205)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:359)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:203)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:199)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:39)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$4(AnalysisHelper.scala:210)
at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1422)
at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1421)
at org.apache.spark.sql.catalyst.plans.logical.SubqueryAlias.mapChildren(basicLogicalOperators.scala:2369)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:210)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:359)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:203)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:199)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:39)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsWithPruning(AnalysisHelper.scala:134)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsWithPruning$(AnalysisHelper.scala:131)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsWithPruning(LogicalPlan.scala:39)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators(AnalysisHelper.scala:85)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators$(AnalysisHelper.scala:84)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:39)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.apply(DataSourceStrategy.scala:330)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.apply(DataSourceStrategy.scala:248)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:232)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeBatch$1(RuleExecutor.scala:229)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$6(RuleExecutor.scala:312)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$RuleExecutionContext$.withContext(RuleExecutor.scala:361)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$5(RuleExecutor.scala:312)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$5$adapted(RuleExecutor.scala:302)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:302)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:188)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:184)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeSameContext(Analyzer.scala:302)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:298)
at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:231)
at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:298)
at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:260)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:175)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:175)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:285)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:366)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:284)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:227)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:293)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:748)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:293)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:996)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:292)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:91)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:83)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:93)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:996)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:91)
at org.apache.spark.sql.DataFrameReader.table(DataFrameReader.scala:688)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:603)
at org.apache.spark.sql.internal.CatalogImpl.refreshTable(CatalogImpl.scala:846)
at org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$metaSync$3(HoodieSparkSqlWriter.scala:938)
at org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$metaSync$3$adapted(HoodieSparkSqlWriter.scala:934)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.hudi.HoodieSparkSqlWriterInternal.metaSync(HoodieSparkSqlWriter.scala:934)
at org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1020)
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:548)
at org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$write$1(HoodieSparkSqlWriter.scala:193)
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:211)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:133)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:171)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:127)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:391)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:159)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$11(SQLExecution.scala:227)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:391)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$10(SQLExecution.scala:227)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:412)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:226)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:996)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:84)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:124)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:115)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:521)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:77)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:521)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:39)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:303)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:299)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:39)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:39)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:497)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:115)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:102)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:165)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:884)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:405)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:365)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:244)
at com.xyz.lhwriter.writer.HudiDataWriter.executeHudiWrite(HudiDataWriter.java:161)
at com.xyz.lhwriter.writer.HudiDataWriter.writeToHudi(HudiDataWriter.java:75)
at com.xyz.lhwriter.writer.HudiWriteExecutor.attemptWrite(HudiWriteExecutor.java:73)
at com.xyz.lhwriter.writer.HudiWriteExecutor.writeWithRetry(HudiWriteExecutor.java:47)
at com.xyz.lhwriter.writer.SequentialGroupProcessor.processSingleGroup(SequentialGroupProcessor.java:61)
at com.xyz.lhwriter.writer.SequentialGroupProcessor.processGroups(SequentialGroupProcessor.java:36)
at com.xyz.lhwriter.writer.HudiWriter.processEntityDataset(HudiWriter.java:394)
at com.xyz.lhwriter.writer.HudiWriter.lambda$processEntityMapBatch$0(HudiWriter.java:287)
at com.xyz.lhwriter.writer.EntityParallelProcessor.processEntitiesSequentially(EntityParallelProcessor.java:117)
at com.xyz.lhwriter.writer.EntityParallelProcessor.processEntities(EntityParallelProcessor.java:92)
at com.xyz.lhwriter.writer.HudiWriter.processEntityMapBatch(HudiWriter.java:277)
at com.xyz.lhwriter.writer.HudiWriter.lambda$writeAllEntitiesToHudi$58d5af8e$1(HudiWriter.java:170)
at org.apache.spark.sql.streaming.DataStreamWriter.$anonfun$foreachBatch$1(DataStreamWriter.scala:505)
at org.apache.spark.sql.streaming.DataStreamWriter.$anonfun$foreachBatch$1$adapted(DataStreamWriter.scala:505)
at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:38)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:732)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:391)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:159)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$11(SQLExecution.scala:227)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:391)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$10(SQLExecution.scala:227)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:412)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:226)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:996)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:84)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:729)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:427)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:425)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:729)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:286)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:427)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:425)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:249)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:67)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:239)
at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:311)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:996)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:289)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$1(StreamExecution.scala:211)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:211)
Caused by: java.nio.file.AccessDeniedException: s3a://bucket-perf-scale/lakehouse/dd/data/51/content/.hoodie: getFileStatus on s3a://bucket-perf-scale/lakehouse/dd/data/51/content/.hoodie: software.amazon.awssdk.services.s3.model.S3Exception: User: arn:aws:sts::744500168534:assumed-role/staging_emr_irsa/aws-sdk-java-1770275951000 is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::bucket-perf-scale" because no identity-based policy allows the s3:ListBucket action (Service: S3, Status Code: 403, Request ID: 98W8TAS2YDPVBCER, Extended Request ID: IoNmWcgGoL36lqev2rjxSeNYnjtxB/zYbmkX04Mzraco9JXGqMzfflLr+uwBPSizRS94YbOzT6I=) (SDK Attempt Count: 1):AccessDenied
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:275)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:162)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4541)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:4403)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:5403)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:3274)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:3293)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:5401)
at org.apache.hudi.storage.hadoop.HoodieHadoopStorage.exists(HoodieHadoopStorage.java:165)
at org.apache.spark.sql.hudi.HoodieSqlCommonUtils$.tableExistsInPath(HoodieSqlCommonUtils.scala:222)
at org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable.<init>(HoodieCatalogTable.scala:88)
at org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable$.apply(HoodieCatalogTable.scala:403)
at org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable$.apply(HoodieCatalogTable.scala:399)
at org.apache.hudi.HoodieFileIndex$.getConfigProperties(HoodieFileIndex.scala:540)
at org.apache.hudi.HoodieFileIndex.<init>(HoodieFileIndex.scala:99)
at org.apache.hudi.HoodieCopyOnWriteSnapshotHadoopFsRelationFactory.<init>(HoodieHadoopFsRelationFactory.scala:371)
at org.apache.hudi.DefaultSource$.createRelation(DefaultSource.scala:341)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:144)
at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$resolveRelation$6(DataSource.scala:369)
at org.apache.spark.util.FileAccessContext$.withContext(FileAccessContext.scala:71)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:366)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:348)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable$$anon$1.call(DataSourceStrategy.scala:268)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable$$anon$1.call(DataSourceStrategy.scala:255)
at org.sparkproject.guava.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)
at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
... 168 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: User: arn:aws:sts::744500168534:assumed-role/staging_emr_irsa/aws-sdk-java-1770275951000 is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::bucket-perf-scale" because no identity-based policy allows the s3:ListBucket action (Service: S3, Status Code: 403, Request ID: 98W8TAS2YDPVBCER, Extended Request ID: IoNmWcgGoL36lqev2rjxSeNYnjtxB/zYbmkX04Mzraco9JXGqMzfflLr+uwBPSizRS94YbOzT6I=) (SDK Attempt Count: 1)
at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:8724)
at software.amazon.awssdk.services.s3.DelegatingS3Client.lambda$listObjectsV2$69(DelegatingS3Client.java:7076)
at software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient.invokeOperation(S3CrossRegionSyncClient.java:74)
at software.amazon.awssdk.services.s3.DelegatingS3Client.listObjectsV2(DelegatingS3Client.java:7076)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$13(S3AFileSystem.java:3497)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:3488)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4516)
... 195 more
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org