You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "xuzifu666 (via GitHub)" <gi...@apache.org> on 2024/03/17 05:05:46 UTC

[I] [SUPPORT] Insert overwrite with replacement instant cannot execute archive [hudi]

xuzifu666 opened a new issue, #10873:
URL: https://github.com/apache/hudi/issues/10873

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1. insert overwrite a table partition with 100times with replacement instant
2. timeline had not execute archived all the time
3. impact query table due to scan timeline instants

**Expected behavior**

execute archive in a regular time

**Environment Description**

* Hudi version : 0.14.0

* Spark version : 3.2.0

* Storage (HDFS/S3/GCS..) : HDFS

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Insert overwrite with replacement instant cannot execute archive [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.

ad1happy2go commented on issue #10873:
URL: https://github.com/apache/hudi/issues/10873#issuecomment-2002570828

   @xuzifu666 When I tried the below code, it was properly archiving. Can you check below or provide your table/writer configurations.
   
   ```
   fake = Faker()
   data = [{"transactionId": fake.uuid4(), "EventTime": "2014-01-01 23:00:01","storeNbr" : "1",
            "FullName": fake.name(), "Address": fake.address(),
            "CompanyName": fake.company(), "JobTitle": fake.job(),
            "EmailAddress": fake.email(), "PhoneNumber": fake.phone_number(),
            "RandomText": fake.sentence(), "City": "US",
            "State": "NYC", "Country": "US"} for _ in range(5)]
   
   hudi_options = {
       "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
       "hoodie.datasource.write.recordkey.field": "transactionId,storeNbr,EventTime",
       "hoodie.datasource.write.precombine.field": "Country",
       "hoodie.table.name": "huditransaction",
       "hoodie.datasource.write.operation" : "insert_overwrite",
       "hoodie.datasource.write.partitionpath.field" : "city"
   }
   
   pandas_df = pd.DataFrame(data)
   df = spark.createDataFrame(pandas_df).withColumn("EventTime", expr("cast(EventTime as timestamp)"))
   
   for i in range(1,100):
       (df.write.format("hudi").options(**hudi_options).mode("append").save(PATH))
   ```
   
   
   <img width="1694" alt="image" src="https://github.com/apache/hudi/assets/63430370/052419d1-2d9e-4d75-b588-fd1c887fee4f">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org