You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2024/11/17 22:07:10 UTC

[PR] limits tablets and offers bulk import as option for ingest [accumulo-testing]

keith-turner opened a new pull request, #287:
URL: https://github.com/apache/accumulo-testing/pull/287

   Two new continuous ingest features are introduced in this change. First options were added to limit the number of tablets written.  Second an options to use bulk ingest was added instead of a batch writer.
   
   These features support running a test like the following.
   
    * create a continuous ingest table with 1000 tablets
    * start 100 continuous ingest clients
    * have each client continually bulk import data to 10 random tablets
   
   This test situation will create a lot of bulk import and subsequent compaction activity for Accumulo to handle.
   
   These changes add bulk import to the `cingest ingest` command.  There is an existing `cingest bulk` command that runs a map reduce job to create bulk files.  These changes do not remove the need for the existing map reduce job, they fill a different purpose.  The map reduce job can generate really large amount of data to bulk import.  These changes allow generating lots of bulk imports w/ small amounts of data. These changes could never generate the amount of data for a single bulk import that the map reduce job could. The following is an example of test scenario that could use both.
   
    * create a continuous ingest table with 1000 tablets
    * use map reduce bulk job to create an initial 10 billion entries in the table
    * start 100 continuous ingest clients
    * have each client continually bulk import data to 10 random tablets
    * stop clients after 12 hours and verify data


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] limits tablets and offers bulk import as option for ingest [accumulo-testing]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #287:
URL: https://github.com/apache/accumulo-testing/pull/287#issuecomment-2481629433

   The following is example output that shows running these changes where it continually bulk imports into 3 random tablets on a table with 20 tablets.  Around every 6s it bulk imports 3 files to 3 tablets w/ one million total key values.
   
   ```
   $ ./bin/cingest ingest -o test.ci.ingest.bulk.workdir=hdfs://localhost:8020/ci_bulk -o test.ci.ingest.max.tablets=3
   2024-11-17T22:11:52,292 [testing.continuous.ContinuousIngest] INFO : Ingest instance ID: 655f7838-dde9-4c28-8c6f-a05211d5c4e1 current time: 1731881512290ms
   2024-11-17T22:11:52,293 [testing.continuous.ContinuousIngest] INFO : A flush will occur after every 1000000 entries written
   2024-11-17T22:11:52,305 [testing.continuous.ContinuousIngest] INFO : Total entries to be written: 9223372036854775807
   2024-11-17T22:11:52,307 [testing.continuous.ContinuousIngest] INFO : DELETES will occur with a probability of 0.10
   2024-11-17T22:11:54,385 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 0cccccccccccccce 1333333333333335
   2024-11-17T22:11:55,480 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 6cccccccccccccd7 733333333333333e
   2024-11-17T22:11:56,367 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 733333333333333e 79999999999999a5
   2024-11-17T22:11:58,049 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/5455e07d-ee97-4fcd-b526-65d75a31697c destinations:3 mutations:1000000 memUsed:331000000 time:791ms
   2024-11-17T22:11:58,054 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4410ms, since last flush: 5749ms, total written: 1000000, total deleted: 0
   2024-11-17T22:11:59,862 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 399999999999999f 4000000000000006
   2024-11-17T22:12:00,930 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 466666666666666d 4cccccccccccccd4
   2024-11-17T22:12:01,976 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 6000000000000009 666666666666667
   2024-11-17T22:12:03,742 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/05d3d626-265e-4b01-a1bd-6053d108cca6 destinations:3 mutations:1000000 memUsed:347000000 time:734ms
   2024-11-17T22:12:03,744 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4515ms, since last flush: 5690ms, total written: 2000000, total deleted: 0
   2024-11-17T22:12:05,524 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 199999999999999c 2000000000000003
   2024-11-17T22:12:06,573 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 4000000000000006 466666666666666d
   2024-11-17T22:12:07,602 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 59999999999999a2 6000000000000009
   2024-11-17T22:12:09,342 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/eee6174e-f539-4a81-a28f-84a0dd8d9932 destinations:3 mutations:1000000 memUsed:347000000 time:719ms
   2024-11-17T22:12:09,344 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4427ms, since last flush: 5600ms, total written: 3000000, total deleted: 0
   2024-11-17T22:12:11,094 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 2000000000000003 266666666666666a
   2024-11-17T22:12:12,168 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 4000000000000006 466666666666666d
   2024-11-17T22:12:13,186 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 59999999999999a2 6000000000000009
   2024-11-17T22:12:14,924 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/4cf6970c-fed7-4f51-bb1c-03274ae05c48 destinations:3 mutations:1000000 memUsed:347000000 time:715ms
   2024-11-17T22:12:14,926 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4405ms, since last flush: 5582ms, total written: 4000000, total deleted: 0accu
   ```
   
   
   I have run the bulk changes and the live ingest into the same table and then successfully ran the verify map reduce job.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] limits tablets and offers bulk import as option for ingest [accumulo-testing]

Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr merged PR #287:
URL: https://github.com/apache/accumulo-testing/pull/287


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org