Hi!
I tried to restore from backup, and it kept crashing with out of memory errors.
Is there something wrong, when the fdbrestore start command consumes so much memory?
I then added the parameter -m 32GB to make it use all the memory on the kubernetes pod, and it worked, but the memory consumption is still a bit worrisome. Here is a plot of the memory consumption of the pod:
As you can see, the fdbrestore command use increasingly more memory, while it goes through the snapshot files. After it is done with loading the files, the memory consumption doubles from 6.5 GB to over 12 GB. It then spends some more time, before the job is submitted successfully.
Is there any rule of thumb to how much memory is needed for a restore? Are there any parameters that I can set, to prevent the process from crashing when it’s about to run out of memory? Is the memory consumption similar for the fdbrestore start command and the backup_workers?
About the backup:
It has about 150 snapshots, and I restore to the last one. The backups and backup_workers are manually created in kubernetes (not through the fdb kubernetes operator), and they are stored in azure blobstore through a minio gateway.
Summary
root@foundationdb-cluster-1-backup-agent-7cfc4d6766-hqfcw:/var/fdb# fdbbackup describe -d ‘blobstore://minio:minio-password@minio-gateway.namespace.svc:9000/automatic_backups/2021-03-09T13-16-06?bucket=foundationdb-backup-container&sc=0’
URL: blobstore://minio:minio-password@minio-gateway.namespace.svc:9000/automatic_backups/2021-03-09T13-16-06?bucket=foundationdb-backup-container&sc=0
Restorable: true
Partitioned logs: false
Snapshot: startVersion=11731745327769 (maxLogEnd -156.43 days) endVersion=11734122762318 (maxLogEnd -156.40 days) totalBytes=252527533240 restorable=true expiredPct=0.00
Snapshot: startVersion=11734160964163 (maxLogEnd -156.40 days) endVersion=11820500498719 (maxLogEnd -155.40 days) totalBytes=252891816480 restorable=true expiredPct=0.00
Snapshot: startVersion=11820519590803 (maxLogEnd -155.40 days) endVersion=11906900438547 (maxLogEnd -154.40 days) totalBytes=253513185920 restorable=true expiredPct=0.00
Snapshot: startVersion=11906916670234 (maxLogEnd -154.40 days) endVersion=11993301498494 (maxLogEnd -153.40 days) totalBytes=254118333398 restorable=true expiredPct=0.00
Snapshot: startVersion=11993346138257 (maxLogEnd -153.40 days) endVersion=12079703061873 (maxLogEnd -152.40 days) totalBytes=254838641940 restorable=true expiredPct=0.00
Snapshot: startVersion=12079735505227 (maxLogEnd -152.40 days) endVersion=12166046799044 (maxLogEnd -151.40 days) totalBytes=255490937686 restorable=true expiredPct=0.00
Snapshot: startVersion=12166053160915 (maxLogEnd -151.40 days) endVersion=12252441567280 (maxLogEnd -150.40 days) totalBytes=256079635391 restorable=true expiredPct=0.00
Snapshot: startVersion=12252453395368 (maxLogEnd -150.40 days) endVersion=12338845736534 (maxLogEnd -149.40 days) totalBytes=256736496240 restorable=true expiredPct=0.00
Snapshot: startVersion=12338890100819 (maxLogEnd -149.40 days) endVersion=12425230802770 (maxLogEnd -148.40 days) totalBytes=257463135944 restorable=true expiredPct=0.00
Snapshot: startVersion=12425289592213 (maxLogEnd -148.40 days) endVersion=12511620834210 (maxLogEnd -147.40 days) totalBytes=258060783359 restorable=true expiredPct=0.00
Snapshot: startVersion=12511637560462 (maxLogEnd -147.40 days) endVersion=12598028105481 (maxLogEnd -146.40 days) totalBytes=258646285433 restorable=true expiredPct=0.00
Snapshot: startVersion=12598042681198 (maxLogEnd -146.40 days) endVersion=12684427639744 (maxLogEnd -145.40 days) totalBytes=259234303318 restorable=true expiredPct=0.00
Snapshot: startVersion=12684453159580 (maxLogEnd -145.40 days) endVersion=12770828779823 (maxLogEnd -144.40 days) totalBytes=259795394429 restorable=true expiredPct=0.00
Snapshot: startVersion=12770840424098 (maxLogEnd -144.40 days) endVersion=12857230245095 (maxLogEnd -143.40 days) totalBytes=260352477859 restorable=true expiredPct=0.00
Snapshot: startVersion=12857245023263 (maxLogEnd -143.40 days) endVersion=12943628400600 (maxLogEnd -142.40 days) totalBytes=260919599811 restorable=true expiredPct=0.00
Snapshot: startVersion=12943661239352 (maxLogEnd -142.40 days) endVersion=13030028070217 (maxLogEnd -141.40 days) totalBytes=261499429897 restorable=true expiredPct=0.00
Snapshot: startVersion=13030041899928 (maxLogEnd -141.40 days) endVersion=13116437485829 (maxLogEnd -140.40 days) totalBytes=262142534900 restorable=true expiredPct=0.00
Snapshot: startVersion=13116441067552 (maxLogEnd -140.40 days) endVersion=13202841354223 (maxLogEnd -139.40 days) totalBytes=262876472326 restorable=true expiredPct=0.00
Snapshot: startVersion=13202861719993 (maxLogEnd -139.40 days) endVersion=13289246244577 (maxLogEnd -138.40 days) totalBytes=269301097485 restorable=true expiredPct=0.00
Snapshot: startVersion=13289260421117 (maxLogEnd -138.40 days) endVersion=13375641240453 (maxLogEnd -137.40 days) totalBytes=270174981567 restorable=true expiredPct=0.00
Snapshot: startVersion=13375655115655 (maxLogEnd -137.40 days) endVersion=13462033154859 (maxLogEnd -136.40 days) totalBytes=270969851922 restorable=true expiredPct=0.00
Snapshot: startVersion=13462065429466 (maxLogEnd -136.40 days) endVersion=13548434643786 (maxLogEnd -135.40 days) totalBytes=293492539339 restorable=true expiredPct=0.00
Snapshot: startVersion=13548454861068 (maxLogEnd -135.40 days) endVersion=13634848498290 (maxLogEnd -134.40 days) totalBytes=308456744309 restorable=true expiredPct=0.00
Snapshot: startVersion=13634853638519 (maxLogEnd -134.40 days) endVersion=13721244352297 (maxLogEnd -133.40 days) totalBytes=309020902010 restorable=true expiredPct=0.00
Snapshot: startVersion=13721260508692 (maxLogEnd -133.40 days) endVersion=13807627886089 (maxLogEnd -132.40 days) totalBytes=309588867080 restorable=true expiredPct=0.00
Snapshot: startVersion=13807633282601 (maxLogEnd -132.40 days) endVersion=13893997875890 (maxLogEnd -131.40 days) totalBytes=310137336185 restorable=true expiredPct=0.00
Snapshot: startVersion=13894013082090 (maxLogEnd -131.40 days) endVersion=13980387561161 (maxLogEnd -130.40 days) totalBytes=310695211726 restorable=true expiredPct=0.00
Snapshot: startVersion=13980402918753 (maxLogEnd -130.40 days) endVersion=14066780096295 (maxLogEnd -129.40 days) totalBytes=311253720357 restorable=true expiredPct=0.00
Snapshot: startVersion=14066785446590 (maxLogEnd -129.40 days) endVersion=14153184344948 (maxLogEnd -128.40 days) totalBytes=311812064319 restorable=true expiredPct=0.00
Snapshot: startVersion=14153226748866 (maxLogEnd -128.40 days) endVersion=14239592282074 (maxLogEnd -127.40 days) totalBytes=312376559566 restorable=true expiredPct=0.00
Snapshot: startVersion=14239603460752 (maxLogEnd -127.40 days) endVersion=14325978499551 (maxLogEnd -126.40 days) totalBytes=312970616532 restorable=true expiredPct=0.00
Snapshot: startVersion=14325993458821 (maxLogEnd -126.40 days) endVersion=14412387030602 (maxLogEnd -125.40 days) totalBytes=313564668730 restorable=true expiredPct=0.00
Snapshot: startVersion=14412405495076 (maxLogEnd -125.40 days) endVersion=14498788715685 (maxLogEnd -124.40 days) totalBytes=314138941164 restorable=true expiredPct=0.00
Snapshot: startVersion=14498799548400 (maxLogEnd -124.40 days) endVersion=14585190211961 (maxLogEnd -123.40 days) totalBytes=314716224786 restorable=true expiredPct=0.00
Snapshot: startVersion=14585212715014 (maxLogEnd -123.40 days) endVersion=14671591320447 (maxLogEnd -122.40 days) totalBytes=315273270441 restorable=true expiredPct=0.00
Snapshot: startVersion=14671604803133 (maxLogEnd -122.40 days) endVersion=14758000758745 (maxLogEnd -121.40 days) totalBytes=315865090661 restorable=true expiredPct=0.00
Snapshot: startVersion=14758005262414 (maxLogEnd -121.40 days) endVersion=14844400212887 (maxLogEnd -120.40 days) totalBytes=316446994143 restorable=true expiredPct=0.00
Snapshot: startVersion=14844403495889 (maxLogEnd -120.40 days) endVersion=14930793539418 (maxLogEnd -119.40 days) totalBytes=317040183866 restorable=true expiredPct=0.00
Snapshot: startVersion=14930811279669 (maxLogEnd -119.40 days) endVersion=15017192722651 (maxLogEnd -118.40 days) totalBytes=317627823576 restorable=true expiredPct=0.00
Snapshot: startVersion=15017210186474 (maxLogEnd -118.40 days) endVersion=15103599194987 (maxLogEnd -117.40 days) totalBytes=324040950139 restorable=true expiredPct=0.00
Snapshot: startVersion=15103653885336 (maxLogEnd -117.40 days) endVersion=15189993534137 (maxLogEnd -116.40 days) totalBytes=324857501914 restorable=true expiredPct=0.00
Snapshot: startVersion=15190012278855 (maxLogEnd -116.40 days) endVersion=15276386552042 (maxLogEnd -115.40 days) totalBytes=325425920340 restorable=true expiredPct=0.00
Snapshot: startVersion=15276405507150 (maxLogEnd -115.40 days) endVersion=15362787680713 (maxLogEnd -114.40 days) totalBytes=332548795588 restorable=true expiredPct=0.00
Snapshot: startVersion=15362836411606 (maxLogEnd -114.40 days) endVersion=15449191198299 (maxLogEnd -113.40 days) totalBytes=390226306999 restorable=true expiredPct=0.00
Snapshot: startVersion=15449203172639 (maxLogEnd -113.40 days) endVersion=15535592524900 (maxLogEnd -112.40 days) totalBytes=475494284317 restorable=true expiredPct=0.00
Snapshot: startVersion=15535601705897 (maxLogEnd -112.40 days) endVersion=15622001843269 (maxLogEnd -111.40 days) totalBytes=506584639875 restorable=true expiredPct=0.00
Snapshot: startVersion=15622011345547 (maxLogEnd -111.40 days) endVersion=15708408333698 (maxLogEnd -110.40 days) totalBytes=519168886430 restorable=true expiredPct=0.00
Snapshot: startVersion=15708436258014 (maxLogEnd -110.40 days) endVersion=15794813800860 (maxLogEnd -109.40 days) totalBytes=523604889011 restorable=true expiredPct=0.00
Snapshot: startVersion=15794816109790 (maxLogEnd -109.40 days) endVersion=15881212870721 (maxLogEnd -108.40 days) totalBytes=528164631920 restorable=true expiredPct=0.00
Snapshot: startVersion=15881228253276 (maxLogEnd -108.40 days) endVersion=15967632510438 (maxLogEnd -107.40 days) totalBytes=611572513604 restorable=true expiredPct=0.00
Snapshot: startVersion=15967652308092 (maxLogEnd -107.40 days) endVersion=16054095880374 (maxLogEnd -106.40 days) totalBytes=816858501407 restorable=true expiredPct=0.00
Snapshot: startVersion=16054099720521 (maxLogEnd -106.40 days) endVersion=16140554110117 (maxLogEnd -105.40 days) totalBytes=1121532737270 restorable=true expiredPct=0.00
Snapshot: startVersion=16140573572366 (maxLogEnd -105.40 days) endVersion=16226991536207 (maxLogEnd -104.40 days) totalBytes=1543840643634 restorable=true expiredPct=0.00
Snapshot: startVersion=16227000873899 (maxLogEnd -104.40 days) endVersion=16313537230731 (maxLogEnd -103.40 days) totalBytes=2142739747641 restorable=true expiredPct=0.00
Snapshot: startVersion=16313546107224 (maxLogEnd -103.40 days) endVersion=16400097505796 (maxLogEnd -102.39 days) totalBytes=2866110328352 restorable=true expiredPct=0.00
Snapshot: startVersion=16400133929793 (maxLogEnd -102.39 days) endVersion=16487223466148 (maxLogEnd -101.39 days) totalBytes=3545749630189 restorable=true expiredPct=0.00
Snapshot: startVersion=16487242316409 (maxLogEnd -101.39 days) endVersion=16579681908626 (maxLogEnd -100.32 days) totalBytes=4183792717357 restorable=true expiredPct=0.00
Snapshot: startVersion=16579709466301 (maxLogEnd -100.32 days) endVersion=16670819331778 (maxLogEnd -99.26 days) totalBytes=4464309526065 restorable=true expiredPct=0.00
Snapshot: startVersion=16670836319492 (maxLogEnd -99.26 days) endVersion=16766296370390 (maxLogEnd -98.16 days) totalBytes=4598754051663 restorable=true expiredPct=0.00
Snapshot: startVersion=16766333049506 (maxLogEnd -98.16 days) endVersion=16876328824258 (maxLogEnd -96.88 days) totalBytes=4839347455909 restorable=true expiredPct=0.00
Snapshot: startVersion=16876383019371 (maxLogEnd -96.88 days) endVersion=16992779693726 (maxLogEnd -95.54 days) totalBytes=4979398808791 restorable=true expiredPct=0.00
Snapshot: startVersion=16992790883003 (maxLogEnd -95.54 days) endVersion=17115863004080 (maxLogEnd -94.11 days) totalBytes=5033542751969 restorable=true expiredPct=0.00
Snapshot: startVersion=17115910234894 (maxLogEnd -94.11 days) endVersion=17232755430039 (maxLogEnd -92.76 days) totalBytes=5053843769452 restorable=true expiredPct=0.00
Snapshot: startVersion=17232773708932 (maxLogEnd -92.76 days) endVersion=17348061809338 (maxLogEnd -91.42 days) totalBytes=5063994567053 restorable=true expiredPct=0.00
Snapshot: startVersion=17348113850577 (maxLogEnd -91.42 days) endVersion=17474255611555 (maxLogEnd -89.96 days) totalBytes=5092581687121 restorable=true expiredPct=0.00
Snapshot: startVersion=17474331977674 (maxLogEnd -89.96 days) endVersion=17604857796120 (maxLogEnd -88.45 days) totalBytes=5119946671087 restorable=true expiredPct=0.00
Snapshot: startVersion=17604907736734 (maxLogEnd -88.45 days) endVersion=17742043439805 (maxLogEnd -86.86 days) totalBytes=5144430792317 restorable=true expiredPct=0.00
Snapshot: startVersion=17742065980755 (maxLogEnd -86.86 days) endVersion=17873284645601 (maxLogEnd -85.34 days) totalBytes=5179207515655 restorable=true expiredPct=0.00
Snapshot: startVersion=17873338412000 (maxLogEnd -85.34 days) endVersion=18008999876843 (maxLogEnd -83.77 days) totalBytes=5239911569797 restorable=true expiredPct=0.00
Snapshot: startVersion=18009032454285 (maxLogEnd -83.77 days) endVersion=18146026606230 (maxLogEnd -82.19 days) totalBytes=5316028425317 restorable=true expiredPct=0.00
Snapshot: startVersion=18146108300513 (maxLogEnd -82.19 days) endVersion=18301339110384 (maxLogEnd -80.39 days) totalBytes=5382778848817 restorable=true expiredPct=0.00
Snapshot: startVersion=18301441455763 (maxLogEnd -80.39 days) endVersion=18456947891725 (maxLogEnd -78.59 days) totalBytes=5424303950495 restorable=true expiredPct=0.00
Snapshot: startVersion=18456965956585 (maxLogEnd -78.59 days) endVersion=18615971098054 (maxLogEnd -76.75 days) totalBytes=5502976931358 restorable=true expiredPct=0.00
Snapshot: startVersion=18615996371155 (maxLogEnd -76.75 days) endVersion=18762602295377 (maxLogEnd -75.05 days) totalBytes=5582112126416 restorable=true expiredPct=0.00
Snapshot: startVersion=18762618609934 (maxLogEnd -75.05 days) endVersion=18913221863417 (maxLogEnd -73.31 days) totalBytes=5591883984418 restorable=true expiredPct=0.00
Snapshot: startVersion=18914232158343 (maxLogEnd -73.30 days) endVersion=19077741641109 (maxLogEnd -71.40 days) totalBytes=5601038716302 restorable=true expiredPct=0.00
Snapshot: startVersion=19077758312349 (maxLogEnd -71.40 days) endVersion=19302333804111 (maxLogEnd -68.80 days) totalBytes=5576988430325 restorable=true expiredPct=0.00
Snapshot: startVersion=19302412963177 (maxLogEnd -68.80 days) endVersion=19460600074137 (maxLogEnd -66.97 days) totalBytes=5483241871794 restorable=true expiredPct=0.00
Snapshot: startVersion=19460655313049 (maxLogEnd -66.97 days) endVersion=19628906138558 (maxLogEnd -65.02 days) totalBytes=5374283361489 restorable=true expiredPct=0.00
Snapshot: startVersion=19629203682502 (maxLogEnd -65.02 days) endVersion=19768007157750 (maxLogEnd -63.41 days) totalBytes=5346827193749 restorable=true expiredPct=0.00
Snapshot: startVersion=19768111395262 (maxLogEnd -63.41 days) endVersion=19876567354622 (maxLogEnd -62.16 days) totalBytes=5319522641584 restorable=true expiredPct=0.00
Snapshot: startVersion=19876598156882 (maxLogEnd -62.16 days) endVersion=19967011637686 (maxLogEnd -61.11 days) totalBytes=5295602398478 restorable=true expiredPct=0.00
Snapshot: startVersion=19967042873669 (maxLogEnd -61.11 days) endVersion=20054886155156 (maxLogEnd -60.09 days) totalBytes=5271519115115 restorable=true expiredPct=0.00
Snapshot: startVersion=20054920873715 (maxLogEnd -60.09 days) endVersion=20158483954987 (maxLogEnd -58.90 days) totalBytes=5169813215856 restorable=true expiredPct=0.00
Snapshot: startVersion=20158581165359 (maxLogEnd -58.89 days) endVersion=20273436422464 (maxLogEnd -57.56 days) totalBytes=5051802384546 restorable=true expiredPct=0.00
Snapshot: startVersion=20273502785376 (maxLogEnd -57.56 days) endVersion=20390966313891 (maxLogEnd -56.20 days) totalBytes=5052760521947 restorable=true expiredPct=0.00
Snapshot: startVersion=20391201033587 (maxLogEnd -56.20 days) endVersion=20511535540370 (maxLogEnd -54.81 days) totalBytes=5053124033609 restorable=true expiredPct=0.00
Snapshot: startVersion=20511682824554 (maxLogEnd -54.81 days) endVersion=20631454304465 (maxLogEnd -53.42 days) totalBytes=5053955773484 restorable=true expiredPct=0.00
Snapshot: startVersion=20631551006356 (maxLogEnd -53.42 days) endVersion=20749193331794 (maxLogEnd -52.06 days) totalBytes=5054897471478 restorable=true expiredPct=0.00
Snapshot: startVersion=20749266280189 (maxLogEnd -52.06 days) endVersion=20872091040292 (maxLogEnd -50.64 days) totalBytes=5055365426876 restorable=true expiredPct=0.00
Snapshot: startVersion=20872168841495 (maxLogEnd -50.63 days) endVersion=20991808342341 (maxLogEnd -49.25 days) totalBytes=5056404732443 restorable=true expiredPct=0.00
Snapshot: startVersion=20991913329083 (maxLogEnd -49.25 days) endVersion=21115387831179 (maxLogEnd -47.82 days) totalBytes=5057077284565 restorable=true expiredPct=0.00
Snapshot: startVersion=21115550050045 (maxLogEnd -47.82 days) endVersion=21238537677177 (maxLogEnd -46.39 days) totalBytes=5057989528623 restorable=true expiredPct=0.00
Snapshot: startVersion=21238679249916 (maxLogEnd -46.39 days) endVersion=21361180358438 (maxLogEnd -44.98 days) totalBytes=5058740762426 restorable=true expiredPct=0.00
Snapshot: startVersion=21361269740530 (maxLogEnd -44.97 days) endVersion=21485617301504 (maxLogEnd -43.53 days) totalBytes=5059564621510 restorable=true expiredPct=0.00
Snapshot: startVersion=21485719721082 (maxLogEnd -43.53 days) endVersion=21610887228404 (maxLogEnd -42.08 days) totalBytes=5060389159031 restorable=true expiredPct=0.00
Snapshot: startVersion=21611040864360 (maxLogEnd -42.08 days) endVersion=21736447765313 (maxLogEnd -40.63 days) totalBytes=5061254379075 restorable=true expiredPct=0.00
Snapshot: startVersion=21736598737723 (maxLogEnd -40.63 days) endVersion=21860660330604 (maxLogEnd -39.19 days) totalBytes=5062116722496 restorable=true expiredPct=0.00
Snapshot: startVersion=21860808839976 (maxLogEnd -39.19 days) endVersion=21984852075899 (maxLogEnd -37.76 days) totalBytes=5062970735341 restorable=true expiredPct=0.00
Snapshot: startVersion=21984903097687 (maxLogEnd -37.76 days) endVersion=22106399359675 (maxLogEnd -36.35 days) totalBytes=5064259439325 restorable=true expiredPct=0.00
Snapshot: startVersion=22106552729011 (maxLogEnd -36.35 days) endVersion=22226784675528 (maxLogEnd -34.96 days) totalBytes=5065115810513 restorable=true expiredPct=0.00
Snapshot: startVersion=22226890459491 (maxLogEnd -34.96 days) endVersion=22349967848443 (maxLogEnd -33.53 days) totalBytes=5065952889989 restorable=true expiredPct=0.00
Snapshot: startVersion=22350023357913 (maxLogEnd -33.53 days) endVersion=22477101217459 (maxLogEnd -32.06 days) totalBytes=5066346219388 restorable=true expiredPct=0.00
Snapshot: startVersion=22477229773859 (maxLogEnd -32.06 days) endVersion=22609187225012 (maxLogEnd -30.53 days) totalBytes=5066847447863 restorable=true expiredPct=0.00
Snapshot: startVersion=22609332826751 (maxLogEnd -30.53 days) endVersion=22742942066841 (maxLogEnd -28.98 days) totalBytes=5067779806464 restorable=true expiredPct=0.00
Snapshot: startVersion=22743030716419 (maxLogEnd -28.98 days) endVersion=22877513113472 (maxLogEnd -27.42 days) totalBytes=5068634242955 restorable=true expiredPct=0.00
Snapshot: startVersion=22877658827306 (maxLogEnd -27.42 days) endVersion=23011533405248 (maxLogEnd -25.87 days) totalBytes=5069583594624 restorable=true expiredPct=0.00
Snapshot: startVersion=23011605070978 (maxLogEnd -25.87 days) endVersion=23144646393644 (maxLogEnd -24.33 days) totalBytes=5070503452880 restorable=true expiredPct=0.00
Snapshot: startVersion=23144798776991 (maxLogEnd -24.33 days) endVersion=23279651330799 (maxLogEnd -22.77 days) totalBytes=5071350374989 restorable=true expiredPct=0.00
Snapshot: startVersion=23279794973952 (maxLogEnd -22.77 days) endVersion=23415695145935 (maxLogEnd -21.20 days) totalBytes=5072196247230 restorable=true expiredPct=0.00
Snapshot: startVersion=23415767829179 (maxLogEnd -21.20 days) endVersion=23552256521675 (maxLogEnd -19.62 days) totalBytes=5073011068653 restorable=true expiredPct=0.00
Snapshot: startVersion=23552425481385 (maxLogEnd -19.61 days) endVersion=23687005002607 (maxLogEnd -18.06 days) totalBytes=5074104213567 restorable=true expiredPct=0.00
Snapshot: startVersion=23687080955208 (maxLogEnd -18.05 days) endVersion=23821589945222 (maxLogEnd -16.50 days) totalBytes=5075013990556 restorable=true expiredPct=0.00
Snapshot: startVersion=23821671790029 (maxLogEnd -16.50 days) endVersion=23957780872814 (maxLogEnd -14.92 days) totalBytes=5075836103460 restorable=true expiredPct=0.00
Snapshot: startVersion=23957873967388 (maxLogEnd -14.92 days) endVersion=24093482235160 (maxLogEnd -13.35 days) totalBytes=5076802489867 restorable=true expiredPct=0.00
Snapshot: startVersion=24093568699163 (maxLogEnd -13.35 days) endVersion=24229240929686 (maxLogEnd -11.78 days) totalBytes=5077775385137 restorable=true expiredPct=0.00
Snapshot: startVersion=24229325915994 (maxLogEnd -11.78 days) endVersion=24362045585286 (maxLogEnd -10.24 days) totalBytes=5078788553463 restorable=true expiredPct=0.00
Snapshot: startVersion=24362140716974 (maxLogEnd -10.24 days) endVersion=24497472793947 (maxLogEnd -8.68 days) totalBytes=5079589443496 restorable=true expiredPct=0.00
Snapshot: startVersion=24497567752092 (maxLogEnd -8.67 days) endVersion=24636878770286 (maxLogEnd -7.06 days) totalBytes=5080452708946 restorable=true expiredPct=0.00
Snapshot: startVersion=24637057474187 (maxLogEnd -7.06 days) endVersion=24940984872699 (maxLogEnd -3.54 days) totalBytes=5082359788221 restorable=true expiredPct=0.00
Snapshot: startVersion=24941502016764 (maxLogEnd -3.54 days) endVersion=25138402762171 (maxLogEnd -1.26 days) totalBytes=5083914409298 restorable=true expiredPct=0.00
SnapshotBytes: 360657245910048
MinLogBeginVersion: 11731745119538 (maxLogEnd -156.43 days)
ContiguousLogEndVersion: 25247024247494 (maxLogEnd -0.00 days)
MaxLogEndVersion: 25247024247494 (maxLogEnd -0.00 days)
MinRestorableVersion: 11734122762318 (maxLogEnd -156.40 days)
MaxRestorableVersion: 25247024247493 (maxLogEnd -0.00 days)
A separate issue was when the restore failed after a 499 error code, and it did not retry. Are there parameters to retry errors like 499?
fdbrestore start -t continuous -r 'blobstore://minio:minio-password@minio-gateway.namespace.svc:9000/automatic_backups/2021-03-09T13-16-06?bucket=foundationdb-backup-container&sc=0' --dest_cluster_file /var/dynamic-conf/cluster-file -v 25246000000000 -n --knob_http_verbose_level=1
Debug logs
[9b67766ce9fcf50dc39596e6a1258014] HTTP code=499 early=0, time=0.193128s GET /foundationdb-backup-container/?max-keys=1000&prefix=data/automatic_backups/2021-03-09T13-16-06/kvranges/snapshot.000023011600418151/50/&delimiter=/&marker=data/automatic_backups/2021-03-09T13-16-06/kvranges/snapshot.000023011600418151/50/range,23080921056124,236cf1ffc38fd7471f0635f9964731a2,1048576 contentLen=0 [476 out, response content len 347]
followed by a bunch of these and a crash “HTTP response code not received or indicated failure”
[c8c8a27ae92db4aadb9d47ecf3d9db6f] HTTP ERROR=operation_cancelled early=0, time=0.970043s GET /foundationdb-backup-container/?max-keys=1000&prefix=data/automatic_backups/2021-03-09T13-16-06/kvranges/snapshot.000023011600418151/0/&delimiter=/&marker=data/automatic_backups/2021-03-09T13-16-06/kvranges/snapshot.000023011600418151/0/range,23013049665647,e9c3393a8ef2ae3e842d3ba0444192f9,1048576 contentLen=0 [474 out]