tree cd6b532c7e53dc04d24df744fd30ae5e39f742e7
parent 96b1ae746167a9d8b848ff9d3dee31fa2e62c9d2
author Paul Kirth <paulkirth@google.com> 1775860370 -0700
committer GitHub <noreply@github.com> 1775860370 -0700
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJp2XqSCRC1aQ7uu5UhlAAAHUIQAELjVPWVe9IiLFra5cgG+1C/
 KYjvMU4lLtCr/rGJCHdFlWXm+bUg+Pr5Pz6lbvADVVzCfRsywaNphgoLwGBEyaiR
 +ErpxuTXf6OW3Bpo/6UA5pexIAVwBSBTf8YUWZ/+SZGAvVSRTqP4BHMmGlQ5AoEb
 N6JMpgEQnigjV+Dqwje1d3ZchD9vqM2lEy5SqonDk9W3nP5KYkJsz/8brMqEdVO7
 BhKwT3EXQ4Eag1E0UC0BwohziZj7SPnzEUOwOqWUTRX3bgYY3rWZwyy37wyRmHT+
 VsQ9LuR/W7uLf4xiidck2p6OivCJfcCW6juC4cX46Au179wj/9PjtBYok+rgwS/h
 BmrxX1C/aB1vAVO0jeuAhuWWLzh4eHrJ7xDS8C3v329y7ZtVZB1rSmDxj7euv2B0
 P2TK033gnv/sboVJeXlN6G96aLW4rrkz9siWAuM5kG4VV7y4r5eMBTb5TSZphPhz
 Rs0MGpq1ngGSbl3Iot2PbmgByALAokD/a8tXf00qtp88K6Aekd5E17ffZz7LejsF
 U+golPnhU06JJM8UPc6bwYbeLhRKYu4OyMshHwCjJX9s1KWF4meGu2/uBjpMLd/u
 Bblz6iCdIL8XcAs85ph/cV5Btj0mYWTCcQBV8KqEtXxOf27/hKFGeniJMSsNdhQ2
 yUfk+6KcN/b8+EJPwf3c
 =ZQ9i
 -----END PGP SIGNATURE-----
 

[clang-doc] Merge data into persistent memory (#190056)

We have a need for persistent memory for the final info. Since each
group processes a single USR at a time, every USR is only ever processed
by a single thread from the thread pool. This means that we can keep per
thread persistent storage for all the info. There is significant
duplicated data between all the serialized records, so we can just merge
the final/unique items into the persistent arena, and clear out the
scratch/transient arena as we process each record in the bitcode.

The patch adds some APIs to help with managing the data, merging, and
allocation of data in the correct arena. It also safely merges and deep
copies data from the transient arenas into persistent storage that is
never reset until the program completes.

This patch reduces memory by another % over the previous patches,
bringing the total savings over the baseline to 57%. Runtime performance
and benchmarks stay mostly flat with modest improvements.

| Metric | Baseline | Prev | This | Culm% | Seq% |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Time | 920.5s | 991.5s | 987.2s | +7.2% | -0.4% |
| Memory | 86.0G | 40.0G | 36.9G | -57.1% | -8.0% |

| Benchmark | Baseline | Prev | This | Culm% | Seq% |
| :--- | :--- | :--- | :--- | :--- | :--- |
| BM_BitcodeReader_Scale/10 | 67.9us | 72.2us | 72.2us | +6.3% | -0.0% |
| BM_BitcodeReader_Scale/10000 | 70.5ms | 22.5ms | 17.3ms | -75.5% |-23.2% |
| BM_BitcodeReader_Scale/4096 | 23.2ms | 6.6ms | 7.1ms | -69.5% | +7.4%|
| BM_BitcodeReader_Scale/512 | 509.4us | 898.7us | 550.5us | +8.1% |-38.7% |
| BM_BitcodeReader_Scale/64 | 114.8us | 133.7us | 120.8us | +5.2% |-9.6% |
| BM_EmitInfoFunction | 1.6us | 1.9us | 1.8us | +12.9% | -3.3% |
| BM_Index_Insertion/10 | 2.3us | 4.1us | 3.5us | +52.4% | -14.7% |
| BM_Index_Insertion/10000 | 3.1ms | 5.3ms | 4.8ms | +53.5% | -10.0% |
| BM_Index_Insertion/4096 | 1.3ms | 2.1ms | 1.9ms | +50.8% | -9.2% |
| BM_Index_Insertion/512 | 153.6us | 251.8us | 227.0us | +47.8% | -9.9%|
| BM_Index_Insertion/64 | 18.1us | 30.2us | 26.7us | +47.9% | -11.7% |
| BM_JSONGenerator_Scale/10000 | 89.6ms | 81.4ms | 83.4ms | -6.9% |+2.5% |
| BM_JSONGenerator_Scale/4096 | 33.7ms | 31.0ms | 32.4ms | -3.9% | +4.5%|
| BM_Mapper_Scale/10000 | 104.3ms | 112.3ms | 103.5ms | -0.8% | -7.9% |
| BM_Mapper_Scale/4096 | 44.3ms | 45.0ms | 43.8ms | -1.2% | -2.5% |
| BM_Mapper_Scale/512 | 7.6ms | 7.7ms | 7.5ms | -1.3% | -2.4% |
| BM_Mapper_Scale/64 | 3.1ms | 3.0ms | 3.0ms | -1.9% | -0.3% |
| BM_MergeInfos_Scale/10000 | 12.2ms | 575.6us | 500.1us | -95.9% |-13.1% |
| BM_MergeInfos_Scale/2 | 1.9us | 1.8us | 1.8us | -4.4% | -1.7% |
| BM_MergeInfos_Scale/4096 | 2.8ms | 205.3us | 200.4us | -92.8% | -2.4%|
| BM_MergeInfos_Scale/512 | 68.9us | 20.5us | 19.5us | -71.7% | -5.1% |
| BM_MergeInfos_Scale/64 | 10.3us | 3.8us | 4.0us | -60.9% | +4.8% |
| BM_MergeInfos_Scale/8 | 2.8us | 1.9us | 1.9us | -31.7% | -1.8% |
| BM_SerializeFunctionInfo | 25.5us | 25.8us | 26.1us | +2.2% | +1.3% |