[clang][Syntax] Optimize expandedTokens for token ranges.
`expandedTokens(SourceRange)` used to do a binary search to get the
expanded tokens belonging to a source range. Each binary search uses
`isBeforeInTranslationUnit` to order two source locations. This is
inherently very slow.
By profiling clangd we found out that users like clangd::SelectionTree
spend 95% of time in `isBeforeInTranslationUnit`. Also it is worth
noting that users of `expandedTokens(SourceRange)` majorly use ranges
provided by AST to query this funciton. The ranges provided by AST are
token ranges (starting at the beginning of a token and ending at the
beginning of another token).
Therefore we can avoid the binary search in majority of the cases by
maintaining an index of ExpandedToken by their SourceLocations. We still
do binary search for ranges which are not token ranges but such
instances are quite low.
Performance:
`~/build/bin/clangd --check=clang/lib/Serialization/ASTReader.cpp`
Before: Took 2:10s to complete.
Now: Took 1:13s to complete.
Differential Revision: https://reviews.llvm.org/D99086
GitOrigin-RevId: aa979084dffba86a3e170826b4e89d90820bb78b
diff --git a/include/clang/Tooling/Syntax/Tokens.h b/include/clang/Tooling/Syntax/Tokens.h
index 98320bd..e4bc155 100644
--- a/include/clang/Tooling/Syntax/Tokens.h
+++ b/include/clang/Tooling/Syntax/Tokens.h
@@ -34,6 +34,7 @@
#include "clang/Basic/TokenKinds.h"
#include "clang/Lex/Token.h"
#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/Compiler.h"
@@ -192,8 +193,13 @@
return ExpandedTokens;
}
+ /// Builds a cache to make future calls to expandedToken(SourceRange) faster.
+ /// Creates an index only once. Further calls to it will be no-op.
+ void indexExpandedTokens();
+
/// Returns the subrange of expandedTokens() corresponding to the closed
/// token range R.
+ /// Consider calling indexExpandedTokens() before for faster lookups.
llvm::ArrayRef<syntax::Token> expandedTokens(SourceRange R) const;
/// Returns the subrange of spelled tokens corresponding to AST node spanning
@@ -366,6 +372,8 @@
/// same stream as 'clang -E' (excluding the preprocessor directives like
/// #file, etc.).
std::vector<syntax::Token> ExpandedTokens;
+ // Index of ExpandedTokens for faster lookups by SourceLocation.
+ llvm::DenseMap<SourceLocation, unsigned> ExpandedTokIndex;
llvm::DenseMap<FileID, MarkedFile> Files;
// The value is never null, pointer instead of reference to avoid disabling
// implicit assignment operator.
diff --git a/lib/Tooling/Syntax/Tokens.cpp b/lib/Tooling/Syntax/Tokens.cpp
index 234df9c..2326e89 100644
--- a/lib/Tooling/Syntax/Tokens.cpp
+++ b/lib/Tooling/Syntax/Tokens.cpp
@@ -183,7 +183,31 @@
return Text.substr(Begin, length());
}
+void TokenBuffer::indexExpandedTokens() {
+ // No-op if the index is already created.
+ if (!ExpandedTokIndex.empty())
+ return;
+ ExpandedTokIndex.reserve(ExpandedTokens.size());
+ // Index ExpandedTokens for faster lookups by SourceLocation.
+ for (size_t I = 0, E = ExpandedTokens.size(); I != E; ++I)
+ ExpandedTokIndex[ExpandedTokens[I].location()] = I;
+}
+
llvm::ArrayRef<syntax::Token> TokenBuffer::expandedTokens(SourceRange R) const {
+ if (!ExpandedTokIndex.empty()) {
+ // Quick lookup if `R` is a token range.
+ // This is a huge win since majority of the users use ranges provided by an
+ // AST. Ranges in AST are token ranges from expanded token stream.
+ const auto B = ExpandedTokIndex.find(R.getBegin());
+ const auto E = ExpandedTokIndex.find(R.getEnd());
+ if (B != ExpandedTokIndex.end() && E != ExpandedTokIndex.end()) {
+ // Add 1 to End to make a half-open range.
+ return {ExpandedTokens.data() + B->getSecond(),
+ ExpandedTokens.data() + E->getSecond() + 1};
+ }
+ }
+ // Slow case. Use `isBeforeInTranslationUnit` to binary search for the
+ // required range.
return getTokensCovering(expandedTokens(), R, *SourceMgr);
}
diff --git a/unittests/Tooling/Syntax/TokensTest.cpp b/unittests/Tooling/Syntax/TokensTest.cpp
index 6a21be6..1768529 100644
--- a/unittests/Tooling/Syntax/TokensTest.cpp
+++ b/unittests/Tooling/Syntax/TokensTest.cpp
@@ -106,6 +106,7 @@
void EndSourceFileAction() override {
assert(Collector && "BeginSourceFileAction was never called");
Result = std::move(*Collector).consume();
+ Result.indexExpandedTokens();
}
std::unique_ptr<ASTConsumer>
