Skip to content

Semantic Tokens ​

clice colors every token of a document: names by what they declare or name, keywords, literals, comments, operators and directives by their spelling. It speaks a vocabulary of its own, made for C and C++, rather than the standard LSP one: a type alias, a template type parameter and a non-type template parameter are three kinds, and a modifier such as controlFlow says what a keyword, a function or a macro does to the flow of control. The clice extensions for VS Code and Neovim color all of it; any other editor colors the names it knows, and the configuration renames clice's to those.

Kinds ​

KindMarks
commenta comment
string, character, numbera literal, its prefix and suffix included
escapean escape sequence in a literal: \n, \x41, {{ in a format string
formatSpecifiera placeholder of a format string: {}, {:>8}, %d
keyworda keyword; module, import, override and final where they act as one; an operator spelled as a word: sizeof, new, and; Objective-C's @interface and its other @ keywords, the attributes of an @property, the - or + opening a method, self and super
primitivea built-in type: int, bool, char8_t
operatoran operator, built-in or overloaded
bracketthe < and > around template arguments and parameters, and around Objective-C protocols and type arguments
directive, headera preprocessor directive; a header name
namespace, modulea namespace or namespace alias; a C++20 module name; an Objective-C category
class, struct, union, enuma class by the keyword that declares it; an Objective-C class
typeAliasa typedef or using alias
typeParametera template type or template template parameter
templateParametera non-type template parameter
concepta concept
interfacean Objective-C protocol
function, methoda free function; a member function, constructors and destructors included, an Objective-C method at every piece of its selector
variable, parameter, fielda variable, a function parameter, a data member, an Objective-C property or instance variable
enumMember, labelan enumerator; a label
macro, macroParametera macro; a parameter of a function-like macro, in its definition
attributean attribute name: nodiscard, gnu::packed
unknowna dependent name only an instantiation resolves: T::reset(), value.size()
conflicta name meaning entities of different kinds at once
plaina token in an untaken branch with no kind of its own, or any with lexical tokens off, sent for its inactive

Modifiers ​

A modifier named like a standard LSP one means what the standard one means, on every kind:

ModifierMeans
declarationdeclares the entity without defining it
definitiondefines the entity
readonlythe object cannot change through the name: a constant, a reference to const, a const pointer, an enumerator, a const method
statica static member or a static local
deprecatedthe entity is deprecated
abstracta pure virtual method, or a class with one
defaultLibrarydeclared in a system header or built into the compiler
documentationinside a documentation comment
modificationthe entity changes there: assigned, incremented, initialized as a member, passed by reference or address to non-const

A name a declaration gives carries where it is visible from: local (in its function or block), member (in its class), internal (in its translation unit, as static and unnamed namespaces make it) or global.

The rest are clice's own, and some mean something of their own on each kind:

ModifierOnMeans
templatea class, alias, variable or functiona template, or a specialization of one
dependentany namedepends on a template parameter, every unknown name carries it; an Objective-C message to id or Class
virtuala method; a classa virtual method; a polymorphic class
constexpra function, variable or templateParameterconstexpr, consteval or constinit, or a non-type template parameter
constructor, destructora methoda constructor; a destructor
callablea macroa function-like macro
a variable, parameter or fieldof a type that can be called: a function pointer, a lambda, a class with operator()
controlFlowa keywordtransfers control: if, for, return, throw, co_await, case
a function[[noreturn]]
a macroits expansion can leave the function: return, break, goto, throw
userDefinedan operator; a literal's suffixcalls an overloaded operator; a user-defined literal
declaratoran operatorthe *, & or && of a declarator, apart from the same operators in expressions
operatora keywordan operator: sizeof, new, static_cast, and, defined
literala keywordtrue, false, nullptr
affixa literalits prefix or suffix: u8, L, 42u, 1.5f
emptya macroexpands to nothing
inactiveany tokenin an untaken preprocessor branch, or an argument a macro drops

Editors ​

  • VS Code: the clice extension declares every kind and modifier with the TextMate scopes themes color, so keyword.controlFlow takes a theme's color for control keywords and escape its color for escape sequences.
  • Neovim: the clice plugin links the @lsp.type.* and @lsp.mod.* groups of clice's names to the groups colorschemes style, as defaults a colorscheme can override.
  • Other editors — Zed, Kate, Emacs, Sublime Text — color the standard LSP names. Renaming clice's kinds to the standard names closest to them colors them there; the renames go in that editor's initializationOptions for clice, not in a clice.toml VS Code reads too:
json
{
  "semantic_tokens": {
    "kinds": {
      "character": "string",
      "escape": "string",
      "formatSpecifier": "string",
      "primitive": "type",
      "bracket": "operator",
      "directive": "macro",
      "header": "string",
      "module": "namespace",
      "union": "struct",
      "typeAlias": "type",
      "templateParameter": "parameter",
      "concept": "type",
      "field": "property",
      "macroParameter": "parameter",
      "attribute": "decorator"
    }
  }
}

Zed asks for semantic tokens only once semantic_tokens is "combined" (or "full") for C and C++; the mapping goes under lsp.clice.initialization_options.

Configuration ​

The [semantic_tokens] section of clice.toml, or the same keys in the editor's initializationOptions, holds the switch for lexical tokens and the two renaming tables. The names are announced once, when the editor connects: they come from the editor's settings and the first workspace folder's clice.toml, and a change takes a server restart. See the configuration reference for details.

Supported

Lexical tokens

With lexical off, only what the editor's own grammar cannot tell is sent

Keywords, literals, comments, built-in operators and directives are left to the editor's grammar. Names, template brackets and declarator operators are still sent, and so is the code of untaken branches, as plain tokens: the grammar's color stays and only dims.

Supported

Kind names

kinds renames the token kinds the server announces

Two kinds given one name merge into it, and a kind renamed to "" is not sent, leaving the editor's own color. The names are announced once, when the editor connects: a change takes a restart.

Supported

Modifier names

modifiers renames the token modifiers the server announces

A modifier renamed to "" is never set. Like the kinds, the names are announced once, when the editor connects.

Lexical Tokens ​

Kinds derived from the token stream itself, independent of the AST.

Supported

Comments

Line, block and documentation comments receive comment tokens, a documentation comment with the documentation modifier

Supported

Literals

Numbers, characters and strings receive literal tokens

Supported

Keywords

Alternative operator spellings are keywords with the operator modifier, and contextual specifiers keep keyword tokens

Supported

Primitive types

Built-in types use a distinct token kind instead of plain keyword

Supported

Literal prefixes and suffixes

A literal's prefix and suffix are tokens of their own, with the affix modifier

They keep the kind of their literal; a user-defined literal's suffix also carries userDefined.

Supported

Escape sequences

Escape sequences inside literals are escape tokens

Supportedclangd#1421

Declarator vs operator disambiguation

A declarator's *, & and && carry the declarator modifier, apart from the same operators in expressions

Supported

Bracket token types

The angle brackets of template arguments and parameters are bracket tokens

They are told apart from the comparisons an editor's grammar takes them for. Parentheses, square brackets and braces, which the editor matches itself, get no token.

Supportedclangd#2209

Attribute names

Attribute names are attribute tokens, and their arguments keep their own kinds

Supported

Format strings

The placeholders of a format string are formatSpecifier tokens

A string passed where a function takes a format string — a basic_format_string parameter as std::format, std::print and {fmt} declare it, or the argument a format(printf, …) or format(scanf, …) attribute names — splits into its text and its placeholders: {} and {:>8.2f} in the first style, %d and %-8.3f in printf's, %ms and %15[^,\n] in scanf's, a raw string's too. Escaped braces and %% are escapes; a string no function reads as a format is plain text.

Supported

Documentation comments

Commands in documentation comments are keywords, the names they document parameters

Inside a /// or /** */ comment, a command such as \param, @brief or @return is a keyword and the name after @param a parameter, both with the documentation modifier; the rest of the comment stays a comment. An ordinary comment has no commands.

Supported

Word operators

Word operators such as sizeof, new and static_cast are keywords with the operator modifier

sizeof, alignof, typeid, new, delete, the named casts and the alternative spellings such as and take a keyword's color, and the modifier tells them apart from other keywords. co_await, which suspends the coroutine, carries controlFlow instead.

Declarations ​

Names classified by the declaration they define or reference.

Supported

Declared names

Each declared name is colored by what it declares

Supported

Namespaces

Namespace definitions, references, nesting and aliases receive namespace tokens

Supported

Types

Type definitions and references keep their respective type kinds

Supported

Functions and methods

Function declarations, definitions and calls receive function tokens

Supported

Variables

Variable declarations and references keep their respective variable kinds

Supported

Templates

A type parameter is a typeParameter, a non-type one a templateParameter, and a template carries template

Supported

Concepts

Concept definitions and constraint uses receive concept tokens

Supported

Labels

Labels and their goto references receive label tokens

Supported

Structured bindings

Structured binding names receive variable tokens at definition and use

The opening [ deliberately carries no token; only the binding names themselves are highlighted.

References ​

Reference sites retain the semantic kind of the declaration they resolve to, including names reached through language-specific lookup rules.

Supportedclangd#122

Member initializer lists

Member initializer lists highlight initialized names as fields

Supportedclangd#2619

Using declarations

The introduced name keeps its target's kind

Dependent names

Dependent names resolve through known primary templates

Dependent members of a known template (Box<T>) resolve to the primary template's declarations and keep their kinds. Members of a bare template parameter have no candidate declaration: they are unknown, or a type after typename. Every dependent name carries dependent.

Supported

Lambda captures

By-copy and by-reference captures reference the captured variable; this stays a keyword

Spelling forms

Names keep their kinds in the less common ways C++ spells them

An auto parameter is a keyword, not a template parameter; new behind :: an operator; the class of a pointer to member a class; the left operand of an overloaded = is modified like a built-in one's; a member template named after template is unknown until an instantiation.

Special Members ​

Constructors, destructors and operator functions read as the methods they are, with modifiers of their own.

Constructors and destructors

Constructors and destructors use method tokens with dedicated modifiers

A destructor's whole name, the ~ and the class name, is one method token.

Supported

Operator names

An operator function's name keeps operator as a keyword, and each call to it is an operator token

Token Modifiers ​

Modifiers add what the kind leaves out: a definition, a static or const symbol, a deprecated one, one from a system header.

Supported

Declaration vs definition

Declaration and definition modifiers distinguish the two sites

Supported

Static

Static members and locals carry the static modifier

Supported

Readonly

Const values and methods, plus enum members, carry the readonly modifier

Readonly is about the object a name stands for: a reference to const and a const pointer are readonly, a pointer to const is not.

Supported

Virtual and abstract

Virtual methods and abstract classes carry virtual or abstract modifiers

Supported

Deprecated

Deprecated declarations and uses carry the deprecated modifier

Supported

Default library

Symbols from system headers carry the default-library modifier

Supportedclangd#352

Scope modifiers

A declared name carries where it is visible from: its function, its class, its file or everywhere

Supportedclangd#839

Mutable arguments

An argument the callee may modify, through a reference or a pointer to non-const, carries modification, as the target of an assignment does

Supportedclangd#1521

User-defined operators

A call to an overloaded operator is an operator token with the userDefined modifier

Supported

Callable variables

A variable, parameter or field whose type can be called is callable

Function pointers, pointers to member functions, lambdas and objects of a class with an operator(), its own or a base's, all carry it, at their declaration and at every use.

Supported

Templates

A class, alias, variable or function template is template wherever it is named

So is a specialization of it, and the < and > around template arguments are bracket tokens, told apart from comparisons.

Supported

Control flow

Keywords that transfer control and functions that never return are controlFlow

if, for, return and the other statements that branch, loop or leave carry it, telling them apart from declaration keywords such as int or const. A [[noreturn]] function carries it too, as does a macro whose expansion can leave the function.

Conflict & Ambiguity ​

C++ allows structurally different entities to share one name. When a single written name refers to entities of different kinds at once, no single token type is correct; such names receive the dedicated conflict token type, which clients typically display in a neutral color.

Supported

Conflicting names

A name that means both a type and a function or variable renders as conflict

Preprocessor ​

Directives, header names and macro expansions receive tokens of their own, while the body of a macro definition keeps its lexical kinds.

Supported

Preprocessor directives

Directives keep their directive kind, and the code of an untaken branch keeps its lexical kinds

Pragma arguments other than numbers stay plain.

Supported

Header names

Quoted and angled header names receive header tokens

Supported

Untaken branches

Tokens in untaken branches keep their lexical kinds and carry the inactive modifier

Text with no kind of its own becomes a plain token, so even a lone } line dims.

Supported

Expansion sites and arguments

Expansion names are macros, written arguments keep their semantics, definition bodies stay lexical but for their parameters

Supportedclangd#2649

Object-like vs function-like macros

A function-like macro carries callable, an object-like one does not

Supported

Empty and control-flow macros

A macro that expands to nothing is empty, one that can leave the function controlFlow

Both modifiers mark the macro's name at its definition and at every expansion. The arguments an empty macro drops never reach the compiler and are inactive, like the code of an untaken branch.

Supported

Macro names in conditions

A name a condition tests is a macro, defined or not

#ifdef, #ifndef, defined(…), #undef and the names an #if tests name macros whether or not one is defined; defined and checks such as __has_include are operators, what such a check tests is no macro, and the file __has_include tests is a header name.

Supported

Arguments a macro drops

The arguments of parameters a macro's definition never names are inactive

Like the code of an untaken branch, they never reach the compiler: all of them for a macro that expands to nothing, the second one here for FIRST.

Leading Directives ​

The includes and macros a file starts with are compiled apart from the rest of it, and their tokens read the same as anywhere else.

Supported

Inactive preamble regions

Untaken branches among the leading directives dim the same way

Supported

Macro definition and expansion

A macro's definition and its expansions receive macro tokens

Modules ​

Module declarations and imports receive tokens for their keywords and names.

Supported

Module declarations

Module declarations tokenize contextual keywords, dotted names and private fragments

Supported

Module partitions

Module declarations tokenize partition names

Supported

module and import as identifiers

Contextual keywords keep their semantic kinds outside module declarations

Languages ​

C and Objective-C sources are colored with the same kinds as C++: an Objective-C entity takes the kind the tables above give it.

Supported

Objective-C sources

Classes, protocols, categories, methods, properties and instance variables of Objective-C are colored by what they are

Every piece of a selector colors as the method it names; @interface, @property and the other @ keywords read as keywords, self and super as this does in C++, and @"text" as a string.

Inactive Code Regions ​

Every token inside an untaken preprocessor branch carries the inactive modifier while keeping its lexical kind, so editors dim the region by styling the modifier without losing the syntax colors underneath. Tokens without a kind of their own in dead code — bare identifiers and plain punctuation — are sent as the unstyled plain kind, giving the whole region token coverage. The arguments a macro drops, such as those of an assert compiled away, dim the same way. The clice VS Code extension renders the regions dimmed out of the box; other editors style the modifier directly (e.g. @lsp.mod.inactive in Neovim).

  • Inactive preprocessor branches are dimmed (clangd#132), with the right boundaries in #elif chains (clangd#602).
  • The syntax colors stay within them (clangd#1664), and they stay distinct from comments (clangd#1545).

A branch the preprocessor skips is dimmed, whichever directive opens it. In a header, the dimmed branches are those its host skips. Code another context of the file compiles keeps its semantic kinds while dimmed. The dimming follows an edit to a condition as you type.

Supported

Untaken branches

Lines of a branch the preprocessor skips are dimmed, an #else after a taken branch included

Supported

Switching the host

A header shows the branches its chosen host takes, and switching hosts updates the dimming at once

Supported

Editing a condition

Editing the macro a condition tests moves the dimming to the branch now skipped, before the file is saved

Protocol Support ​

  • Full document semantic tokens (textDocument/semanticTokens/full).
  • Range requests (textDocument/semanticTokens/range): the tokens a range overlaps.
  • Delta updates (textDocument/semanticTokens/full/delta): the changes since the answer the editor holds.
  • UTF-16 positions, or UTF-8 or UTF-32 for an editor that offers them; a token spanning lines is split per line.

Known Limitations ​

  • While a file's compile is in progress — when it is first opened, or after a header it includes changed — its tokens come from the project index. They name everything the finished compile names, but leave out what only the compile knows: dependent and unknown names, declarator operators, the brackets of a lambda's template parameters, format string placeholders, untaken branches and the arguments macros drop. These appear once the compile finishes.
  • Code no control flow reaches, and the branch of an if constexpr an instantiation discards, are not dimmed: the first takes a control flow analysis, the second differs from one instantiation to the next (clangd#1828).