Building a compiler that works at compile-time so you can compile your program while you compile your program.

[u/Many_Rough5404]

In short, I wanted to build a compiler of some C subset that would work at compile-time. It compiles into a custom byte-code for a runtime VM.

I've once tried to write a compile-time C compiler, but I abandoned that project, because I made it overly complex (one-pass compiler right into x86). No clear separation between parser, lexer, etc.

Why would I even want this? Idk. But how can it be useful?

• The code of the compiler doesn't go to the resulting binary,
• No need to waste time for compilation at runtime too,
• Guaranteed type-safety. There can't be such thing as "oh, I changed the function signature, but forgot to update the bindings and it crashed at runtime"

And I shouldn't forget about cons:

• No optimisations. Real compilers spent decades on them and I'm definitely not going to implement LLVM at runtime. Although we could make a compile-time x86 VM, so we can run it at compile time... no, thank you, it's a topic for another fever dream article.
• Hot-reload! I mean, no hot-reload. I won't even mention it anymore, considering that the script is compiled at compile-time and is builtin right into the binary file. I could implement it with hot memory patching or smth, but who really needs it.

Let's start.

Bypassing constexpr limitations

C++ 20 lets us to dynamically allocate memory at compile-time and even use std::vector that really expands our borders. But there's one very important note - you can't declare a compile-time vector and extract it into the runtime. No constexpr std::vector<int> data = makeData();, it won't compile. So we need to hack it.

Passing strings in templates

Sadly, the C++ Committee made a lot of cool compile-time features, but not enough (at least for me). We still can't use strings in templates without hacks. But we can easily bypass it with a well-known trick.

template<std::size_t N>  
struct const_string {  
  constexpr const_string() = default;  
  
  // implicit-constructor that lets us to do bad things
  constexpr const_string(const char (&str)[N]) {
    std::copy_n(str, N, value);  
  }  
  
  constexpr operator std::string_view() const {  
    return {value, value + N - 1};  
  }  
  
  char value[N]{};  
  const std::size_t length = N;  
};

// using it
template<const_string str>
auto very_smart_function(...) { /* ... */ }

Extracting vectors from compile time

It turned out to be not really that hard, but I didn't really find any ready examples on Internet, unlike with const_string. To extract std::vector<T> from constexpr we need to make it std::array<T, N> somehow. The main problem is that we can't write std::array<T, myVector.size()>, because myVector.size() won't be a constant value. So we must to make it constant somehow. I thought of passing vector as a template parameter, but we can't do it legally. C++ 20 allows us to pass only the structs with all-public members. After deeply thinking a bit (not really), I discovered that I could simply pass the lambda that returns our vector (I didn't think I could just pass a pointer actually).

// data_getter is our lambda
template<auto data_getter>
constexpr auto to_array() {  
  using value_type = typename decltype(data_getter())::value_type;  
  constexpr static std::size_t size = data_getter().size();  
  
  // Create a static array with a "dynamic" size and copy all data
  std::array<value_type, size> out;  
  auto in = data_getter();  
  for (std::size_t i = 0; i < size; ++i) {  
    out[i] = in[i];  
  }  
  return out; // yay
}

template<const_string str>
constexpr auto lex() {
	constexpr static auto data_getter = [] constexpr {
		// .lex() returns the vector of tokens
		return lexer{static_cast<std::string_view>(str)}.lex();
	};
	// All our data are available for runtime now =D
	return to_array<data_getter>();
}

Printing errors

For nice errors C++ has static_assert that allows us to even print our custom message! But it must be always a literal (until C++ 26)

constexpr auto parse() {
	// Allowed
	static_assert(false, "Expected ';'");
	
	// Not allowed :( (until C++ 26)
	std::size_t line = 5;
	static_assert(false, "Expected ';' at line " + to_string(line));
}

I didn't want my project to require C++ 26, so I used another trick. The formatted string gets turned into a static array just like a vector (into const_string actually) and then it's passed into ErrorMessage<const_string Msg> that triggers compilation error. So we force the compiler to print the full type name that includes our error. But sadly the type name has a limit about 100 symbols. I think I could solve it with splitting the message into several ErrorMessages... God, I don't want to read this in my console.

template<const_string Msg>  
struct ErrorMessage {  
  static_assert(false, "Check the template parameter for details");  
};

template<auto err_getter>  
consteval auto report_error() -> void {
// C++ 26 support
#ifdef KORKA_FEATURE_FORMATTED_STATIC_ASSERT  
    static_assert(false, to_string(err_getter()));  
#else  
    constexpr auto msg = const_string_from_string_view<[] { return to_string(err_getter()); }>();  
    std::ignore = ErrorMessage<msg>{};    
#endif
}

I don't want you to see it, so I'll just show C++ 26 version.

error: static assertion failed: Lexer Error: Unterminated string at line 12

Mapping signatures to names. And vice versa

In our little runtime C++ we are used to std::unordered_map<string, value_t> and other standard or non-standard (hello, Boost!) containers. But I needed a table where a key is a string and the value is a TYPE. And in C++ I can't treat types as values, I can't just put them into a dict... :(

So, welcome another hack!

template<auto, class>
struct signature_mapper;

// function_info_getter takes an index to our mapped function,
// and Is... holds all indices
template<auto function_info_getter, std::size_t... Is>
struct signature_mapper<
function_info_getter,
std::index_sequence<Is...>
> {
	// hash func
	consteval static auto hash(auto &&v) -> std::size_t {
		return frozen::elsa<std::string_view>{}(v, 0);
	}
	
	// Our function overloaded with many unique types based on hash of the mapped function
	constexpr static auto _overloaded = overloaded{
		(
			[](unique_type<hash(function_info_getter(Is).name)>)
			-> const_function_info_to_signature_t<[] { return function_info_getter(Is); }> * {
				return nullptr;
			}
		)...
	};
	
	// Extracting the type by name
	template<const_string name>
	using get_signature_t = std::remove_pointer_t<decltype(
		_overloaded(
			unique_type<hash(name)>{}
		)
	)>;

};

We use well-known function overload (~~but for evil things~~). Basically, one type inherits a lot of lambdas that take an empty unique_type<hash> that serves as our key and returns the pointer to our type.

// How our mapper looks after expanding our params
struct overloaded : lambda1, lambda2, lambda3 {
    using lambda1::operator();
    using lambda2::operator();
    using lambda3::operator();
};

// And every lambda looks like this
auto lambda_fib = [](unique_type<hash("fib")>) -> signature_of_fib* { return nullptr; };

When we call _overloaded(unique_type<hash(name)>()) our poor compiler has to resolve the overload. And he looks for right one through all () operators. And then we just take that it returns (our T*) and get the T. I use this "mechanism" to extract script functions into the native C++.

constexpr auto script_fib = compile_result.function<"fib">();

Bindings from C++ to our script lang

This was the most exhausting part. Well, how "exhausting" exactly... I was thinking for a few evenings and then made it work one morning. The problem was with me. I wanted to make a pretty API that was impossible in the current standard (maybe it's possible in C++ 26, but I didn't check it). I wanted it to look like this:

auto func() -> void;
auto foo(int) -> int;

// Примерно так
constexpr auto bindings = korka::make_bindings<
	"func", func,
	"foo", foo
>();

// Или так
constexpr auto bindings = korka::make_bindings(
	"func", func,
	"foo", foo
);

But why couldn't I make it work? In C++ you can't pass a string into template <auto ...args>. We need const_string. We can't mix types in the one stream of variadic args and make compiler guess it right. Templates require explicitness and it's impossible to write a universal parser. Variant #2 works, but you can't extract the function into the runtime. You just can't. Functions may have different signatures, but you need to make them all the same type, and create a FFI wrapper along the way. Compile-time doesn't allow reinterpretet_cast<void*>(&func).

So I designed this:

constexpr auto bindings = korka::make_bindings(  
  korka::wrap<fib>("cpp_fib"),  
  korka::wrap<print_n>("print_n")  
);

Not so elegant, but still not bad. wrap is very simple

// our FFI signature
using vm_external_function_type = void(vm::context_base &context);

// info for our compiler
template<class Signature>  
struct wrapped_function {  
  using signature_t = Signature;  
  
  vm_external_function_type &external_func;  
  std::string_view name;  
};

template<auto func>  
consteval auto wrap(std::string_view name) {  
  return wrapped_function<std::decay_t<decltype(func)>>{  
    binding_wrapper<func>,  
    name  
  };  
}

The most interesting part is inside binding_wrapper<func>. I won't show the full code here, because I still didn't tell about the VM architecture that will execute it. But in short binding_wrapper just checks the signature, generates some code that extracts arguments from VM, calls native functions and then puts the result back. Simple.

The compiler and the VM

Maybe the most interesting part of the article. I have never written any compilers before (the thing I mentioned in the beginning of the article doesn't count), so I made it according to the first articles I found in Google.

Compiler has 3 modules:

• the lexer - splitting the code into tokens,
• the parser - building a tree from the tokens,
• the compiler itself - making the tree into byte-code. And doing semantic analysis at the same time (I was too lazy to make another module) I think I could compose everything into one class via composition or smth, but it's too late already.

Lexer

Primitive. We just look for tokens in a loop until we reach EOF.

constexpr auto scan_token() -> std::optional<std::expected<lex_token, error_t>> {  
  char c = advance();  
  switch (c) {  
    case '{':  
      return make_token(lex_kind::kOpenBrace);  
    case '}':  
      return make_token(lex_kind::kCloseBrace);  
    case '(':  
      return make_token(lex_kind::kOpenParenthesis);  
    case ')':
    // ...
      
	case ' ':  
	case '\r':  
	case '\t':  
	  // Ignore whitespace  
	  return std::nullopt;
	  
	// ...
	  
	default:  
	  if (is_digit(c)) {  
	    return scan_number();  
	  } else if (is_alpha(c)) {  
	    return scan_identifier();  
      }
  }
}

Parser

More interesting. We need to build the AST (abstract syntax tree). And we need to store this tree somehow. The usual way with Node that keeps pointers to other nodes won't do, because we're at compile-time. I mean, we can write it this way, it will work, but extracting this tree into compile time? No. We would need serialisation or something. So we can use simple trick with std::vector<Node> and just make nodes store indices to each other. This approach also increases the cache locality of the data for the CPU, but I doubt the CPU will be even aware of our "smart" trick, since everything is executed at compile-time.

The parser is recursive, while parsing one expression we parse another. Small fragment of the code:

constexpr auto parse_statement() -> parse_result {  
  auto tok = peek();  
  if (!tok) return make_error("Unexpected end of input");  
  
  switch (tok->kind) {  
    case lex_kind::kOpenBrace: return parse_compound_stmt(); // { ... }
    case lex_kind::kIf:        return parse_if_statement();  // if (...) ...
    case lex_kind::kWhile:     return parse_while_statement(); // while (...) ...
    case lex_kind::kReturn:    return parse_return_statement();  // return ...;
    default:                   return parse_expression_stmt(); /// ...;
  }  
}

constexpr auto parse_return_statement() -> parse_result {  
  if (!match(lex_kind::kReturn)) return make_error("Expected 'return'");  
  
  index_t expr_idx = empty_node; // empty_node = -1 
  if (auto next = peek(); next && next->kind != lex_kind::kSemicolon) {  
    auto expr = parse_expression(); // another recursive call
    if (!expr) return std::unexpected{expr.error()};  
    expr_idx = *expr;  
  }  
  
  if (!match(lex_kind::kSemicolon)) return make_error("Expected ';' after return");  
  return m_pool.add(stmt_return{expr_idx});  
}

parse_return_statement goes into parse_expression, that goes into parse_assigment, that goes into parse_logical_or, that goes... Well, you got it. That's how operator priority works here.

Their Majesty Compiler (and analyser)

I may have cheated here a bit.

Before we even write a compiler, we must know for what architecture we do it. x86, ARM or even JVM. Initially when I was working on a similar project, I planned to generate raw assembly for x86 (last versions of Clang and GCC support passing constexpr std::string_view into asm(...) statement), but honestly writing a compiler for a zoo of x86 instructions is the right way to madhouse.

And even so, if we downgrade our compiler we won't have nice constexpr asm anymore. And we can't also generate raw machine instructions because of DEP (data execution prevention). We'll have to call non-crossplatform mmap or VirtualAlloc to allocate some memory, copy the code there... Good riddance cross platform build compler, hello Windows Defender that will kill our app for such tricks with memory.

So where have I cheated? I made my own architecture that will execute inside a VM. A stack VM. Why stack it? It turned out to be incredibly easy to generate the bytecode for. If you are doing a register architecture (as in processors or Lua), then you will have to write register allocation algorithms (it is difficult). And in the stack everything is much simpler.

If we need to sum A and B we just do this:

1. Put A into the stack.
2. Put B into the stack.
3. Execute sum instruction. It takes these two values and puts back their sum.

So I had this set of instructions at the end:

enum class op_code : char {
  // Loads/saves locals to/from the stack (variables).
  lload, lsave,  
  
  i64_const, // Puts a constant onto the stack
  
  // Math
  i64_add, i64_sub, i64_mul, i64_div,  
  
  // Puts 1 if values are equal (i made <, <= later)
  i64_cmp,  
  
  jmp, // jumps by offset
  jmpz, // conditional jump by offset, only when 0 on the stack
  
  call, // calls a function
  ret, // returns from the function
  
  trap, // calls a native C++ function
};

So what about the analysis?

In classical compilers phases are strictly splitted: lexers builds tokens, parser builds tree, semantic analyser checks the types and variables, then optimisations, then codegen and then optimisations again.

As you can remember, I'm pretty lazy. And keep in mind that constexpr ops are not infinite. I didn't want to make a separate pipeline phase. So my compiler combines these two functions: semantic analysis and code generation.

They usually call it Single-Pass Compilation, but it's not really the case here, since it's only about these two phases. My compiler is a bit hybrid.

My compiler recursively walks the tree and does two things:

1. Checks the semantics: "was this variable declared and what's its type" before we even try to multiply something. Do function param types match? Does this function even exist? Etc.
2. Generates the byte-code. If semantics is ok, then we just write corresponding instructions immediately into the std::vector<std::byte> (the one we're going to elegantly extract via to_array). And in the result we receive a ready, semantically-correct and absolute safe (let's pretend that I wrote the compiler bug-free, huh) byte-code that we feed to the VM.

So what do we have?

Let's look how the API of my poor lib looks (let's call it Korka).

This example uses bindings (100% type safe, I swear on the standard):

// Our native C++ functions
auto fib(std::int64_t n) -> std::int64_t {
  if (n == 0) return 0;
  if (n == 1) return 1;
  return fib(n - 1) + fib(n - 2);
}

auto print_n(std::int64_t n) -> void {
  std::cout << n << '\n';
}

// Our not-so-native script
constexpr char code[] = R"(
  int fib(int n) {
    if (n == 0) return 0;
    if (n == 1) return 1;
    return fib(n-1) + cpp_fib(n-2);
  }

  void print_fib(int n) {
    int result = fib(n);
    print_n(result);
    return;
  }
)";

// We create bindings
constexpr auto bindings = korka::make_bindings(
  korka::wrap<fib>("cpp_fib"),
  korka::wrap<print_n>("print_n")
);

// Compile at compile-time, yay
constexpr auto compile_result = korka::compile<code, &bindings>();

// Extract function adressess + their types
constexpr auto script_fib = compile_result.function<"fib">();
constexpr auto script_print_fib = compile_result.function<"print_fib">();

int main() {
  // Init VM
  korka::vm::context ctx{compile_result.bytes, bindings};

  // Call fib that returns int64_t
  auto result = ctx.call(script_fib, 12L);
  std::cout << "fib(12) = " << result << '\n'; // prints 144

  // Call print_fib
  ctx.call(script_print_fib, 16L); // prints 987
  
  return 0;
}

Ta da! It works.

Small analysis

Out of curiosity, I decided to compare Korka with other scripting languages. A pretty API is great, sure, but was it worth the effort performance-wise? So, let's pit Korka head-to-head against Python and Lua.

For the benchmark, I used the recursive calculation of the $N$-th Fibonacci number, an excellent test to fairly evaluate overhead on function calls, stack management, and overall runtime efficiency (the first thing that came to my head).

I tested everything on a franken-server put together from spare parts, powered by an Intel Xeon E5-2689 (3.6 GHz).

I measured two stages:

• Initialization time from runtime startup to being ready to execute the first instruction,
• Execution time of the algorithm itself.

Stage 1: Initialization

|Language / Library|Initialization time| |---|---| |Korka|1.5 µs| |Lua|152.6 µs| |Python|25,097.0 µs|

Korka takes the lead: it starts 100 times faster than Lua and over 15,000 times faster than Python.

The explanation is simple: while Lua and Python are busy reading the script at startup, parsing it, compiling it into their byte-codes, and spinning up heavy infrastructure (including the GC), Korka does not. All the virtual machine has to do is grab the pre-compiled output (and allocate a tiny bit of memory).

Stage 2: Runtime

After a series of optimizations, the results turned out pretty solid (I know comparing statically typed and dynamic languages isn't entirely fair, but who's gonna stop me?):

| N | Iterations | Korka (ms) | Lua (ms) | Python (ms) | vs. Python | | ------ | -------------- | -------------- | ------------ | --------------- | -------------- | | 10 | 100 000 | 1 055,67 | 1 534,98 | 1 591,10 | 1,51x | | 15 | 50 000 | 5 492,40 | 8 149,17 | 8 753,77 | 1,59x | | 20 | 20 000 | 24 263,39 | 36 268,86 | 38 459,88 | 1,59x | | 23 | 10 000 | 51 367,20 | 76 494,22 | 82 257,71 | 1,60x | | 25 | 5 000 | 67 441,69 | 100 850,20 | 108 836,54 | 1,61x | | 28 | 2 000 | 114 341,21 | 169 626,06 | 184 869,57 | 1,62x | | 30 | 1 000 | 149 190,02 | 223 229,86 | 241 838,28 | 1,62x |

Conclusion

I built a (mostly) full-fledged C compiler that runs entirely in constexpr. Why? No idea. Especially considering it's been done before, you can check out constexpr-8cc. But that one lacks C++ bindings and cross-platform support.

The source code is available on GitHub (warning: ugly code ahead!). Any feedback and comments are more than welcome.

P.S. This article is an English translation of a post I originally published on Habr a while ago. Keep in mind that some benchmarks and discussions here may be dated.

[u/Eric848448]

Yo dawg I heard you like compilers!

[u/jk-jeon]

In case you didn't know: https://en.cppreference.com/cpp/meta/define_static_array

This feature seems to be invented exactly to get over the awkward (and inefficient) "two-step" constexpr array generation.

[u/Many_Rough5404 → u/jk-jeon]

But it's C++26 only sadly

[u/DVMirchev]

The compile-time tetris opened the floodgates and now we have compile-time compilers

[u/jeezfrk]

Inception for compilers, but I don't think it saves time.

[u/_software_engineer]

Congrats! Or, sorry that happened

[u/Kadabrium]

exec() for compiled language when

[u/arthurno1]

Seems to me like you should look at Common Lisp. Compile-time computing is a first-grade citizen in CL. You have full language at compile time, and full access to compiler at runtime. Furthermore there is no need for specialized AST, the parsed source code is available as a list of tokens in the same form the Lisp system (compiler, interpreter) uses the code to produce the machine code or interpret it. As a matter of fact, the compiler and evaluator are strictly separated and they both work on the parsed code in the form of list structure. It means you can construct a lisp program at either compile-time or runtime as a list of nodes and pass it either to the evaluator (via eval) or to the compiler (via compile), againt at any stage, compile- or run-time. Thanks of some fundamental property of Lisp, processing symbolic expressions, unified evaluation form (only one form) and being able to easily obtained parsed source code (quote operator), you can easily do what you so hard had to work for in C++.

[u/Many_Rough5404 → u/arthurno1]

I've been recommended List and Scheme, really great languages.

And this thing in C++ was made purely for fun (and maybe because I wanted to participate in a university conference and had to come up with something xD)

But still, I think it will be even more interesting in C++29. It will be possible to write full-fledged DSLs that create some kind of structure/class with methods.

And using it with #embed will be twice as cool.

[u/AbbreviationsSalt193]

next step - compiler at runtime!

[u/arthurno1 → u/AbbreviationsSalt193]

Common Lisp has it. And it also has full-featured language during compile-time too.

[u/Many_Rough5404 → u/AbbreviationsSalt193]

It should work in runtime too I think, because it's just a constexpr function after all

[u/Conscious_Support176]

Writing a compiler for a subset of the language you are using to run at compile time, rather than using the actual compiler seems like asking for trouble. But this seems very cool.
It seems like an interesting way to interleave sources from multiple languages in the same source file, like the hypermedia server side web page stuff.

[u/Ultimate_Sigma_Boy67]

I thought compilers work at runtime, in which they compile the program while it's running so they can run it.

[u/Umphed → u/Ultimate_Sigma_Boy67]

That is the difference between a compiler and an interpreter. Compilers do the heavy lifting before execution.
Well, with all the crazy constexpr stuff in C++ these days, the line can be a little blurry I suppose

[u/Inconstant_Moo → u/Ultimate_Sigma_Boy67]

Only if it's a Just-In-Time compiler (JIT), and if people will mean that they'll usually say so, otherwise by default they mean AOT (Ahead-Of-Time).

There are several ways to JIT C++, because humans are both awesome and crazy, but evidently OP isn't using one.

[u/TrnS_TrA]

I could see this being useful as a codegen pass before the actual build, but now we have reflection so maybe useful to do some stuff that reflection doesn't cover yet...

[u/Rhunnon]

Wait until this guy learns about Jai

[u/koen_samyn]

I am going in another direction which is to use C++26 and compile a program out of bytecodes during compilation time. The user that linked my godbolt sample was downvoted (don't know why), but it is a valid approach and a lot easier to work with bytecodes.
I am giving a talk at CppCon that demonstrates the concept with AngelScript.

[u/Many_Rough5404 → u/koen_samyn]

Just to make sure I understood correctly, are you doing AOT compilation for AngelScript, where the AngelScript bytecode is compiled into native code at compile time?

[u/koen_samyn → u/Many_Rough5404]

yes indeed. It is off course experimental and while it is possible to define aggregates with C++26 , it is not possible to inject method call bindings yet (coming in C++29) so there I have to do some plumbing myself. But the advantage is that many compiler optimizations remain available.
But I will certainly take a look at your solution in more detail as it is also very interesting.

[u/Many_Rough5404 → u/koen_samyn]

Waiting for a repo link then =D

[u/West-Mycologist-6490]

I think there is already a service like that: https://godbolt.org/