in Courses

Building a Transpiler from scratch

Available coupons:

Course overview

In recent compilers implementation, it became popular to translate from one high-level language to another high-level language. Examples might be desugaring new version of JavaScript to an older version of JavaScript, or languages like TypeScript, etc. Such high-level compilers are known today as transpilers, and what is the topic of our course.

It proved to be very practical, since we can fully rely on an existing runtime such as a JavaScript engine, avoid dealing with lower-level constructs like memory, bytecode instructions, etc.

If you would like to explore the world of compilers and implement your own programming language, starting from the high-level compilation is a good hands-on practical approach, and which we apply in this class.

Often related books on compilers go to theoretical aspects, stucking in parsing, regular grammars, etc – not explaining how actually to build a practical transpiler. I believe we should be able to build a transpiler for a full programming language, end-to-end, in 2-4 hours — with a content going straight to the point, showed in live coding sessions as pair-programming and described in a comprehensible way.

In the Building a Transpiler from scratch class we focus on compiling a high-level concurrent programming language with message-passing processes, down to JavaScript. That is, in addition to the deep dive on the transpiling, you will get a better understanding of how concurrent functional languages, such as Erlang, or even an operating system itself with its processes and threads, work today.

Implementing a transpiler will also increase your engineering level, as it touches several aspects of data structures and algorithms.

How to?

You can watch preview lectures, and also enroll to the full course, covering implementation of a transpiler from scratch, in animated and live-annotated format. See details below what is in the course.

Available coupons:

Prerequisites

An optional prerequisite for this class is the Building an Interpreter from scratch (aka Essentials of Interpretation) course, where we build an AST-interpreter for a full programming language. Unless you already have understanding of how programming languages work at this level, i.e. what eval, a closure, a scope chain, environments, and other constructs are — it is recommended to take the interpreters class as a prerequisite.

Watch the introduction video for the details.

Who this class is for?

This class is for any curious engineer, who would like to gain skills of building complex systems (and building a transpiler for a programming language is an advanced engineering task!), and obtain a transferable knowledge for building such systems.

If you are interested specifically in compilers, PL and Type theory, and want to build a transpiler for your programming language, this class is also for you.

What is used for implementation?

We build the transpiler in JavaScript, compiling to JavaScript.

JavaScript, being the most popular programming language, should be accessible for many engineers, and also our goal is to provide a simple and concise implementation, not focusing on specifics of a host language. So the code should be portable to any language of your taste and choice: TypeScript, Rust, OCaml, C++, Python, etc.

Note: we want our students to actually follow, understand and implement every detail of the Transpiler themselves, instead of just copy-pasting from final solution. Even though the full source code for the transpiler is presented in the video lectures, the code repository for the project contains /* Implement here */ assignments, which students have to solve.

What’s specific in this class?

The main features of these lectures are:

  • Concise and straight to the point. Each lecture is self-sufficient, concise, and describes information directly related to the topic, not distracting on unrelated materials or talks.
  • Animated presentation combined with live-editing notes. This makes understanding of the topics easier, and shows how the object structures are connected. Static slides simply don’t work for a complex content.
  • Live coding session end-to-end with assignments. The full source code, starting from scratch, and up to the very end is presented in the video lectures
What is in the course?

The course is divided into four parts, in total of 18 lectures, and many sub-topics in each lecture. Below is the table of contents and curriculum.

Part 1: Transpiler pipeline

In this part we start talking about transpiler pipeline, parsing, code generation, and exploring AST nodes.

  • Lecture 1: Transpiler overview | Numbers
    • Course overview and agenda
    • Parsing pipeline
    • Tokenizer module (Lexical analysis)
    • Parser module (Syntactic analysis)
    • Abstract Syntax Tree (AST)
    • Transpiler pipeline
    • Eva MPP language overview
    • Concurrent process execution
    • Message-passing mechanism
    • Setting up implementation
    • Building number transform

  • Lecture 2: Strings | Blocks and Statements
    • Strings implementation
    • Expression | Statements
    • Blocks: groups of statements
    • AST Explorer
    • ExpressionStatement
    • Semicolon statement delimiter
    • Pretty-printing
    • Program as implicit block

  • Lecture 3: Parsing: S-expression to AST
    • S-expression grammar
    • Syntax tool
    • Parser generator
    • Saving compiled code to file
    • Build system

  • Lecture 4: Variable declaration | Assignment expression
    • Variable declaration
    • Variable access
    • Assignment expression
    • Naming convention

  • Lecture 5: Function calls | Runtime
    • Call expression
    • Passing arguments
    • Runtime module
    • Native functions

  • Lecture 6: Binary and Unary expressions
    • Math operators
    • Binary expressions
    • Comparison operators
    • Logical expressions
    • Unary expressions

  • Lecture 7: Control flow: If statement | While loops
    • Control flow
    • If statement
    • While loops
    • Syntactic sugar
Part 2: Functions and Processes

In this part we focus implementing our runtime components, such as Process and Scheduler classes, and discuss cooperative multitasking.

  • Lecture 8: Function declarations | Return statement
    • Function declarations
    • Return statement
    • Automatic function block
    • Process functions
    • Spawn function

  • Lecture 9: Primer on generators
    • JavaScript generators
    • Async generators
    • Yield statement
    • Cooperative multitasking
    • Process scheduler
    • Round-robin scheduling
    • Run queue
    • Promises
    • For-await loop

  • Lecture 10: Process | Scheduler class
    • Process class
    • Scheduler class
    • Global scheduler
    • Runtime functions

  • Lecture 11: Spawning processes | JS Transform
    • JS Transform
    • Generator functions
    • Spawn calls
    • Cooperative multitasking
    • Preemptive multitasking
    • Context switch
    • Yield statements
Part 3: Pattern matching

In this part we implement packed data structures, such as Lists and Records, and also build support for pattern matching.

  • Lecture 12: Records and Lists
    • List values
    • Record structures
    • Member expression
    • Elements access
    • Property access

  • Lecture 13: Pattern matching | Decomposition
    • Destrucruring assignment
    • Value match
    • Pattern match
    • Catch-all match
    • Binding allocation
    • If-tests

  • Lecture 14: Pattern matching statement
    • Extracting variables
    • Try-catch wrapper
    • Nested patterns
Part 4: Message passing

In the final part we implement the message passing mechanism, do a case study analysis, using the compiled code from Web, and build the final executable.

  • Lecture 15: Awaiting messages | Process mailbox
    • Send command
    • Receive command
    • Process mailbox
    • Awaiting messages

  • Lecture 16: Receive statement | Recursive calls
    • Receive function
    • Recursive process calls
    • Delegating yield
    • Await expression
    • Syntactic sugar

  • Lecture 17: Using compiled code from Web
    • Case study
    • Concurrent processes
    • Compiling for Web
    • Self argument
    • Sleep call
    • Compiler optimizations

  • Lecture 18: Final executable | Next steps
    • Transpiler CLI
    • Compiler optimizations
    • Tail recursion
    • Non-recursive process loops
    • Next steps

I hope you’ll enjoy the class and will be glad to discuss any questions and suggestion in comments.

Write a Comment

Comment