Lite: An Implementation Overview

Introduction

This write-up outlines some of the implementation details of the lite text editor. At the time of writing this lite is version 1.06.

Overview

Lite is written mainly in Lua (5.2) with the lowest level parts written in C. It is written with the intent of being lightweight, easy on the eyes, responsive and trivial to modify and extend. To achieve this the core implementation aims to be as simple as possible: complexity is only brought in when it's proven the simplest method isn't practical.

Core

The core of lite implements documents, syntax highlighting, a cooperative-threading system, basic logging and the general ui layout system. Overall the core is in charge of handling user input, running per-frame tasks, and rendering the resultant frame early enough to meet the frames-per-second specified in config.fps.

Threads

Lite implements a cooperative-threading system where by background tasks — such as full-document syntax highlighting and updating the internal list of project files — can be performed incrementally. Cooperative threads in lite are implemented using lua's coroutines and are referred internally as simply threads.

A thread can be started by using the core.add_thread() function, this function takes a function, and an optional object which is used as a weak reference to that thread, eg. if a doc is passed as the weak reference, when that doc is garbage collected the coinciding thread will also cease to exist.

As threads in lite are cooperative they're expected to regularly call couroutine.yield(). Typically this yielding is done every n lines for line-based tasks (eg. full-project search, syntax highlighting). When a thread yields the core will continue to render the frame, or — if there is still enough time left in the current frame — pass control to another thread. Threads can also pass a numerical argument to coroutine.yield() to wait for the number of seconds specified:

core.add_thread(function()
  -- print "hello world" every 10 seconds
  while true do
    print("hello world")
    coroutine.yield(10)
  end
end

Practical usage examples of lite's threading can be seen in the core's project file scanning (data/core/init.lua) and the incremental syntax highlighting (data/core/doc/highlighter.lua).

Documents

All loaded text files in lite are stored in Docs. A doc keeps a table of a text file's lines (Doc.lines), undo and redo state, the current syntax (Doc.syntax), syntax highlighting state (Doc.highlighter) and caret/selection state. Docs are opened using the core.open_doc() function; if the function is called and a doc of that filename is already open then the existing doc is returned such that more-than-one doc representing the same file should never be present.

Lite stores all the open docs in the core.docs table; each frame lite checks the number of views referencing each doc and releases those with zero references — this is an instance of lite favoring polling over callbacks or event listeners. This approach is done throughout lite; in avoiding event listeners we also avoid having to manage unregistering event handlers or worry about having those which are never unregistered through error. This leads to much simpler, less error-prone code.

Syntax Highlighting

Lite implements a simple syntax highlighter based around lua patterns, this is split up into 3 parts:

• core.syntax: stores language syntax specifications
• core.tokenizer: converts text into table of tokens using a given syntax
• core.doc.highlighter: stores token state for lines of a doc and handles full-document incremental highlighting

A language syntax is stored as a table of lua patterns mapped to to their resultant token type (.patterns) and a table of symbols mapped to their token type (.symbols). Patterns in the pattern table can also be a range of the following format:

{ range_start_pattern, range_end_pattern [, escape_character] }

Ranges can be used for matching strings:

{ '"', '"', '\\' }

An example of syntax specifications can be seen in any of the data/plugins/language_*.lua files.

The highlighter persists range state between lines by storing the pattern table index of the current range. As the highlighter works in increments of lines, ranges are useful for multi-line comments or strings which a single pattern would fail on.

When the tokenizer.tokenize() function is called it iterates the syntax's pattern table from top to bottom, such that patterns higher up in the table take precedence. When it matches a pattern it pushes the token to the resultant table, if the token's content matches any of the keys in the .symbols table that value is used instead of the matched token value; this acts both as a more convenient way of specifying keywords as well as an optimization as the tokenizer can attempt matching fewer patterns, even if there are many keywords specified. The function returns a state integer which is used to indicate which pattern range it is currently inside if it is inside any.

core.doc.highlighter is in charge of providing tokenized lines when displaying a document; each doc stores a Highlighter object. The highlighter assures highlighting information for any line that is requested is always available, even if the full-document incremental highlighter has not yet reached that line.

When a request is made to the highlighter for a specific line the highlighter first checks its cache for the line; if it does not have the line cached — or, if the line's text has changed since its cached version — it immediately tokenizes the line, continuing from the tokenizer state left from the previous line. The highlighter's max_wanted_line field is set to that line's value if it's greater than the existing max_wanted_line.

A doc's highlighter creates a thread when it's created which it uses to do full-document highlighting starting from its first_invalid_line up until the max_wanted_line. When a change occurs in a document the first_invalid_line is set to the first line of the change if it's greater than the current first_invalid_line.

This implementation results in the following characteristics:

• Displayed lines are always syntax highlighted with, at the very least, highlighting which is valid to the current screen's content
• Displayed lines are eventually highlighted with full-document aware highlighting
• The highlighting is performed per-line, and so — with the exception of extremely long lines — can be done incrementally without a negative effect on responsiveness
• Highlighting is only performed up until the point in the document it is required, for example, up until the last visible line

User Interface

Each part of lite's UI is implemented as a View, that is, a document is shown in the UI as a DocView, the status bar at the bottom the StatusView etc. Each view exists within the RootView (core.root_view) which is in charge of calling the :update() and :draw() functions of each view, managing the layout of the views, and routing user input to relevant views.

The rootview is implemented as a binary tree of Nodes. Each node is one of the following types:

• hsplit: Contains two children side-by-side and a divider between them
• vsplit: Contains two children one-above-the-other and a divider between them
• leaf: Contains one or more views, and, if there is more than one view, a tab at the top for each view.

Each frame the rootview calculates its layout based on the divider position of each split and sets the .size field of each node and view accordingly.

The rootview also has a concept of locked nodes; the size of a locked node is not determined by the divider position but instead is set from the view's .size field. Locked nodes cannot be resized by the user as normal nodes would be, they cannot be closed and cannot open additional views. The locked-node feature is used for, amongst other things, the status bar at the bottom of the screen which we typically want to be always present and immutable.

lite takes the approach of effectively redrawing everything whenever it needs to redraw anything, this lends itself to simplified UI code. There are no event listeners or callbacks to manage. If something changes the state of the UI (eg. the caret is moved), the core.redraw boolean is set to true to signify that something has changed and the UI is redrawn on that frame. For further simplicity any user input events will also set core.redraw, such that there's a guarantee any change which resulted from user input will result in the screen contents being updated.

Although the idea of redrawing everything each time may appear wasteful, lite employs a technique where by only regions of the screen which have changed between frames are actually redrawn — these regions can be displayed by calling renderer.show_debug(true). This technique is implemented on the C level of the code and is invisible to the lua portion which acts as if it's doing a full redraw each time.

An overview of how this is achieved is detailed in the Cached Software Rendering write-up.

Plugins

Lite's plugin system is based around lua's module system; each plugin is simply a lua module (that is, a single .lua file, or, a directory containing a init.lua file). All modules in the data/plugins directory are automatically loaded when lite starts — this is done after initializing the core and before loading the user and project modules. Thus to install a plugin you simply place the file in the plugins directory; to uninstall a plugin you delete the file.

A specific order of which plugins are loaded is not guaranteed by lite but, as plugins are just regular lua modules, a plugin can simply call require on another plugin if it needs to assure that plugin is loaded before it, or depends on functionality from that plugin.

The lua which plugins are made from and that which lite is made from are no different, effectively giving plugins the ability to modify lite in almost any way the user sees fit. Instead of implementing an event listener system which would both litter the implementation with code to emit events, as well as limit the places lite can be modified to those which emit these events, lite instead encourages plugins to simply override existing functions directly. For example, if we wanted a plugin to run our build.sh script each time a document is saved it could be implemented as such:

local Doc = require "core.doc"

local save = Doc.save
function Doc:save(...)
  save(self, ...)
  system.exec("./build.sh")
end

Plugins are free to override drawing functions (for drawing lines under words), keymap handling (to implement modal editing), create their own views (for project-wide text search) or any other part of lite.

A plugin is never expected to be able to unload itself, instead assuming that lite is restarted for cases where the user wants to remove or add a plugin. No care has to be taken to properly keep track of what changes a given plugin has made as those changes will never need to be undone.

Plugins can be downloaded from lite's plugins repo.