The EuOS Kernel

By Jeffrey Fielding

August 20, 1999

Before I write the kernel, I think it's a good idea for me to describe all the APIs and internal workings of the kernel so that 7it is easier for programmers (including myself) to continue the development of EuOS and programs for it.

The purpose of this document is to define the basic design of EuOS - specifically the kernel. The APIs and source code are documented elsewhere.

If you have anything to add to this document, please e-mail me and I'll update it.

Goals

First, I will describe my goals in the design of the EuOS kernel. I want (and I think most programmers will agree with me) a powerful, secure, stable, simple, and extendible operating system. It to be the simplest operating system ever for the average user. Also, the power user should be able to extend almost every part of the operating system. The kernel and the general design of the operating system should make it virtually immune to malicious programs. It should also be at least as stable as Linux, and it should warn the user before running unstable programs.

Reaching the goals - basic operating system design

Security

Euphoria is great for security for three reasons. First, it allows data to be protected. Data that isn't declared as global is not at all accessible by other programs. Second, it is high-level and provides only a few system functions. Finally, it allows built-in functions to be overridden.

The kernel will override all system functions and provide a safe implementation by looking-up the privileges of the currently running program. Programs are by default highly restricted unless the user says otherwise by modifying the privileges or by installing a secure program from a trusted source.

Programs are even restricted when it comes to file access. By default, programs can only access their directory. The user can allow the program to access other directories, such as the documents. The user can also disallow programs from deleting files or make the operating system keep a backup whenever a program changes a file. System files are locked out of every program, and are only accessible by the operating system.

Stability

Unfortunately, this is going to be a problem until there are some changes to the Euphoria interpreter. Since the programs are interpreted by the same interpreter as the kernel, when a program crashes, the whole operating system crashes. This is why the operating system will be extra careful with stability.

If the operating system detects an error, it should do its best to recover the program that generated it. If there is no way to prevent the program from crashing, the operating system should store whatever data it can about the program's current state before killing it. If the program supports it, the kernel will later re-start the program in restore mode and give it the backup data.

Also, programs give the operating system a stability index when they are loaded. The operating system can then warn the user or backup data before running the program if the program is unstable.

The last error, if any, is stored in the thread's data structure until checked by the getLastError() function. getLastError() returns 0 if there was no error, or a positive integer constant defined in the kernel specifying the condition.

Simplicity

Everything should be well documented and conform with the basic Euphoria coding standards and maintain the elegance of Euphoria. Here are some basic rules that the code should follow:

• Routine and variable names are lowercase except for the first letter of each word (except for the first). For example, addProgram. When the words are abbreviated, such as getc, the whole name is lowercase.
• Constant names are all upper case, and words are separated by an underscore (_).

Extendibility

Virtually every feature in EuOS will have some way of being extended. Each extendible feature also has a security level that is required for a program to extend it. The kernel will use routine_ids for extensions, so it will be very easy to dynamically add and remove extensions.

Power

All of these features make EuOS a very powerful operating system. Although it might not be as fast as other operating systems (like Linux - it could quite possibly be faster than Windows), it will still be fast enough. As processors get faster, it is more important to have good software than highly optimized software. Still, some routines may be written in assembly for speed, and programs with privileges will be able to use assembly routines. The operating system will also provide routines for creating assembly routines, and it will check the routines as they are created to make sure they are safe.

General Description of the kernel

Multitasking

For now, there isn't much choice. EuOS has to use a cooperative multitasking task management system unless the interpreter is changed to support multitasking. Each program is divided into one or more small functions. The functions are called by the kernel and return the routine_id of the next function to call, or -1 to end the thread or task.

Tasks have only three priorities to make a cooperative multitasking scheduler like that in ESE. The lowest priority is sleep, where a task is suspended until its priority is changed. Threads with the highest priority are like event generators in ESE - each is run once for every time that a normal priority task is run.

Another important factor in multitasking is handling events. An event-based system allows the operating system to free-up processor time that would otherwise be spent waiting for an event. It also adds a layer of abstraction to keep the programs simple and let them work together.

Events and multitasking work a lot like in ESE, but with some differences. First, event handlers must accept a single sequence as an argument, not an unlimited number and type like in ESE. This will allow the events to be extended without making older programs obsolete. Also, all event handlers are procedure and run completely. This means they should either be very fast, or start a new thread with a longer handler.

Finally, event generators and multitasking event handlers are replaced by threads with the only difference between the two being their priority. High priority threads must be very fast, and normal priority threads should be fast, but don't need to be as fast as high priority threads. I suggest that high priority threads like device drivers install an interrupt handler if they are allowed to and check a value in memory instead of requesting the status of the device each time if possible.

Threads are called without any arguments. They can read and write data using kernel routines. This allows data to be unique to each instance of a task or thread, and allows the kernel to backup the data.

Interprocess Communications

Although most interprocess communications will occur through events, it is also possible to use pipes by having one program register a virtual file, and another opening it (I will explain more of the details of the file system later). The kernel provides routines for creating an event for communications between programs, and routines for finding the event for another program. Then a program can communicate with another program by firing that event with whatever parameters necessary, and possibly can establish a separate event for a separate communications channel.

Memory Management

Since most data is stored by the Euphoria interpreter, there is no need for complicated memory management interfaces in the kernel. The memory management will depend mostly on the "distribution or the interpreter on which the operating system is running, and not on the kernel. The kernel does, however, support virtual memory when programs use the allocate and free routines.

When a program calls allocate, the kernel tries to allocate the requested block. If it can't get a big enough block, it frees previously allocated and unlocked memory after saving it to a file and retries. If there isn't any unlocked memory big enough to fit the request, it returns 0. Note: the kernel keeps track of memory allocated to each task. It will free any memory used by a task when the task exits.

Special Programs: Drivers, Daemons, Libraries, and Shells

In EuOS, there are several different program types. The usual program type is a program. When the user runs a program, the program creates some threads, possibly runs operations on input, and usually outputs something.

A driver handles input and output, and possibly processes it somehow. It provides a main routine like a program, but the main routine only registers the driver with the OS. The main routine can also set any initial values that are used by all instances of the driver.

Another important thing to note about a driver is that it is extendible. For example, there is a standard keyboard driver that should work with most computers, and this driver is used by default for all keyboard operations. If there is another type of keyboard that isn't supported by the standard keyboard driver, another driver can extend it so that any existing program can use the new keyboard driver. Drivers also can have multiple instances for multiple devices. They must provide an initialization routine to initialize the device and any instance-specific variables, a set of routines to access the device, a set of threads that poll the device and generate events, and a routine to close the device.

A demon is a program that runs in sleep mode most of the time and just listens for an event. The main routine of a demon initializes a new instance of the demon (sets any variables, processes any command-line arguments, and creates one or more event handlers), then returns -2 (sleep mode).

A library is just a library - it registers all of its routines with the kernel in its main routine, then does nothing until a program calls it. Libraries have only one instance, but they can act like they have multiple instances by storing separate data for each program.

From the user's point of view, a shell is a user interface. From the programmer's point of view, a shell is a powerful mix of programs, demons, and drivers. A shell is basically responsible for redirecting the input and output of programs to the correct device and processing requests by programs. It is also responsible for running programs.

There are two different types of shells: a text based shell and a GUI shell. A text based shell can start a GUI shell, but the text based shell's input and output are then turned off until the GUI shell finishes. A GUI shell can start a text based shell, but in a window. When a text based shell tries to start a GUI shell when it is already in a GUI shell, the GUI shell returned to the text based shell is the current GUI shell. A text based shell can therefore run a GUI program, and it will automatically start a GUI shell. Likewise, a GUI shell can start a text based program, which will start in its own text based shell window.

File System

The file system, like much of the rest of the kernel, is designed to be extendible. The kernel has no specific implementation of the file system. It only has an interface and routines for extending that interface. Most of the file system code will be distribution-specific, but it will be completely compatible between distributions.

The file system is a lot like Linux in that there is a root directory onto which everything is mounted. The actual format of file names is not important to the file system. Anyone can install a new parser for a new path format, or they can use the default Linux-like path format of /directory/subdirectory/file. This is built into the kernel and is the default unless otherwise specified. The actual internal representation used by the kernel is a sequence of the form {directories,file}. For example, /directory/subdirectory/file is represented as {{"directory","subdirectory"},"file"}.

The most basic extension of the file system is the ability to create a virtual file or directory. Any program can create a virtual path, but only privileged programs can create a virtual path that overwrites an existing path. No program can create a virtual path in the system directory, or overwrite the system directory - only the OS and the distribution are allowed to do that.

Two other very useful extensions are based on virtual files and directories. The first and most important is the mounter. A mounter is a driver that can mount a device onto the file system, like in Linux. Mounters register themselves with the operating system, and provide routines to mount, unmount, verify, fix, and optimize devices. The verify routine checks whether the specified device is supported by the mounter. The fix and optimize routines are optional. Fix performs a scandisk-like operation, and optimize tweaks the file system to speed it up. This might include defraging it, for example. When a mount command is issued, the kernel tries to mount the specified device using one of the mounters. It will try to autodetect the file system using the verify routines unless it is specified.

The second extension is one which I haven't seen implemented in any other operating system before. This extension tries to eliminate problems with file types. There are three parts to this. First, the operating system stores the file type of each file in a file. Next, when a program tries to open a file, it can open it for one of the default modes, or it can specify in place of the mode a mode followed by a desired type.

The third part is an interface for file conversion programs. These file conversion programs register themselves with the kernel, and announce which routine should be used to convert some file type to another. The kernel then records this data, and when a program tries to open a file as a different file type than it actually is, it searches backwards from the desired type to the source type, and runs a series of conversions on the data. This ties in with virtual files because the kernel tells the first routine to make a virtual file of the source file in a different format for the next routine. The kernel tells the next routine to make the virtual source file another virtual file for the next routine etc. until it gets to the program, which is given a virtual file in the format it desires. If the kernel is unable to find a conversion, it returns an error.

User Interface

The user interface is one of the most difficult to design parts of the operating system - especially in EuOS. Since EuOS allows multiple users, it is necessary to keep track of each user, his or her choice of user interface (text or GUI, and which one), and all the I/O resources used by that user (mouse, keyboard, video, and possibly more in the future). On top of that, the operating system has to track the programs the user is running.

All of the above is simple compared to the task of designing the interface that lets all programs work with all GUIs (except for GUI programs on a text interface). Also, the controls must be extendable just like the rest of the operating system, so the operating system must keep track of these too.

In a user interface, there are four elements - the program, the shell, input, and output. In a GUI shell, there is a fourth, the controls. Controls include buttons, text boxes, windows etc. In EuOS, controls are represented as drivers which extend the Control driver (see the APIs).