[In General] How are OSes made?

[Curious Gorge]

I was thinking about a question I asked last week that someone answered very well. The person said that no program can interface with the hardware in the same was as the OS.

That statement prompted another question and what better place to ask it than here, right?

Well how is it Operating Systems are made? I only want a general answer. I know most OSes are written in plain ole' C, but what I don't understand is how you develop code for hardware which does not have an OS.

How does a person even get the code on to the hardware?

I would assume that typically an OS does the management of the computers components, so how is this done?

[Ancient Dragon]

There's generally two layers of software -- os kernel which has direct access to all hardware, and application programs which are protected from direct hardware access. What you are asking is how to write os device drivers, which run at the kernel level. For MS-Windows you will need these tools. Here are some other links and tutorials to get you started.

[rubberman]

If you want to know exactly how to create an operating system from scratch, there are a number of texts out there to help teach you, such as Andrew Tanenbaum's "Operating Systems: Design and Implementation" - where he shows how the Minix operating system works. This was, as I understand it, the inspiration for Linus Torvalds to create Linux. Since all Linux source code is available to study and adapt, that is another source (so to speak) to draw from.

In any case, to integrate operating system code to various processor architectures, you will need to also be able to write some (not a lot) of assembler language for the target processors. Some things require very low-level access that higher-level languages, even C, don't provide.

The process of getting code onto the hardware is called "bootstrapping".

Your questions are good, but the answers are too complex to cover here in a single post. :-) Try to keep your questions focussed/narrow and they will be easier to answer in a reasonable frame.

BTW, another open source operating system is FreeDOS, an MS-DOS clone. That may be another place to start in your OS explorations!

[mike_2000_17]

Another open-source operating system you might want to look at if you are interested in this is the FreeRTOS project. This is a very small real-time operating system designed for embedded system (micro-controllers, i.e., very tiny computers on a single chip). This might be an easier introduction to the practical aspects of OS development, because the source code is only a few thousand lines of code (as opposed to Linux which has 15 million lines!).

But one thing is for sure, OS code is never pretty. It's a long and tedious sequence of bit-fiddling and book-keeping.

how is it Operating Systems are made?

Obviously, there is far too much here to explain it all, and the details are far beyond my own knowledge of the subject. But, essentially, operating systems are written like any other library or application, except that you have almost nothing to begin with. Without an operating system, you don't have file I/O, you don't have threads, you don't have peripherals of any kind, you don't have dynamic memory allocation, you don't have any protections (such as preventing access to the wrong memory), and so on... this means that every little task can become quite tedious, i.e., very "low-level". But, by definition, the code is kind of "simple", close to the metal.

Generally, the architecture of an operating system has many parts, one for each of the main "features". But the most important concept is that of kernel-space vs. user-space. An operating system would generally be composed of a background program and a set of libraries (that can be called). The kernel code runs on bare hardware and can thus manipulate memory directly and things like that. We call that privileged realm the "kernel-space". The operating system hosts the execution of many (or just one) processes, i.e., applications or programs. These programs rely on the operating system for everything they do that pertains to files or memory allocations or any other "feature" that the OS provides. We call that realm the "user-space". The "users" (or programs) interact with the kernel by calling functions of its library components, which, in turn, accomplishes those tasks in the kernel-space, and thus, we call them kernel-calls.

what I don't understand is how you develop code for hardware which does not have an OS.

Well, as you are probably realizing by now, a computer architecture is quite a huge layered cake. There are layers and layers and layers. And there are plenty of layers below the operating system too. There is firmware, which are just small programmable integrate circuits with a fixed (or "baked in") program that just does some very simple tasks. There are even smaller PICs that can barely execute programs of more than a few lines of assembly code. There are normal integrated circuits (non-programmable) that perform some very basic logic. Basically, when you look at your computer's motherboard, there are all these chips all over the place on that circuit board. Well, they each serve a purpose. And most of them make it so that accessing hardware and buses is relatively easy and standardized. On PCs, you have the BIOS as a fairly feature-rich piece of firmware that can actually do a lot (HDD I/O, simple graphics, run the PCI bus, etc..), which gives you a significant head-start compared to other much less feature-full environments (such as micro-controllers).

This all contributes to making it so that you can run code, useful code, without the need for an OS. But just to give you an idea, this allows you to do things like reading and writing raw memory from RAM or HDD, or communicate with and discover devices on the PCI bus or other buses.

It's all a matter of building things up layer by layer. The lower you go, the more tedious and simple it gets. For example, on a micro-controller that I worked with a while back, to do the communication with a serial port (USB-like port), I would first have to set a register (a fixed piece of memory on the chip itself) to some specific value that links the output pin with the clock pin and thus, enables the transmission of the output bit (high-low) at each clock tick. Then, I would change the value (0 or 1) of that output pin between each clock-tick to send each bit on the port, and then reset the register when 8 bits had been sent, and thus, completing the transmission of one byte through the serial port. That's not a complicated piece of code to write, but it's tedious. And once you can send one byte, you can send as many as you want, and then, you have a working serial / USB port. And that's how it builds up. But this is for a small micro-controller... PCs have a lot more powerful firmware around it that takes care of those extremely tedious details.

The way you develop code for some "bare" hardware is essentially through manufacturer-provided libraries (for some of the basic, tedious stuff) and through a cross-compilation environment (and simulator). Cross-compilation means that you compile a program for a platform other than the one you are currently using. So, you code and compile on your computer, but you target some other hardware, and so, the produced code will work on that hardware. Then, if you want to test that code, you will generally use a simulator, like a virtual machine or an emulator, which are basically programs that simulate the real hardware and can execute programs targeted for it.

How does a person even get the code on to the hardware?

Well, if it's a PC, then it's easy, because the BIOS is already capable of booting up a complete operating system. And so, all you need is to put your "code" on the hard-drive (or USB or CD) and boot up the computer, and that's it. For micro-controllers and stuff, things get a bit more complicated. But again, it's just more low-level. For example, a typical micro-controller will have a boot-loader that loads up before your own program, and if you set a jumper to the correct position, it will boot-up in a kind of "ready to receive program" sort of state, and at that point, you just "flash" the program through the USB / serial port. Whichever the way you have to do it, it's not "magic", it's quite simple actually.

an OS does the management of the computers components, so how is this done?

Most components are plugged into the rest of the computer through some "bus" (an internal communication channel), and they generally communicate through simple protocols (which are usually, semi-standardized). These are simple "give command, get reply" protocols. Like, "command the HDD to read N bytes from physical address X" and receive the N bytes as a reply. It's mostly that simple. The OS kernel will contain (or integrate) a number of device drivers whose job it is to provide the functions that users need (users in the "user-space") by communicating with the hardware component, through the bus, using that protocol. The protocol and the features are often vendor-specific, which is why vendors are generally responsible for providing the drivers. That's about all there is to it. I've written drivers before, it's really not much more than that (when the hardware is simple).

That is about as much detail as I can provide, it's pretty much all I know, and about as far/deep as I know.

[fliponymous]

The operating system is, simply, what empowers your software to manage the hardware. Clearly some OSes are more sophisticated than others.

At its very core, a computer starts executing at a fixed address, meaning that when the computer starts up, it sets the program counter to a pre-defined address, and just starts executing machine code.

In most computers, this "bootstrapping" process immediately initializes known peripherals (like, say, a disk drive). Once initialized, the bootstrap process will use some predefined sequence to leverage those peripherals. Using the disk driver again, the process might read code from the first sector of the hard drive, place it in a know space within RAM, and then jump to that address.

These predefined sequence (the start of the CPU, the loading of the disk) allows the programmers to star adding more and more code at the early parts of the CPU startup, which over time can, eventually, start up very sophisticated programs.

In the modern world, with sophisticated peripherals, advanced CPU architectures, and vast, vast resources (GBs or RAM, TB of Disk, and very fast CPUs), the operating system can support quite powerful abstractions for the developer (multiple processes, virtual memory, loadable drivers, etc.).

But for a simple system, with constrained resourced, you don't really need a whole lot for an "OS".

As a simple example, many small controller computers have very small "OS"es, and some may simply be considered a "monitor", offering little more than easy access to a serial port (or a terminal, or LCD display). Certainly, there's not a lot of needs for a large OS in these conditions.

But also consider something like a classic Forth system. Here, you have a system with an "OS", that gives you disk I/O, console I/O, memory management, plus the actual programming language as well as an assembler, and this fits in less than 8K of memory on an 8-Bit machine.

or the old days of CP/M with its BIOS and BDOS.

CP/M is a good example of where a simple OS works well as a abstraction layer to allow portable programs to run on a vast array of hardware, but even then the system took less than 8K of RAM to start up and run.

A far cry from the MBs of memory used by modern OSes. But, to be fair, we HAVE MBs of memory, and our lives are MUCH MUCH simpler (mostly), and far more functional, because of it.

Writing an OS is fun because it's interesting to make the HARDWARE print "Hello World" shoving data 1 byte at a time out some obscure I/O port, or stuffing it in to some magic memory address.

Get a x86 emulator and party down getting a boot sector to say your name. It's a giggly treat.

[Curious Gorge]

I don't have nearly enough time to read all of that but thanks you guys are very helpful! I will finish reading these posts later on. I placed the words [In General] in the subject line because I just wanted a general answer, and you guys were more than thourough so thank you!