The Basics of Information Theory Writings index       Home

Suppose that you want to tell someone something, that is, convey to them some information.

A Single Two-way Choice

Further suppose that you are not using English or some other natural language, but that you only can speak in some code language, which only has two symbols.

In Morse Code, these symbols are called "dot" and "dash", or sometimes "dit" and "dah" (Reference: Wikipedia), but the point is simply that there are just two of them.

To convey the information consisting of a single choice made between two alternatives, we can just use one of those symbols.

A popular choice of symbols is 0 and 1, which happen to also be numbers.

That is the basic Unit of Information, and it is called "one bit".

In a physical device, such as a computer, the two symbols are represented somehow, such as:

Some examples:

Now, that's interesting as far as it goes, but things are rarely that simple. Let's combine those things:

We can string symbols together, in a given order, let's say that's { Pet, Glasses, Shirt }.

To inform someone of the choices involved requires 3 bits of information, and since each one has two possibilities, there are 2 x 2 x 2, or 8, possible "words".

Multi-way Choices

What about cases where there are more than two choices?

Then we need more than a single bit.

Let's say our shirt manufacturer gets bored with just two colors of shirts, and wants to add a few more to just Red and Blue:

Orange, Yellow, Green, Violet, Black, White - 6 more, for a total of 8.

As we saw above, where there were eight { Pet, Glasses, Shirt } combinations, we will need 3 bits of information to distinguish these 8 choices. To wit,

Got it? Test your understanding


How many bits of information are conveyed by a single Latin letter, A - Z ?