An extremely simple single-layer feedforward 2 x 2 neural network is the subject of this article. Because I feel it is important to understand some essential features of neural networks without the help of a computer. The network at hand can be completely described, mathematically, by elementary linear algebra. A working example with two inputs and one output is leading to the general case. A counter example with two outputs instead of one is presented as well. It is concluded that the network with one output has learning capability and the network with two outputs has not. The behaviour of the first network can be formulated in geometric terms: all points on a straight line through two given points in the input plane give the desired output. There are no other inputs that do the job. The network with two outputs, on the contrary, is not able to make any generalization. It does not learn from experience, so to speak. It's kind of surprising that the more intelligent network is characterized by a singular matrix, and the dumber network by a regular matrix of weights.