Separating the Wheat from the Chaff

From Coding Horror: Separating Programming Sheep from Non-Programming Goats I learned of a paper, The camel has two humps, which describes a test that allows teachers to differentiate students likely to do well studying computer science from those who will likely never ‘get it’.

This paper sounds awfully similar to the physics conceptual test mentioned on American RadioWorks program, Don’t Lecture Me, that uses questions with little to no arithmetic to probe students for their level of understanding. The physicsts don’t use their test to cull the herd, but rather to let the professor assess which concepts the students haven’t yet assimilated. Astoundingly, the computer science paper proclaims:

We point out that programming teaching is useless for those who are bound to fail and pointless for those who are certain to succeed.

Evidence for this statment comes from low retention rates among computer science departments (30–60% fail the first programming course). But that observation doesn’t necessarily mean the students are ‘bound to fail’. Rather, I see it as evidence that the current methodology, lecturing, has not only failed to transmit information to the students, but also leads to such criminally dismal expectations.

One of the more promising experiences related in the paper demonstrates that we have to match education techniques to our learning experience. We should only use those tools which leverage how the brain learns.

Programming teachers, being programmers and therefore formalists, are particularly prone to the ‘deductive fallacy’, the notion that there is a rational way in which knowledge can be laid out, through which students should be led step-by-step. One of us even wrote a book [8] which attempted to teach programming via formal reasoning. Expert programmers can justify their programs, he argued, so let’s teach novices to do the same! The novices protested that they didn’t know what counted as a justification, and Bornat was pushed further and further into formal reasoning. After seventeen years or so of futile effort, he was set free by a casual remark of Thomas Green’s, who observed “people don’t learn like that”, introducing him to the notion of inductive, exploratory learning.

Programmers, both professional and beginner, spend much time dubugging. Much language research is therefore devoted to finding and building systems that either (a) help to prevent the writing of bugs (ex: static type systems) or (b) help to discover them once written. As far as that goes, it is interesting to note from this observation:

Thomas Green put forward the notion of cognitive dimensions to characterise programming languages and programming problems [12]. … He is able to measure the difficulty levels of different languages (some are much worse than others) and even of particular constructs in particular languages. If-then-else is good, for example, if you want to answer the question “what happened next?” but bad if the question is “why did that happen?”, whereas Dijkstra’s guarded commands are precisely vice-versa.

That one form of language construct, if-then-else, would be easy for a beginner to write in, but the other form, guarded commands, aids debugging.

The difficulty in learning a skill such as programming, lies principally in forming a mental model of how the computer executes the program. Forming an accurate model implies both the ability to describe your problem as a program and the ability of expert programmers to justify their programs. This understanding of how comprehension works drives the construction of the conceptual test, both in physics and computer science.

The paper has a very important finding:

… in the first administration [students with no prior experience programming] they divided into three distinct groups with no overlap at all:

• 44% used the same model for all, or almost all, of the questions. We call this the consistent group.
• 39% used different models for different questions. We call this the the inconsistent group.
• The remaining 8% refused to answer all or almost all of the questions. We call this the blank group.

…
Remarkably, it is the consistent group, and almost exclusively the consistent group, that is successful.
…
It has taken us some time to dare to believe in our own results. It now seems to us, although we are aware that at this point we do not have sufficient data, and so it must remain a speculation, that what distinguishes the three groups in the first test is their different attitudes to meaninglessness. Formal logical proofs, and therefore programs – formal logical proofs that particular computations possible, expressed in a formal system called a programming language – are utterly meaningless. To write a computer program you have to come to terms with this, to accept that whatever you might want the program to mean, the machine will blindly follow its meaningless rules and come to some meaningless conclusion. In the test the consistent group showed a pre-acceptance of this fact: they are capable of seeing mathematical calculation problems in terms of rules, and can follow those rules wheresoever they may lead. The inconsistent group, on the other hand, looks for meaning where it is not. The blank group knows that it is looking at meaninglessness, and refuses to deal with it.

The test, which teases out these speculations, involves the notion of assignment. It iterates through different statement orderings, and variable names allowed to carry misleading implications about the underlying values. Many possible answers are given for each question, so as to arrive at the mental model that a student might have used. They also come with a space for free response or side-work. For example,

1. Read the following statements and tick the correct answer in the front column.


int a = 10;

int b = 20;



a = b;



The new values of a and b are:



[ ] a = 30  b = 0

[ ] a = 30  b = 20

[ ] a = 20  b = 0

[ ] a = 20  b = 20

[ ] a = 10  b = 10

[ ] a = 10  b = 20

[ ] a = 20  b = 10

[ ] a = 0   b = 20

if none, give the correct values:

    a =        b =

So the open question is: Is it possible for a teacher to help the inconsistent students learn a consistent mental model? If so, then we shouldn’t give up, we should bring the learning techniques which make it possible into the classroom. I conjecture that those techniques will also help the students with consistent models correct the bugs in their mental model faster. So dispensing with lecture, clearly shown not to work, and replacing it with newer, more effective techniques derived from cog sci, will benefit us all.