Language problems
Sir: I should like to applaud Donald Michie's article (`Jasper Lupo's Spaghetti Code', 3 June) which tackles, quite success- fully I think, the task of alerting the lay reader to some of the societal dangers of computer technology.
However, while self-accountability of software systems is to be recommended, he did give the false and dangerous impress- ion that the necessary technology is avail- able, and that our task is merely(!) to persuade government contractors to de- mand it in their software systems. Self- explanation of behaviour is a much-touted feature of expert systems' technology, but in reality it is little more than a crude trace of what happened in the programme. This is very useful, and it is a step in the right `I got busted.' direction, but it is a far cry from self- accountability in any real sense.
The danger of Michie's proposals is that a crude self-accountability component of software systems could cause more prob- lems than it solves. If users of a computer system believe that the system is capable of explaining its actions, then they are even less likely to delve into its innards to find out for themselves why the system is doing what it is observed to be doing — they'll just ask it to tell them. In the absence of a comprehensive and robust technology for self-accountability, and in the presence of a contractual requirement that the system accounts for its actions, the system desig- ners are likely to cobble together some crude parody of self-accountability (reasoning quite rightly that no one will be able really to understand what they've done anyway). Then, in the state of awe that these amazing and totally mysterious beasts can all too easily engender, the feeble human user of the resultant system can do little more than accept the compu- ter's own explanation of its actions — need I go further?
What is the answer? There is no simple panacea. But a healthy scepticism for computer systems is a good starting point. If the behaviour of a computer cannot be explained to the intelligent layman, then its results, decisions, predictions, whatever, should not be accepted either. The prob- lem is compounded here because the hu- man explanation may also bear little rela- tion to reality — but we are more likely to cross-examine a fellow human being than a computer. But you can begin by rejecting all explanations of mistakes, incompetence, and mismanagement that blame the com- puter. The claim that the problem is caused by this inscrutable, and by implication unchangeable, thing, the computer, should never be accepted.
Derek Partridge
Professor of Computer Science, Department of Computer Science, Prince of Wales Road, Exeter
Previous page
