Looking For A Paradigm
My name is Alan Carter. I'm a software engineer with over 20 years experience and a long term interest in teaching and understanding the semiotics of what we are doing when we write programs. I was delighted to discover the Feyerabend Project, because the recognition that we have made a wrong turn somewhere is one that I share.
I've looked through the past postings to this group, and I offer the thoughts below in the hope of opening some fruitful areas of debate. I hope these notions are at least an amusing complement to the "Dilbert" calendars you will be finding in your Christmas stockings!
Listen to groups of able programmers (hackers) who have had a chance to get their teeth into an interesting problem. You'll hear them talking about the "deep structure" of the problem and/or the architecture they are constructing in response to it. Some groups use the term "deep structure", others just talk about it. The deep structure contains features, which groups give jargon names to. A bidirectional search might be called "the bubble sort" by one group (whether or not any sorting is involved), while the same thing might be called "the Cylon warrior" by another group. This use of jargon is very confusing for people who are not aware of deep structure (suits). Despite the suits' firm belief to the contrary, it is minted dynamically in a poetic response to a need for words. The jargon does not have a static list of definitions anywhere, nor are there "conventional" terms for the phenomena that are tagged with jargon, which the hackers dislike using because they are "immature". The dynamically evolving jargon is used because it is needed, and one hacker's ability to understand the others' free form jargon minting comes from a shared poetic sensibility, and the objective existence of the features of the deep structure, such that those features are there for all the hackers to see them. (In The New Hackers Dictionary Eric S. Raymond refers to this way that hackers seem to arrive at the same terms for the phenomena they wish to reference independently of each other. This may offer some insight into the Sapir-Whorf hypothesis, which suggests that language and cognition may be related, and which seems silly unless one has spent time talking with Native Americans as Sapir and Whorf did before they made their suggestion. But that is off topic.)
The objective existence of the deep structure space, populated with features (design patterns) is central to this view of programming. But what is it? One way to think about a program is as a theory of its problem domain. If we can write a program to sense traffic and control signals and we can keep our town out of gridlock then we would be able to claim a good understanding of traffic flow. Our understanding, captured in our program, would be a theory of traffic. We usually want our theories to be necessary and sufficient. When our theories are expressed as programs, this means that we can start by choosing our notations for explaining our programs, and the programming language(s) we will use, in order to make the rest of the job as necessary and sufficient (and in practical situations as communicable) as possible. When we make a necessary and sufficient theory, we get an elegant program, and have fun. In this view then, a program is an abstraction of some part of the real world. The deep structure of the problem domain and the deep structure of the program are the same thing. A program is a theory of part of the real world, which has got deep structure.
The idea that the real world has deep structure is hardly new - it's why we attempt to make physical theories. Since Netwon's day, no-one has doubted that the real world is algorithmically redundant. Simple and deep laws can be found that explain complex superficial behaviours. The only difference between hackers and other people who make physical theories is that most people get to make one or two theories in a lifetime (if they are lucky) while hackers expect to (and are expected to) produce them in production line quantities. If there was any known way of making programs in an automated, proceduralised way, then Deep Blue would be churning out reams of fundamental physics, and Nobel Prizes would only get handed out for the fuzzy subjects nowadays!
How do other people make physical theories? They guess! Richard Feynman is wonderfully blunt about this at:
The important guesses are not informed by deductive reasoning. They are inductive operations. This fact caused Karl Popper a lot of grief, although why it should be upsetting for any creature who has arrived rather late in the universe's story is itself rather odd. Of course, over the years a great deal has been written about the epistomology of making physical theories, but when it comes down to it we have little more than nice old Dr. Einstein's assertions that it is at root an aesthetical process. That ties in with the experience of writing elegant programs being fun, but perhaps as engineers who are expected to produce physical theories on a production line basis, we need to revisit this topic, if only to save ourselves from flouncing around and talking mysterian mumbo jumbo in an overly precious kind of a way.
In recent years we have learned something very interesting about the real world. It turns out that self-similarities across scales are found wherever we look. We live in a fractal universe. Take a look at:
where Mandelbrot himself talks about fractals. Particularly look at the page "Random Fractals and the Stock Market", where we learn that:
"Current evidence suggests a fractal distribution of galaxies out to 500 million light years, and there is yet no compelling evidence to believe the fractal distribution of galaxies does not extend throughout the universe."
Any frame of video data, be it a city block or a forest glade, is a spacial fractal - as anyone who avails themselves of wavelet image compression accepts whenever they surf the web. Your heart rate is a fractal time series. And stock market movements are fractals. But the stock market movements are a little odd. Unlike leaves on trees or windows in buildings or electrical exitations in heart muscle, they do not describe any direct physical phenomena. Sure, they have physical phemomena at their roots, including the amount of sunlight and precipitation falling on cotton plants, and the amount of cocaine infesting speculators' brains. Abstract and sum all those physical phenomena, and what comes out but a neat little fractal! And then the drug crazed speculator hits his comedown and goes and fiddles with his position in Yen futures, and that comes out as a fractal too! The universe would seem to be fractal in space, time and layers of bstraction. Perhaps the only way to get a universe that is fractal in space and time that fits together is for it to also be fractal in layers of abstraction too, so perhaps that statement is not necessary, but it's helpful here to stress the point. Because here we have a physically observable phenomenon that exhibits deep structure. It isn't a musing of the philosophers. Forget Jungian archetypes and Platonic essences. We can see fractals in action, and we know some things about their properties.
Now a fractal universe could (conceptually at least) be fractally compressed. It's a big idea - a fractal generator which when run deterministically would produce all the galaxies, including the Milky Way, the Earth, and the leaves on the tree outside Your house. We might think of this universal generator as a necessary and sufficient theory of the universe. It would contain no redundancy. Any fractal compression of any part of the universe, such as a frame of video data or the behaviour of traffic, would be in some sense a subset of the universal generator. The compression efficiency would not be as great as the universal generator can achieve, and it would have a "domain of applicability" outside of which it would break down and not give correct predicitions.
We have now started to encounter ideas which we can experience practically when writing programs. Hackers are often aware of situations (and suits cannot understand this) where in order to make the programming task simpler it is necessary to widen its scope. Hackers and suits have very different approaches to risk reduction because of this. We also know that programs that are elegant and based on an appreciation of the deep structure of the problem domain tend to grow logarithically with function points, while programs that are based on transliterations of requirements statements grow exponentially with function points. In this view the logarithmic growers are produced using inductive reasoning and run the fractal generator backwards, while the exponential growers are produced using deductive reasoning and run the fractal generator forwards. Feature interaction undergoes combinatorical explosion and the system quite literally descends into chaos. We've never found a procedural approach to programming that avoids this, and here we see that we never will. If the real world we are modelling with our programs is one vast multifractal, that chaotic interactions will inevitably occur if we run the generator forwards, causing an explosion of superficial richness and a worse than exponential growth of management overhead in a futile attempt to contain it. On the other hand, if we are willing to think inductively, we will equally reliably find ourselves getting "lucky", and will continue to find opportunities to repackage redundancy in our source code. The "refactoring" task in Extreme Programming has a very sound qualitative (at least) theoretical foundation.
We can also see why we can't automate the inductive reasoning process. We are always looking for a theory, a fractal generator, which is a subset of the universal generator, which contains no redundancy. If any subset of the universal generator could be procedurally related to any other part, those two parts would be redundant. All elegant programs are therefore "off a beat" with respect to all other elegant programs, although we can expect to see non-exact similarities in the deep structural characters of different elegant programs, because the universal generator contains such self-similarities like the way the patterns evolve as one moves around the interesting part of the Mandelbrot set. So knowing how to solve a 3x3x3 Rubic Cude is helpful when looking at the 4x4x4 one, but we can't just transliterate.
This also illuminates a truth appreciated by many effective programmers, that programming is best learned on the job, in the ancient apprentice, journeyman, master model. In this model on the job learning provides the time and range of experience that allows the student to build up a good repertoire of known patterns to use, "over and over again and never the same way twice". It also provides real world problems of sufficient richness to allow the "fractal mining" skills of inductive reasoning to be used. Simplistic examples that can be dissected in a lecture room in a single day just don't contain the real world fractal richness that the student must learn to discern and manipulate.
These ideas tell us what won't work as well as what will work. If we bear in mind what will and won't work when we look at the state of commercial programming, there is an opportunity to "form an impression" (obtain the results of inductive reasoning, reflect the deep structure of a problem domain) that our culture (the whole developed world) is fixated on what won't work, and is averse to what will work, to a level that is best understood as pathological. It is in this situation that we now sense that programming has taken a wrong turn. As Richard points out on Dreamsongs, when we write a program, we are doing it for the first time. In this situation, it should be obvious to the stupidest of minds that setting things up like a car factory won't help. After all, in the car factory it isn't the production line that maps to the working programmer at all. The production line maps to the computer running the completed program. What maps to the programmer is the design engineer who set up the production line. If we want to take lessons from industial mass production it is that design engineer we need to copy. That really should be obvious to anyone. But it isn't.
Most people seem to be in a situation similar to the unfortunates in Oliver Sacks' "The Man Who Mistook His Wife For A Hat", who had lost half their visual field due to brain damage. It isn't that they know there's stuff on their left that they can't see - the very concept of the left hand side of things has gone, and they experience no sense of loss, although they can only ever eat half their lunch. Most people's inductive faculty - particularly in commercial contexts - would seem to be completely dormant, and they become fixated on behaving robotically to an unhealthy degree. Thus they become convinced that programming is a trivial task of transliteration, which can only be performed procedurally because there is no other conceivable way to think, and which has *nothing to it* - which is exactly what we see.
What happens when people who are somehow obsessed with ritualism and have their inductive faculties asleep try to do work that can only be done inductively? They start playing "let's pretend" in a way that is reminiscent of small children pretending to be operating a shop! "Good morning Mrs. Jones... A kilo of cheese today... I'll just wrap it up...". In modern commercial programming everyone is a "genius", all problems are "trivial", endless meetings are held where (usually tool configuration) issues of inconceivable complexity are discussed in a gabble, gabble, gabble where not one single microsecond is ever wasted. But just like the childrens' shop, nothing ever comes out! The last new commercial products were Netscape (itself originally a free product paid for by the US taxpayer) and the Booch 94 edition of Rational Rose, nearly 10 years ago. Gone are the days when we got VisiCalc, NFS, integrated two-phase commits and so on every year. And if we can avail ourselves of an existing program by just copying it, what has software engineering been doing for the last 10 years if not playing "let's pretend"? The strange thing is, no-one seems to have noticed! Sure, the sand kicked up by maintaining artificial scarcity and obsolescence of commercial software, and the shift to generating profits by hoodwinking greedy investors and perpetrating accounting fraud instead of selling useful services to customers for money has gone some way to conceal the truth, but can that explain this extraordinary blindspot? The emperor has been parading around stark naked for 10 years, and no-one has noticed!
The answer may lie in the idea that most people are vulnerable to getting stuck in a deductivist, proceduralised mindset that is little better than an industrial robot, and in which the inductive faculty becomes dormant. So long as they are going through the motions every day, they neither realise nor care that the entire activity is pointless and thanks to automated factories we could now be just as wealthy if most people stayed in bed and learned to play musical instruments of their choice. Perhaps we have been in economic post scarcity since the mid 1970s, but are unable to take the next step and enjoy it because of a mind numbing culture that has evolved over thousands of years, and which is now absorbing all spare human time in pretend "administrative" ritualism. A culture in which there isn't even a word for setting out to obtain an insight - which is the core task of programming! (Hackers use the word "grok", but that isn't an English word. It was coined by the inductivist Robert Heinlein and entered hacker culture through SF fandom.)
The outlook for commercial programming may be bleak. Unless a firm's senior management has retained an awareness that there is more to solving problems and delighting customers than behaving reactively and policing reactive behaviour, and is able to hire and support staff who can lead the way out of stagnant introspection, programming is impossible. And probably as a result of the degree of human ritualism that automated production has made possible, there are few firms today that are in that happy state. But that need not mean the end of programming. We all know that the free software sector, which uses mutual convergence to a deep structure with objective existence instead of management structures that try to tell the problem domain what its structure is without first looking, can achieve productivity, flexibility, creativity and co-operation that the commercial sector has not achieved in years. Perhaps those of us with an interest in the semiotics of programming will be able to spend our time more productively by supplying the inductivists of the free software sector with a paradigm and language that will allow them to reach greater maturity, than by wasting our time on a lost cause. Like the old joke says, You cant get there from here.
I'd be delighted to learn what people think of the suggestions so far. Anyone without anything better to do over Christmas can find more on how this train of thought applies to reawakening the inductive faculty in practical teaching situations at:
It's just raw tape transcripts of lectures I fear, but those lectures, used in a practical workplace environment, did repeatably reawaken programming teams' inductive faculties and improved their productivity by a factor of 10. Sadly, the presence of such teams is no longer tolerated within worstening ritualised commercial organisations.
The wider sociological, philosophical and physical background of these ideas is discussed at:
Although perhaps I should say that they make the suggestions in this email sound like the time of day!
I noticed one poster mentioned Vernor Vinge's "A Deepness In The Sky". There's an attempt to "bring you Focus" at:
- On Friday, December 20, 2002, at 09:06 AM, Alan Carter wrote:
> I've looked through the past postings to this group, and I offer theI've read Alan's (clearly labeled) rant and find much of it compelling.
> thoughts below in the hope of opening some fruitful areas of debate.
He argues that the discovery of deep structure is the under-appreciated
necessity for progress in our field and possibly others. He writes as
if he has made this argument many times. I've only glanced at his
citations but notice that some are reviewed on wiki.
I offer a warm welcome to Alan and wish everyone a happy holiday. Best
regards. -- Ward
503-245-5633 v mailto:ward@...
503-246-5587 f http://c2.com
- Alan, I enjoyed your rant quite a lot.
I offer up a pointer to a fun essay by Danny Hillis named
"Intelligence as an Emergent Behavior or, The Songs of Eden"
I think it speaks to what you mention concerning the link between
language and cognition.
Another good essay by Danny is "Richard Feynman and The Connection
I can't get enough of that Feynman character. He is alive in my head
and talks to me, what a wonderful delusion.