Ming the Mechanic:
Software is hard

The NewsLog of Flemming Funch
 Software is hard2007-02-05 15:21
by Flemming Funch

Software is hard, an article/interview on Salon, based on Salon co-founder Scott Rosenberg's book Dreaming in Code. About, well, why programming is hard, and why it particularly is so hard to estimate how long a given development task will take. I certainly recognize that problem. The book's sub-title is "Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software". That's talking about Chandler, which is the particular focus for the book. Chandler is an ambitious project of creating a new way of organizing personal information. It is conceived, financed and managed by Mitch Kapor, the guy who invented Lotus 1-2-3. He hired the smartest people he could find, and was even prepared that it would take a while. But it seems to take forever to even get a preview release of some kind out the door.

The seminal work about why software is surprisingly hard is of course The Mythical Man-Month by Fred Brooks, which was about the development of the IBM360 operating system, which also took forever. Part of the wisdom from that experience was that as you add more people to a software project, the complexities of the communication between them grow exponentially. It becomes much harder for anybody to know what is going on, and very hard to be on the same page, so the majority of the effort is wasted in trying. Microsoft is of course a good current example. Very smart people, but the projects get so complex that it takes thousands of people and billions of dollars, and the product ends up being several years late, and it is an incoherent mess. Brooks' solution was that programming should be done by small teams, 3-5 people, with very well-defined roles, where one person would be responsible for the overall conceptual integrity of the project.

Scott Rosenberg doesn't seem to have anything terribly revolutionary to add, but he does formulate what he somewhat jokingly calls "Rosenberg's Law". Essentially, it is the (somewhat obvious) wisdom that software is only hard and difficult to estimate when one is doing something new, something that hasn't been done before. Which is right. The people who can do very disciplined on-time software projects usually can do so because they do something that has already been done. You know, if a customer needs a website with 10 pages and 5 graphics and a menu, and that's what you do every day, it wouldn't be too surprising if you can provide an exact quote on that.

The traditional "ideal" way of carrying out the software development cycle would be a sequence of studies, of feasibility and requirements, and an analysis of exactly what needs to be done, resulting in some specs handed to the programmer, who "just" needs to do it. That has never really worked, but, in principle, if it already is perfectly clear exactly what needs to be done, of course it isn't hard. It just never is clear, because people usually don't know exactly what they want before they see some stuff they don't want, and they have a choice. So that way of developing software is going out of style. It has to be more interactive than that. Shorter cycles, involving both the programmers and the users in reviewing the progress, frequently. Which tends to be how one does it nowadays.

Part of the trouble with software is that programmers are only having fun if they're doing something new. So, even if there might be an existing solution, which would be boring, but reliable, most programmers would prefer to make their own solution. And there's no ultimate formal way of doing something new which is partially unknown.

What is missing is really tools for modeling things to do. Oh, there are diagrams one can do, but that isn't it. One would need to model the real thing, no matter what it is. Which, unfortunately, takes about the same amount of work as doing the project. So, the general problem might only be solved at around the same time when most programming will no longer be necessary. I.e. you interactively work out the model of what to do in real-time, and when you're done, the software is done too. No separation between the specification and the doing. Would be great. There are systems that do that to some degree, but so far nobody's succeeded in making it general enough.

The ultimate software project would be to invent a system that makes programming obsolete, by making it so simple that anybody can do it, very quickly. Unfortunately that's a hard.

[< Back] [Ming the Mechanic]



20 Jun 2007 @ 15:57 by anandavala : Making software easy
Hi Ming, I've been scanning through your news log - great stuff!

About this article and the idea of WYSIWYG programming so people needn't know anything about programming to do it - they'd just need to understand the system they were trying to build and the rest would be handled by the computer...

I've made considerable headway on that issue - the core is essentially built and the methodology is worked out but I'm no good at interface design and all that other fiddly stuff - my mind resonates with the experimental/theoretical programming side of things but it doesn't resonate with the 'normal' software engineering side of things...

There are proof-of-concept applications that show it in action - not the design interface yet - it's just a simple user interface that lets people interact with simple virtual realities. It's just proof-of-concept stuff to show that the mathematics works and is efficient (it all stems from a new mathematical general-system simulator) that fell out of some metaphysical research I was doing into the kinds of computational reality generative processes that could give rise to a reality like our own.

There's more information in this article if you're interested: http://www.newciv.org/mem/prof-newslog.php?did=550&vid=550&xmode=show_article&artid=000550-000005&amode=standard&aoffset=0&time=1182353915  

20 Jun 2007 @ 23:29 by ming : Easy software
Cool, I'm going to look that over carefully. I'm skeptical of any claim of very universal programs that generates programs, exactly because I've worked quite a bit on it myself, and failed. Seems like it ought to be simple, but somehow there are always some details that trip it up. See {link:http://www.thingamy.com/|thingamy} for example. Anyway, more later.  

22 Jun 2007 @ 01:32 by anandavala : it ought to be simple
I think the problem many people face is because they approach it at too high a level. A silly example is trying to make a universal construction kit out of little rods and connectors. But what the universe does is make information/spirit that flows and gives rise to interconnected networks of systems, these interact and create space and time and quarks and leptons, which then go on through particles and atoms and molecules and so on until we end up with little rods and connectors.

I wasn't looking to create this software - I was looking for the Source of all that exists and when I happened to mathematically model the information theoretic flow of spirit as it creates the first and most intangible level of manifest existence I noticed that it was logically equivalent to system theory.

Then when I ran those models in a computer I realised that they create fully interactive virtual realities wherein we can construct whatever systems we like. All that is given is an abstract existential space and we can put whatever we like into that space.

As for automatic translation of those systems into software - that hasn't been a priority for me but it could easily be done in terms of procedural languages because the information paths can be translated into simple algorithmic strings which can be computer in c or fortran or whatever. But through the parallels with system theory it could also be translated into object oriented languages. Here the systems are objects and the interaction paths are methods and pointers that interconnect them.

But you're right - I'm sure there are some hard puzzles to be solved in the details - in the long run I'd bank on leaving the models in SMN format and changing the computer to suit SMN, by implementing it in silicon and getting rid of the whole idea of coding and linear streams of computation - it should all be massively parallel on a single chip. What I'd love to see is the parallels between SMN and quantum physics to be exploited to use it as a general operating system for quantum computers when they arrive, then we could make some serious artificial universes...  

22 Jun 2007 @ 22:38 by ming : Elementary software
Heheh, you didn't make me any less skeptical. But I promise to look at it soon.  

23 Jun 2007 @ 05:35 by anandavala : Be Skeptical - Be Very Skeptical
Skepticism - the application of reason to any and all ideas - no sacred cows allowed...

But I wasn't trying to sell the idea - just explaining myself a little better.

I'm just as happy if nobody looks into it - I still have serious doubts if the world is mature enough to handle that much power yet - it might be safer just to leave it in the box until we come of age ;P

But ultimately that decision is left up to fate, that's why I make it available but don't push the idea.  

23 Aug 2015 @ 00:51 by Billy Vandory @ : re: Be Skeptical - Be Very Skeptical
Ok, Mr. BabblesALot, best you keep it in the box. Yes the post is 8 years old. Where are you now>  

Other stories in
2014-11-01 17:33: The conversation of work
2007-02-24 14:20: Writing books in HTML/CSS
2006-11-19 21:30: Thingamy
2005-12-14 15:15: Ruby on Rails
2005-03-19 16:04: Comment and Refererrer Spam
2005-02-23 21:34: Wikipedia
2005-02-22 17:32: Mail
2005-02-10 16:00: More Google wizardry
2005-02-04 15:14: The Six Laws of the New Software
2005-02-02 18:37: Blog, Ping and Spam

[< Back] [Ming the Mechanic] [PermaLink]? 

Link to this article as: http://ming.tv/flemming2.php/__show_article/_a000010-001779.htm
Main Page: ming.tv