In college, I enjoyed writing software which actually did things like many programmers and avoided anything that could be viewed as "programs for other programmers". By majoring in math/computer science, I avoided the compiler class altogether and I pretty much slept through my database class. So how is it that now 20 years later am I spending a lot of time thinking about precisely these tools?
Through my experience, I agree with many others that despite the advances made in modern development environments, there are still some basic subtle but systemic problems with existing languages and tools that inhibit productivity of developers and increase the implementation, and maintenance costs and overall reduce the user satisfaction.
I think we can make a much more flexible, powerful and easy to use programming environment which layers on top of existing runtimes to minimize the duplication of effort. You can read more about my current design ideas here, but to give you an idea of where my thought processes have come from, I've put together a brief biography of my career and experiences here.
In college I developed a strong interest in 3D computer graphics. I got a job at the Brown University research lab and wrote a ray tracer, a 3D scan-line renderer. I then got a job at Stellar Computer in 1986, a startup trying to build a graphics supercomputer. I wrote hardware test programs, hardware simulators, an X window system port to a hardware simulator then to a real computer. I then wrote a 3D graphics interactive tool to help demo the hardware and then helped out on the AVS project - a visual programming data flow editor for building scientific visualization applications.
It wasn't long before the visual programming problem became more interesting to me than the 3D graphics problems. I could see how the pictures we were making were helping people, but we ran into the limits of major innovation pretty quickly. Making programs easier to build, more flexibly customized via a distributed, visual programming environment is an interesting and challenging problem which I don't think has yet been solved in a very complete way. To me that was the real potential of the visual programming environment - get it into the hands of real end-user/programmers and let them build the applications for their particular niche. If we did not get out of the labs, the market would continue to be very small since the programmers and the users were the same person for much of our installed base.
At this point, I became chief architect of AVS's second generation visual programming tool called AVS/Express (a poor name since if anything, it was not "AVS lite" but "AVS on steroids"). It was a general purpose visual programming environment, though it was only marketed and sold for visualization applications (despite my pleas for an open source, cheap product for general application development). I don't know whether it would have been successful as a general programming environment, but I know that the lack of a marketing program surely limited its potential. Since then, I've seen a bunch of much less flexible systems developed and sold horizontally though still only applicable to fairly vertical application areas.
AVS/Express had a language named "V" that I designed. Now V was not a full featured language with loops, flow of control etc. Instead, it only defined "the functional skeleton" of the program. This consisted of a definition of persistent objected oriented state, an execution environment for triggering updates when data changed and a flexible execution model for scheduling execution of methods mainly triggered when UI elements changed data connected to computational elements.
As an object oriented system, you would define templates (perhaps associated with a C++ class) and instances associated with a specific application wide instance of a class. The system maintained a set of properties which could have connections to each other either directly or through an expression. Objects could trigger events to fire methods in C/C++ code and there was a wrapper api to directly set/get properties or we'd generate a C++ base class with operator overloading. Persistence meta-data was attached to the properties so you could manage state save/restore by just setting the right attributes on the right properties.
It also had a persistent state storage mechanism though that was I think the most limited piece (persistence is a much more complicated problem than I realized at the time!) Data types were defined via primitive types, or other objects. Each object could either be in a "template" mode or "instance" mode. In template mode, they were inactive and could either be used to create instances or could be used to define a reference to an instance which "matched". Match was like instanceof though you could match values as well (e.g. if you had a template with numDimensions=2, it would only match and thus allow you to connect an object with numDimensions=2).
When values in the model changed, the code could register for notifications. Users could create connections between properties using expressions. UI widgets were created using this system as well as computational modules and this system let people drag and drop UIs, then hook up the fields to modules either using V or the visual programming environment. Macros could be created by grouping modules and you could redefine a new view of the same underlying data. This language was implemented in C and modules could be written using Fortran, C, and C++.
It took some getting used to as the concepts were different but those who broke through the learning curve found it to be a productive system for building applications.
There are some cool things in AVS/Express that I still haven't seen make it to the mainstream programming environment. In my view, its main strengths were:
Now it is true that creating an individual module is harder than just writing a C++ class with similar functionality, but code reuse is much closer to the idea of snapping together reusable parts. The visual display of the objects and connections could be useful as well in navigating functionality. From a given UI control, you could open up its module view and vice versa. You could also edit the V description of a component directly.
The problems with AVS/Express were:
Besides, there was a new language Java that fixed many of the problems of C++ and was very intriguiging. With reflection, garbage collection, "easy" to use threads and synchronization, serialization, exception handling and a "crash free" runtime I felt like it was the right base for the next environment. One big missing piece was a component model but that was shortly added in JavaBeans which I jumped on immediately by attending the first JavaBeans conference. In my view, it does have a few problems though - programmers lose the ability to use a.b - the must call a.setB(...). Also, properties changed events are not mandated in the spec and the code to implement a property sprawls over many lines of code (particularly if you implement changed events) when it should just be two words, perhaps with optinal modifiers. There was no standard way to serialize a class using an ascii format too - just binary. Folks adopted .properties files as the defacto means but not in any way that promoted interoperability. Binary was solved with .ser files in I think a cool way though I haven't used them much.
But it was cool seeing a language adopt a single component model but then saddly we immediately saw the quick adoption of JavaScript, the rapid growth of perl, and php during Java's rise to fame. Standards such as J2EE were rushed onto the fore and complicated the environment as specs were jammed out at a rate so fast as to make the existing interfaces overly complicated filled with lots of spec which existed because of the lack of basic mechanisms in the environment. We ended up with massive unnecessary specs like EJB to try and bridge the gap between constituencies that couldn't grasp the full complexity that Java had to offer.
I started working at ATG in 1996, taking a cut in pay and seniority to build websites in Java. I liked the people there and the environment was cool and relaxed and I was learning all about the internet. And most importantly, I could bike to work. I really didn't think we had a ton of long-term prospects given that at the time Java was not very commercially viable.
At first I loved Java. Unlike C++, it had a clean, simple stripped down object model which was easy for a C programmer to learn. And so many programming problems just disappear with Java. No pointer problems, interfaces, multi-threaded primitives that are built in. Serialization supported by the language. Perhaps most importantly, a good standardized set of I/O, networking, threading, and collections, date, utilities etc. Most people I talked who made the switch between C/C++ and Java think programmer productivity is 2-2.5X faster with Java (probably more for some applications and less for others).
By fixing the "misused pointer" problem in Java, one could rely on localized changes in a program having localized effects in the outcome. I no longer needed a debugger because code inspection was a faster way to solve most of my bugs. In some extreme cases I released code without testing the code path because analytically I could prove to myself it would work (frighteningly, in many cases I had no choice as that code path was near impossible to reproduce!)
Java was designed to be multi-threaded up front so this stuff was in the language. It made it much easier to build multi-threaded applications which share data and it turns out that having lots of threads share data is the best way to minimize OS overhead, make most efficient use of memory to cache content and make web sites fast.
Because of all of the new applications being developed for the web, a 2.5X productivity gain on this type of complex high end software made a big difference getting this stuff out in time and that made a big difference in the market you could obtain. Sites did not crash due to the good exception handling and memory safety. No buffer overflow problems. Eventually, we stopped getting turned away because we were Java and started getting the deal because we were Java. That started a long run where our customer list grew and we achieved success. I think by keeping good software engineering principles and our customer's interest primary in the corporate agenda. We made it a point to take on challenging projects for important (i.e. visible) customers, and get them out on time.
At ATG, we wanted to build resuable software components for web sites. The Java programming environment has nice language facilities for helping accomplish this task, but so far, it still lacks a good persistence architecture and a good model for defining and configuring instances of components. For a while, people would put configuration in monolithic properties files and then spend their time copying the around and doing massive merges and propagations of all of the changes for various configurations. Now people seem to be doing the same thing but with XML files instead of .properties files.
Of course there are usually admin UIs which edit the files so people rarely need to do this by hand for the major applications. But if you're a company building reconfigurable components, you need to do a lot of work for each component to manage its configuration.
To deal with these problems, at ATG, we built a simple naming system which manages the lifecycle, scope, and configuration. It provides administration UIs (both HTML and Java) for editing component configuration and viewing state of live components. We also built a general persistence architecture. This contains support for versioning, file system, SQL data (including a flexible and robust O/R mapping). For ATG, all of these had to be efficient and scalable (i.e. thousands of simultaenous users) and so persistence includes caching and various methods of synchronizing caches between cluster members.
In the ATG system, there is also a JSP tag library which directly edits properties of JavaBeans, and generic components which can be used to insert, delete, or update properties of any persistent object. These high power tools reduce the amount of Java code required to build applications and also improve the readability and maintainability of a web site. In addition, the ATG libraries were designed in a such a way as to have a clean boundary between the various types of programmers. Admins could use UIs or edit .properties files on a file system. Designers could edit jsp template pages with HTML, augmented HTML tags, a powerful macro/substitution mechanism, and tags which allow HTML to invoke Java which invokes HTML which invokes Java recursively to allow the most flexible and cleanest separation of Java with HTML.
Though ATG never attempted to market this software to the horizontal software development market with a low cost, open source oriented product even though these tools are very well suited to just about any web site you build. ATG gained a cult following among services companies which got their money on their productivity savings - in some cases, customers developed sites without using any of the vertical application functionality.
I just recently have switched to a consulting role at ATG from full time so that I can take some time off and look for the next big challenge in my career. I still have a lot of respect for Java and think it is very competitive among existing development environments for any new software project. At the same time, I feel like there are some important things missing from the current Java/J2EE environment and so there is potential both to improve development efficiency but more importantly reduce the maintenance and costs for modifying applications. Though Java may have improved overall developer productivty by 2 or 2.5X, it certainly has not lowered the cost of the developers you need to maintain your site.
In fact, with the high degree of client/server interactions on many sites and multi-threaded computing involved in most Java applications, the complexity of applications has in many ways increased from old single user applications. If you look around at J2EE, much of the complexity of the interfaces and applications created comes because Java lacked from the beginning some of the basic features required by all applications: persistence of all kinds (including configuration and naming of instances of objects). JNDI adds naming and some folks are using JMX as a partial solution to the configuration problem but neither is adequate. Instead, we are forced to learn and manipulate web.xml, application.xml and other files which impose a rigid, inflexible model on how we store program configuration. Vendors must spend a lot of time making UIs to generate these files and once generated, they become part of the legacy of your application. If the UIs are good, you can use them to maintain these files and if they are really good, maybe they integrate with your source control system. But what happens when you need to set a port number or something which is different for a given installation? Of course, you copy that file and change that one value (often, this entails unjarring and rejarring the file). And when the original changes, you do this over and over again or eventually write a script to do it.
When I reach this point in one of my own designs, I feel like it is time to take a step back and see if maybe you try and build the same types of objects in a different environment, maybe the result will turn out simpler and cleaner. I have this strong feeling with where we are with Java and J2EE now. It needs to be taken apart and rebuilt - this time adding a few parts earlier on such as persistence, improved instance state management and changing some fundamental things: a different memory model, a different compilation model, etc. Java and J2EE may well be the runtime engine for this new system (or perhaps one of several) but we need to simply and unify many of the concepts and interfaces and add the necessary missing features to the core.
Naturally there is a ton of work being done in language research and in the little time I've had since I started working on this, I think there are some good ideas out there. Paul Graham is putting a more readable syntax on top of Lisp in the Arc project. And Charles Simonyi is combining intentional programming with aspect oriented programming in his new company. I think that there are some really good ideas here. One is that we need to get back to a language with a single fundamental building block. In Lisp, everything is a list and this makes things like reflection, introspection of code, simpler and more natural. I think a language should have a list-like data structure as its basic building block.
From what I can tell about Intentional Programming, one of the main ideas there is that you first create an in memory model of the computer program - the abstract syntax tree.
This model contains an extendable set of language constructs. Each new language construct either turns itself into other constructs or produces code in the runtime environment. In this way, you c an add higher level operators to the language. You might add an operator to express a particular design pattern or maybe an operand which modifies the behavior of some other construct. Once you combine the program elements, you generate the code for your particular application. At this level, even the runtime environment could be coded into the system by specifying the appropriate code generation logic.
You can also produce language constructs which manipulate the tree as a whole for cross cutting concerns like persistence, security, logging, etc (aspect oriented programming). This means you can write code which separates concerns which will improve code reusability and maintainability.
All code editing in intentional programming is done via a structured code editor. This has some advantages as the computer can perform completion, immediate syntax error detection, etc. though it bothers me that from what I can tell, there is no underlying human readable language in which the AST stores itself. Given what little info I have on this right now, it is too hard to say if it will be a suitable base. I'm now thinking that the most important basic piece is the objected oriented data framework which manages persistence. I've recently been looking for a good cheap object oriented database.
More as to why I'm looking into this now. I have a feeling that the browser is not the last user interface paradigm switch which is going to cause a massive change in the software landscape. That will be the perfect time to usher in a new development paradigm - this may be the perfect time to design it though so that it is ready when the development push begins. Good tools will improve the complexity of the programs which we can create. In my view, much of the problems with our current software tools is that they are too simplified for the real complexity of the problems they are asked to solve. People want to delegate complex tasks to their computers, but the computers cannot even implement simple metaphor's properly. Neal Stephenson' laments how when he attempted to "save" his document, it cleanly destroyed the old version and then lost the new one when the machine crashed. An example of this complexity is my view that the proper metaphor for a user's view of the file system is a versioned one where complex rules determine when a particular version of a file should be saved and yet this complexity has made it into only a fraction of the programs we use today. All programs should implement undo and yet currently with all programming environments that I know, this is a difficult manual task that must be designed in from the beginning.
When we move towards voice computing, we'll need program's to be a lot more complicated and a lot more personalized. The computer's model of the user will become a much bigger factor in making this successful - both in terms of the ability to recognize the voice and words, and also predict the validitity of a particular recognition to improve reliabitility. We'll need to be able to simulate a given program sequence, run some tests to verify that a given sequence is going to yield the proper results and if it fails, roll this operation back seamlessly and try a different parsing of the same inputs.
I've got a lot of ideas for how to do this but clearly it is a huge amount of work. But I look around and see how many new language projects are going on and I know that others feel the same sense of dissatisfaction with what is out there. I'm still investigating what exactly is going on in these projects, but so far, I haven't found one that has caught my eye as a solution to the big problems we are facing, at least not in the specific combination that I envision. No doubt, I'm missing quite of bit of the detail on these projects so please let me know if I'm wrong. I apologize in advance for lack of attribution on ideas I mention - my research is far from complete at this stage of the project. If you see or know of something that sounds like what I'm describing or have any feedback, I'd be interested to hear it. For sure, I'm looking for a new set of collaborators I can help build a language that will get me psyched to code in again :)
Again, for more info see my My Quest for a Better Programming Environment.