Tuesday, July 19, 2011

Management verses the Great Migration to Open Source Software

Visit the High Level Logic (HLL) Website
— Home of XPL (eXtensible Process Language)

It's like XML, but you can store source code in it.

Among the most irritating PR supports for poor management are the list of personnel “studies” purporting miracle paths to greater efficiency and lower costs. They include claims such as that recruitment and retention of good engineers are more dependent on having the right plants in the lobby than salaries and that life in cubicles promotes productive human interaction. When none of that works, then retention isn't important anyway. Might as well outsource. And it just so happens that the people behind the “studies” lease engineers as well as cubicles.

The proliferation of such ideas in the 1990s interrupted growth of a healthy sustainable technical culture that was sprouting software engineering companies at just about the right pace to provide balance between in-house IT line staff and other specialized workers. Government and government contractors were among the largest contributors to the problem as they colluded to control labor costs while maintaining their own large cubical habitats for migrating engineers. Moreover, engineers were doing well and humans tend act like humans. Seen from the inside, the emotional backlash from other professionals has been at times extreme. It was better to ship a quarter of the nation's economy to a third world country than to let “those people” live in nice houses.

What's next, letting engineers make engineering decisions? In this dark age when human resource “experts” and bean counters reigned, good engineering suffered greatly. Following their miracle plans resulted in spiraling costs and poor quality. Even the future of engineering suffered, as a great many engineers moved from company to company, project to project, without being able to stay on a consistent professional development path. As the backlash continued, experience was undervalued and many development organizations became overburdened with cheaper, less experienced engineers that took more than ten times longer to produce results that were less than a tenth as good (which also dramatically increased maintenance costs).

Where has all the innovation gone? I'm among those who have tired of Big Software selling their latest versions of 20 year old ideas as technical breakthroughs – a continuation of the great line of miracle cures for whatever ails you. It seems like too many years have passed since they should have backed up and fixed what they were already producing. Maybe now we wouldn't all be suffering so much from computer viruses and browser lock-ups and would have built the great software engineering tools that would facilitate creativity and support real business needs rather than making sloppy use of the latest bingles and bangles sometimes slightly more convenient. Since none of that really works well enough to be economically sustainable, maybe you should outsource your business operations to the new miracle on the block; “the cloud.” Never mind that there have been companies that have wanted to run your IT department since at least the days of magnetic core memory.

You might be wondering by now; what has all this to do with open source software? It's easy. Look – while the movers and shakers in the industry are busy screwing around with other things and company managers are preoccupied with the status that a nice purse and heals seems to bring, someone should be interested in producing good software. While I disagree completely that engineers are somehow less interested in being paid than everyone else (I don't care what kind of plants are in the lobby) there are thousands who would work for less at least temporarily or give of their own time to do work they can be proud of for a change and to be able to take credit for it. What's the point in spending decades developing professional knowledge and skills if you're forced to suppress all but the most menial parts? Many people actually do go into engineering with a desire to be creative and to create good things.

And why should I write this article now? You might be thinking that I'm a decade or two behind the times with my thoughts. After all, open source caught on and caught on big (except among young, naive, pimple-faced Big Software kids of course). It's become so mainstream that it is in common use and constantly growing in importance in industry. And that is the coming problem. The structure of the industry hasn't fallen back into a sustainable form dictated by market demand and a competition among professionals over who can provide the best products and services. The one thing that made the most sense was done by engineers working outside the mainstream. Where significant industry investment was involved, it was primarily to counter monopoly behavior that rendered them unable to compete in traditional ways.

Here is the part of the article where you should be interested in a solution. I know I am. I long for the days when men were real men, women were real women, small furry creatures from Alpha Centuri were real small furry creatures from Alpha Centuri, and most importantly in the context of this article – spirits were brave and engineers were real engineers. Perhaps engineers need a guild and a code of honor that they will not violate – not even for the many little queens that control most of our salaries.

Support for open-source software and projects from big industry players are at times and especially in particular instances very much appreciated no matter why they occur. Sun's support for Apache for example, has been extremely important to Internet use and the Java language and development tools are extremely important to continued innovation and development. Oracle continues some very good parts of what Sun started. Google has pushed ahead with implementation of full HTML5 support for Chrome and has made its plug-in available to drive HTML5 in other browsers. Red Hat, which started from open-source, supports a variety of open-source projects. A few years ago, Google vowed to use open-source and support for standards to return control of the overall direction of software development to consumers.

But let me suggest another path of more direct interest to the most important players; the users themselves. Not just Pam and Paul sitting in their living rooms shopping and emailing, although I'm happy to acknowledge the users that occasionally click on the PayPal button to give something back for a free software package they like. (I have done so myself.) It's the corporate users that would benefit the most from adopting open-source projects. There is in fact, a rather old idea that we can call "customer financing." When companies identify their need for software, it can benefit them greatly to individually or jointly support R&D for development of competitive products that are useful to them. It's a strategy that avoids becoming dependent on expensive fads that do not address their business needs in the best possible way.

To what extent we might be able to teach managers to think this way? What I know is that some companies are better prepared than others for such tasks. Some are themselves engineering companies, not focused on software in particular. Or companies that depend on technology to such a great extent, that they are not lacking in technical competence. These companies are most likely already routinely investing in specialized outside R&D in addition to their own internal R&D.

(To be continued ... ?)

Another perspective, time O'Reilly (1999): Ten Myths about Open Source Software