Artfully Clean

Imagine buying a house and then never doing the laundry, vacuuming, taking out the trash, or washing the dishes. You just let stuff pile up until you run out of space and then you throw your up hands and go “oh, it must be time for a new house.” Sounds ludicrous eh?

In a highly competitive development environment though you build tables and files and directory folders, develop software, test a bit, and away you go. You patch up the bugs, optimize the queries, add a handful of indices, and you’re flying.

After a couple years though people are wondering why things take so long. Unsurprisingly the typical reason is a failure to archive and then remove the old data. Nobody is taking out the trash! Data retention and archival policies never get baked into the original specifications as the sponsors seldom see any immediate competitive value from them. Eventually however a clean system becomes less of a luxury and more a necessity. Folks start to notice the pile of dirty dishes.

The effort required to retrofit for cleanliness is about the same as the effort for planning for this up front. Strictly operationally however it is better to delay this expenditure until a system becomes a mess as you will then have a better understanding of the corners with the most fuzzballs.

Artful Ownership

In a relationship-driven business world, job security and software maintainability are directly and inversely related. Gothically this happens both through a carrot and stick approach from the developer toward the management team and also from a developer to his own future self.

Promoting ownership stimulates the desire to create maintainable code. Much like the difference between renting an apartment and owning a house, the developer who recognizes that he owns (and is responsible for) the code has a vested interest in being able to go back years from now to understand how it works.

Management teams that swap out developers like machine cogs create a separation between a developer and his code: in essence the developer becomes a "renter" and he will deliberately obfuscate his work. One because he has no incentive to spend the extra effort to make it maintainable, and two because he will strive for job security by obscurity: he punishes the management for his insecurity by complicating their desire for robotic interchangeable developers.

In some cases though software may be a one-shot throwaway: if management expects the march of technology to obsolete a software platform rapidly then it might not particularly care whether it is maintainable. Hence if a developer would rather own than rent, he better darn well be sure that he works on creating a substantial and mission-critical system that his employer will still be relying on, with the same platform, six or seven years from now.

Yet... ownership is a chimera. It is an intentional feeling rather than an actual reality. Legally the developer does not own the fruits of his work. How can management promote the feeling of ownership even though the legal reality is otherwise? If it fails to do so it can never expect developers to create lasting platforms.

Artful Modality

In the many years that I've done software development I've noticed that systems written in-house tend toward two extremes. On one side you have useful software designed for a small quantity of people to do a very specific task, and on the other side you have somewhat clumsy software designed for two thirds of the company to support a wide variety of operations.

The smaller, task-specific software has tens and hundreds of flavors, yet it only survives three or four years until it gets replaced by a new incarnation. The multipurpose clunky software has just one flavor but seems to live for fifteen years, usually well beyond its prime. How come there is no middle ground?

Well just like in animation, sociology creates an "uncanny valley" of in-house software that doesn't comfortably exist. And this is due very much to the nature of people and the work that they do. In most jobs people tend to leverage certain knowledge and skills to support the company with specific tasks. A capable software designer can create very nifty systems that can maintain incredible complexity with the understanding that a moderately competent user will be trained on its use.

Such systems can be local successes even though they don't translate well to other users. And they don't survive for long because they don't incorporate the dynamics of both system-level and human-level changes. People change jobs and don't communicate all of the knowledge. Small systems fall into disuse because they are people and knowledge specific. They are still good for what they are good at: improving local productivity.

Large systems live past their prime for similar sociological reasons. The rate of change in people-skills overwhelms the complexity of linking processes. Linked processes become unlinked, and the training materials don't keep up with the changes in the software.

Large systems still serve a useful purpose however, to the extent that they organize people to work together. Where does the personality of a corporation really exist? In between the modes of small and large software.

More Artful Beauty

We must choose in a hundred little ways how maintainable to make our software. I wish to give you some guidance, some signposts: when you are deep in the trenches facing these hundreds of little choices, what key questions that you should ask yourself? By answering them whenever you are at a design fork-in-the-road, you will know which path to follow.

You can certainly go overboard on the maintainability spectrum. After all you do have a product to deliver. But a fair and reasonable amount of refactoring is necessary to allow the software to retain its value. So every time you roll up your sleeves and dive into the code, ask yourself these eight questions:

1) Got whitespace?
I like to see code blocked out into sections in a manner akin to how a writer blocks off paragraphs. I don't need your otherwise obvious comment lines in front of each section, just a plain intervening blank line lets me know that you are moving on to another train of thought.

2) Do you have to scroll right to read it?
If so then you are probably combining too many conditions in a single statement. Refactor into a nested "if" or if you're already deeply nested make a new function.

3) Can it have a better name?
Variables and functions should say what they do and do what they say.

4) If you were to place a debug checkpoint, what would be good to know here?
It sure compresses the coding when you do dataset.table.rows.columns[4], but it's a royal pain for debugging. If you're going more than two dots deep set a work variable for the object partway down.

5) Is your module getting too long? I chatted about appropriate module sizes in an earlier post.

6) Are the parameters that control the whole bailiwick commented at the top of your code? Are there too many of them in different places?
If you are gathering input parms from every conceivable source, maybe it's time to consolidate. You may even need to create an intermediate feeder object that collects and denormalizes all this stuff.

7) Are the logic sections... ah.... logical?
When I see nested "if" statements inside of case statements I cringe. Testing for not A or not B makes my eyelid quiver. Clean up your logic to test for eligible conditions (as opposed to ineligible conditions) and set intermediate Boolean flags.

8) Will you be able to understand this piece of code three years from now?
Perform the mental experiment of imagining that after you compile this into production, you won't see it again for another three years. Will you be able to understand what the code does? Leave yourself enough breadcrumbs to find a safe path back to understanding.

Yeah following these guidelines is a little more work, but as the saying goes: hey you can pay me now or you can pay me later.

The Art in Beauty

Whether we are consciously aware of it or not, in the midst of development we make hundreds of tiny decisions that reflect a choice in "beauty". Mostly these boil down to a decision for what level of elegance we will apply to a design approach: whether to hack away at something until it works or whether to design the code with an eye toward easing future maintainability.

In the old days we used the shortcut terminology of "table-driven-design" to signify the latter case, but nowadays designing for maintainability is much more than this (see for example these earlier posts).

For now though consider this interesting metaphor to beauty: just as a person can suffer equally from both too much beauty or too little beauty, the same can be said for the elegance and maintainability of a software system. If something is hacked together in an ugly and hasty fashion then not only will you have to support it forever, but you will grow gray or lose all your hair in the process. Conversely if you design something that can be maintained and extended simply by adding rules to a SQL table then your employer doesn't really require your continued ongoing services.

You must therefore choose an appropriate level of elegance in the same way that people choose their level of beauty. This has all sorts of philosophical and aesthetic consequences that I examine in a different category.

Artistic Specialty

To some extent folks adopt their software development specialization depending upon their own skills and tastes: do they feel more comfortable with a broader and lighter background or do they prefer to be deeper and more focused? Managers may also have a strong influence over how employees develop their skills simply by the tone and expectations that they set.

Obviously software development requires some specialization, since the technological complexity and the rate of change in our profession precludes a sole person from spending the time required to be a technical expert at all the myriad skills that span the available range of tools. If you wish to employ an expert a direct corollary will be that you therefore choose a specialist. If you plant a honeysuckle tree, you are going to get hummingbirds: that's the sort of specialist this tree desires.

Selecting specialists for software development presents some challenging issues. Unlike hiring engineers to design cars -- where once the model rolls off the production line the engineers move along to their next project -- hiring engineers for a software project is likely hiring them for the entire life of the system.

In this instance the problem that a manager faces is that quality, job security, and concurrency are in opposition. Asking your staff to specialize may give them greater job security and improve the quality of your operations, but they will lose their awareness of how to incorporate and apply new tools and hence lose their personal marketability.

If you hire a specialist to do the job then the result is going to be somewhat unmaintainable by anyone but that specialist. If you hire a generalist it will be far easier to maintain the resultant system, although the final product may well appear less slick and polished.

Ultimately then who you hire depends upon the "positioning" your company is trying to achieve. Are they looking for a short-lasting expensive best of breed solution? Or something long term, stable, and adequate? What sort of tree exactly are you planting?

Artistic Goals

Developing software is both about balancing competing forces and optimizing numerous goals. Down in the trenches all of this happens within the craft of describing to the computer what to do by using a programming language or development tools, but it is still beneficial to keep in mind at a much higher level what should be the general objectives of /any/ large software system:

+ Validity
Your software must meet its functional requirements

+ Parsimony
Keep it simple, avoid bloatware, use a declarative approach that saves intermediate values

+ Incrementality
Plan from the start to make your software readily extensible by incorporating real-world modeling

+ Perspicuity
Strive for clarity and aesthetics; make both the use and the maintainability of your software intuitive

+ Modularity
Remember the advantage of being maintainable and reusable, stay functionally focused, but loosely coupled

+ Resiliency
You need to be tolerant of system faults and provide excellent error recovery

+ Adaptable
Hardware can change -- be flexible for changing environments

+ Performance
Strive to be both efficient and fast

... and yes, these are in precise order of importance.

Artistic Ownership

The managers at many places I have worked confuse the fruits of our work (lines and lines of programming code) with “intellectual property” and with knowledge. It is a convenient deception, but it ultimately causes more problems than it solves. Let’s take a look at precisely what these concepts are, why code and knowledge get conflated, and the problems that ensue.

This is somewhat akin to confusing traintracks with a railroad. A manager walks in and says “oh, that guy is a direct path to Denver, that guy goes to our hub in Chicago.” But a railroad is a whole lot more than traintracks. It is also the locomotives, semaphore crossings, stations, leased rights of way, you get the idea.

“Lines of code” is a semi-fluid corpus of rules and instructions that implement business logic. These change as customer or management needs allow and they also get modified in reaction to bug fixes and refinements in technology. They are like traintracks. In nearly all cases the software code is the property of the owners of the company. So sure the railroad owns the traintracks. Yet there is so much more.

In the context of software development “Intellectual Property” is actually a narrowly defined term. Occasionally it may be a patent, but for commercially marketed software it is more likely a copyright. Most internal-use software (the bulk of what we create) is Intellectual Property by means of “trade secret” and employment work-for-hire agreements. So these are like the leases held on the rights of way.

The code isn’t an “asset” in the sense that a physical piece of property is an asset; it lacks a particularly liquid “market” and the rules and peculiarities it encodes keep little value when transferred to a non-related owner. Selling your traintracks to Australia doesn’t do them much good.

“Knowledge” in this context encompasses three overlapping concepts. First it is a domain-specific understanding of how things are supposed to work conceptually or legally. Second it is the cultural understanding of who is proficient at maintaining what. And third it is the historical insight into how things evolved to reach their current state. Software developers employ this knowledge (which resides primarily in their heads) to produce lines of code from the influx of rules, regulations, requirements, and bug reports. So the knowledge is the locomotives, semaphore crossings, and stations.

It is convenient for a manager to jump from the work-for-hire agreement to the (mistaken) impression that he “owns” the knowledge in developers’ heads. In this way he conflates the human “resources” through the midpoint of Intellectual Property to the endpoint of an “asset” that he owns and manages. He confuses the traintracks with the railroad. Although convenient, it is wrong in that it loses pieces in the translation, creating dangerous side effects. It is like thinking you can pick up the traintracks, move them to India, and suddenly have the Southern Indian & Pacific railway.

Knowledge isn’t transferable nor is it intellectual property. And if you move people from their areas of expertise the company effectively loses their knowledge. Sure it still owns their code, but after the company saddles somebody new with maintaining the code they have to reacquire the knowledge of how to do so. They have to relearn how to run the railroad.

Artistic Standards

SOPs (or as we often call them nowadays “best practices”) get the usual bum rap but I like them as an educational tool. Standards and best practices are fine to the extent that they promote a safe and productive workplace without adversely affecting a sense of job security. For example “Always use camel case” is a good standard to improve productivity. “Keep your source code in three places” is a best practice that enhances safety.

Wielding standards like an axe over your employees’ heads will only create defensive animosity. I prefer to use them as search terms when I want to find a template or an example to share with somebody. For instance I might Google “best practices SQL maintenance” and see what it returns.

If your employees feel however that the purpose of your standards is to turn them into interchangeable cogs then at best they will only implement them halfheartedly and at worst they will spoof compliance. Make positive, non-corrosive best practices your standard operating procedure.