Artful Coworkers

Employees may spend a fair amount of effort (especially in larger corporations where you find a concentration of regulatory specialists) mining the veins of gold sequestered away between the seams of operations and regulations. Sometimes this is explicitly an objective of management, at other times it is an implicit cultural undercurrent.

How you approach and handle this probably is more influenced by your ethos and personal beliefs than any of your other working relationships.

Society's regulations have both a written and an intentional form; the two are similar but not the same. To me, the intent of the law carries more weight than the letter of the law. You or your coworkers may disagree with the intent of the law, but unless you have a very clear and deep understanding of how the law came about you run a serious risk if violating its intent.

Resolving differences of opinion on this matter should be one of the first things you do before starting any large project that has a potential to move any boundary that affects this balance.

A full 65 to 75 percent of the politics of analysis happens on the boundary between employees and customers. Their interests are misaligned in any business because the distribution of profit margins either goes to one (in the form of wages) or to the other (in the form of competitive pricing). And you are developing a project that will impact sales or profit margin, because otherwise you wouldn't be working on it.

How this boundary moves however is none of your business, it is beyond your concern. Yet at the same time your analysis will sink into the mucky depths of oblivion unless you manage the politics of this boundary to appeal to both sides.

To the employees you must present an image that your creation will make their life easier without jeopardizing their jobs. To the customers (vis a vis the management) you need to present an end point that affords them greater efficiencies and value for their dollar. You may be unable to guarantee that you can sufficiently meet both targets, but it should be your objective nevertheless.

Artful Query

Every so often I get called upon to troubleshoot the performance of some SQL queries (that I didn't design). Frankly I view this activity as both a waste of time and an indication of severe design failure. Yes it /is/ possible to optimize queries post hoc, and if you must work your way through this task I will give you some guidance shortly. It certainly isn't, however the optimal way to get work done.

First let me explain why this is a design failure. A critical task during development is building a test environment mocked up to full scale. If you fail to do this then you won't fully appreciate how to best design your queries to begin with.

Databases are all about reading data from disk. Yeah sure there is some writing too, but this is typically dwarfed a couple orders of magnitude by the task of reading the data. Databases are heavily IO bound.

Two things contribute to the time it takes to read table data or index pages: the "width" of each row and the quantity of rows. The width of a row is determined both by the number of columns as well as how much data each field contains.

Do you want fast reads? Use narrow tables and indexes (or use vertical partitioning). To limit the quantity of row reads, use horizontal partitioning or appropriate cluster indexing on tables with millions of rows. And guess what, you can't choose the best partitioning approach during development until you scale up.

Understand the above and you will have good baseline strategies for designing queries in the first place. Hence given the chance to "optimize" existing queries, a far better approach (if you have the time) is to refactor the application to take advantage of this knowledge.

Persist sub-queries to narrow temporary work tables. Target your query sequence to limit the selection criteria to the typically minimal rowset before finally joining in the "wide" information.

If you can't afford a full refactoring though, then you will have to resort to basic database query optimization techniques. Learn to understand the "explain plan" and to identify the time resource hogs. Use nolock hints where possible. Experiment with other hints to modify join order. Direct reads by a key are fast, full scans are slow, so build indices on columns used in joins.

Yeah you can soup up your Toyota Corolla queries to run at 160 miles per hour, but it would have been far better to design your queries like a Ferrari from the start.

Artful Bug Squashing

Bugs are just a fact of life in software development: nobody likes to see them, but so many demands confound a programmer’s time that it becomes impossible to design down to every last exception. Sometimes the bugs even pre-exist in the tools that we use, the development environments, or the compilers.

However much I’m reluctant to admit it, all the same we’ve got a vested interest in keeping a fair share of bugs in the works; if our offerings are perfect then we essentially program ourselves out of our jobs. Imagine if all the schoolkids picked up every last piece of trash and wiped down all the tables when they finished lunch. What would all the janitors do?

You can prevent some bugs by utilizing certain development techniques. One of my favorites is to rename all of your variables and functions to make them distinctively descriptive and non-overlapping. This adds another couple of days to the project, but in the long run it pays off twice: first in easier maintenance with standard names. Second the process of editing and replacing catches many bugs by itself.

As I mentioned in an earlier post, another useful trick during development is to make specific plans to keep audit trails (or trace logs). I usually ask the staff to define a global Boolean called Testing that when true should cause nearly all activity to log to external files.

When persisting to a database I prefer to avoid deleting records: instead I include an “obsolete” flag and set it True when the record should be ignored. Depending on the audit requirements of your system you may even prefer to append a copy of a record for modification, so that if necessary you can always go back to prior versions of a row (note however that this strategy is quite costly from a storage perspective).

Legally many systems also need to track the identity of the user and a timestamp when any rows or fields were modified. If you also include a field to identify the module name and version of the modifying process you can greatly aid version-debugging and historical corrections.

Tracking and fixing bugs is an art of its own. Under pressure rank a bug’s squash-priority by the nature of its criticality. Certainly items that involve the health and safety of your clients, your staff, and yourself come first. Items that adversely impact production cycles are the next highest importance. Bugs that interfere with someone accomplishing their work come in third (especially if you lack an immediate workaround).

Next you will have a large hodgepodge grouping of nuisance bugs, resource bottlenecks, and business-critical enhancements. Everything else will likely go into the bottom of your priority list for “when you have the time”. A useful trick-of-the-trade is to dedicate one-half to one day each week for the lowest priority items; that way you can at least address some of them. Keep the list of unfinished items however… at some point several months to a couple years down the road you will want to revisit many of them to see if they still have merit.

Artful Matchmaking

Fanning the spark of an idea for an IT project is always a tricky, precarious task. Somewhat like internal marketing, a Business Analyst is selling a vision. More than this however the analyst mentally surveys the capabilities and availabilities of everyone he knows and attempts to match this to a vision that he can impute to the desires of management. So it feels very much like matchmaking, except in the business-to-project realm.

To the managers you are selling an investment. Your job is to convince them that they will get an honorable and lucrative return on their investment in equipment and manpower. To the software developers you are selling professional advancement, knowledge enrichment, and a reasonable amount of job security.

And similar to romantic matchmaking sometimes you have to use a bit of deception in order to get the parties together despite their different opinions. Many times you will actually have a deeper awareness of the best interests of the managers and programmers than either party will know alone. Hence a bit of artful promising goes a long way. Is this unethical? Not if you're careful to make sure that the ends justify the means.

It's all about some balance and possibilities and getting folks over the hump of their current comfort levels to move them through a low spot to finally arrive at a new, higher comfort.

When an idea is just a concept -- before it is even a glimmer in a manager's eye -- when it is just a spark of "I wonder if it might make sense..." or "maybe we should look at..." the Business Analyst combs the psychic brainwaves of the company's employees and decides what is Possible.

Artful Independence

If you design things from the start with testing in mind, then your finished system will be both stronger and easier to maintain. One way to accomplish this is to provide a "testing" checkbox (or menu pulldown) in your user interface. Doing so causes a complete design cascade in how you think about development.

It sort of provides you the same comfort level as if you were slipping behind the wheel in a Driver's Training car. It feels like authentic driving but with the added safety that if you forget to step on the brake you actually won't damage anything, as the instructor is watching out for you anyway.

It also allows you the wonderful opportunity to test production interfaces with a development-version of the backing database. Using this practice also promotes looser coupling between components. Of course, make it abundantly clear on the interface (say by changing the global form background color) when the client is in test mode.

This design method does incur some additional logical overhead. Usually you will set a "Testing" Boolean and have places where you evaluate it to set, for example, the database connection strings. This means you also have to provide both the production and development paths and parameters in your config fie.

The saving grace with this approach is that it pretty much clears you from the embarrassment of the all-too-frequent mistake of accidentally copying the development parameters into the production installation. So declare your independence from installation woes: code for testing from the outset and make the development configuration selectable from the interface.

The Art in Two

I've found that you can add a considerable amount of value and "cleanliness" during construction by designing and developing your software on two separate computers. Not simultaneously, but rather in an occasionally alternating fashion. The main point of this is to assure that you have enough of the referenced dll libraries in your project (along with any ancillary files) without constantly cranking around the older useless modules.

Make a full copy of your project folder, clear out the post-compile (and debug) junk, and then zip it up. Mail it to your other computer. Now on the receiving end unzip that puppy and see if you can compile it into something reasonable.

If you use two vastly different computers for this process, say a laptop running Windows10 and a wide screen desktop running Windows 7, you'll get the added quality checkpoint of verifying that your interface is presentable across platforms.

Yeah you need to install your development environment on both computers but this has some subtle advantages as well. Go ahead and square off your development: use the power of two!

Artful Purchasing

When you are charged with buying a software package for your company you will brave the pressure of many conflicting forces. Of course the vendors and their employees have a near manic desire to have you select them over their competitors. Be very wary however of vendors who make the first effort to contact you... I mentioned one such salesman horror-story in an earlier post. It's always better to be at the "selecting" end of the vendor relationship than at the solicitation end.

Your staff may have their preferences based upon their own familiarities or tastes. Yet considering the fiduciary responsibility you owe your employer, your overriding role would seem to require you to execute due diligence to evaluate the vendors on a level playing field.

Begin with some exploratory internet searches to find software with keywords representing some of the more specific requirements. This will give you a clue to the names of the main contenders. Then run a query with the top three package names, along with the words "comparison" or "review." This should retrieve some industry zines with a more thorough examination of what the vendors offer.

Now create a spreadsheet for yourself with "weighted" line-items for the characteristics you need. Remember to add lines for Vendor stability and service record: you're not just buying software, you're buying a service relationship with a company. How are they run? Are they stable? How is their customer service? It can easily take a few days to conduct a thorough review.

After you narrow the field ask the top two choices if you can get a demo. Since the actual folks in your company who will be using the package day-to-day may not all have the skill levels to be comfortable with the software, allow them time to examine it and gather their impressions.

Large purchases may require considerably more effort in the review, including issuing formal RFPs, evaluating the impact of any BPR that may need to occur, and determining how you will integrate the new purchase into existing systems. Don't be shy to retain professional assistance for larger selection efforts.

Artful Ownership

In a relationship-driven business world, job security and software maintainability are directly and inversely related. Gothically this happens both through a carrot and stick approach from the developer toward the management team and also from a developer to his own future self.

Promoting ownership stimulates the desire to create maintainable code. Much like the difference between renting an apartment and owning a house, the developer who recognizes that he owns (and is responsible for) the code has a vested interest in being able to go back years from now to understand how it works.

Management teams that swap out developers like machine cogs create a separation between a developer and his code: in essence the developer becomes a "renter" and he will deliberately obfuscate his work. One because he has no incentive to spend the extra effort to make it maintainable, and two because he will strive for job security by obscurity: he punishes the management for his insecurity by complicating their desire for robotic interchangeable developers.

In some cases though software may be a one-shot throwaway: if management expects the march of technology to obsolete a software platform rapidly then it might not particularly care whether it is maintainable. Hence if a developer would rather own than rent, he better darn well be sure that he works on creating a substantial and mission-critical system that his employer will still be relying on, with the same platform, six or seven years from now.

Yet... ownership is a chimera. It is an intentional feeling rather than an actual reality. Legally the developer does not own the fruits of his work. How can management promote the feeling of ownership even though the legal reality is otherwise? If it fails to do so it can never expect developers to create lasting platforms.

Artful Verification

Sometimes despite your desires to remain logical and analytical (you did after all choose to be a system design professional) you will find yourself in a position that requires you to pass judgment on others. This is never more true than when you are selecting a software package vendor. You have to be friendly but still cynical; vendors send you salespeople who know how to read your desires and make promises that cater to them. As a buyer you need to keep a critical eye toward what you know to be the facts.

Back in the early 1980s, when personal computers had 5 1/4 inch floppy disks and still went zyg zyg beep to start up, when ten megabyte hard disks were all the rage, I had the opportunity to review software vendors to provide hard disk security systems. The concern back then was that somebody could boot up from a floppy, switch to the C drive, and read all your data.

I was impressed by one vendor claiming that they not only could provide complete security, but actually prevent an unauthorized person from accessing the hard disk at all. As I had already reviewed the bios code and was familiar with how a computer loaded its bootstrap, I found their claim to be, eh, interesting. I asked to see a demo and they arranged for a salesman to drop by.

He gave a fairly nice demo, showing how their software could be configured so that only specific people could see certain files. at the end of the demo I commented "gee nice. Hey I was wondering about this item I read here about total protection." I told him that I seriously doubted their software could prevent somebody from destroying all the data on the C drive. He adamantly assured me that it would. Well then, was he confident enough that he would let me have a crack at his demo computer? Without a flinch he said sure!

I carried up a boot disk with the old DOS debug program on it, turned off his computer, put the disk in, and turned the computer on again. Zyg zyg beep! At the DOS prompt I typed debug and then used a command to write all zeroes to the boot sector on the hard disk. "Well uh, I think I just wiped your C drive," I shrugged. The salesperson didn't believe me. I removed my floppy disk and watched in a pitying amusement as he tried to restart his computer. Naturally the C drive was no longer readable at all (now he would have to reformat it). Needless to say we didn't purchase their software. Moral of the story: when you are buying software, trust... but verify.

Artistic Communication

Technical documentation is one of the least appreciated aspects of software development -- not the written words themselves, but the process of creating them is deeply undervalued. Effective technical documentation is like a Hail Mary pass from the developer in the present to the developer in the future with the intervening crosswinds of bug fixes and changes to the business rules complicating the reception.

The developer is usually the best person to author technical documentation: he understands the implicit trail of bread crumbs that he has already left in the code for somebody else to follow. If he was involved in the project from the start then he can also relate to the politics and priorities that the team faced from the design and implementation stages.

A technical document should begin with a Background section that describes the rationale behind the project and the challenges that sufaced during design and implementation. This should alert future developers to any political or technical potholes that they should avoid.

Next an Architecture section should briefly describe data flows, timing constraints, and security layout. Next some Entity Relationship diagrams: not the whole thing but just the key tables and their relationships. Then a high-level UML and quick use-case diagram, focusing on how employee responsibilities overlap with components of the system. Finally, reference the design specifications that can, at this point, direct the developer to the whole ball of wax.

Good technical documentation isn't extraordinarily deep or complicated provided that you focus on its main intent. Sure you understand the system (because you created it), but standing at the other end of the football field a future developer can easily get buried in minutiae. Wipe away those confounding crosswinds and throw that future developer a touchdown.

More Artful Testing

In the software development world we have a whole cabinet full of testing tasks that can contribute to quality work; here is a list of those most commonly used, in an approximate order of where they happen along the development cycle:

Impact Analysis
Usability testing
Code Review
Risk assessment
Stress testing
Test bed preparation
Test case creation
Test driver development
Functional testing
Regression testing
Boundary Value Analysis
Branch coverage
Error seeding
System Test
Parallel testing
Documentation Review
Integration testing
Acceptance testing
Post implementation validation

Therefore all of our software should be perfect and bug free. Right? In the real world our ability to spend time and money on software testing gets constrained however by the dynamics of staffing, skills, and delivery schedules. So much as I described in this earlier post about appropriate balance, we must carefully choose which test tools to remove from our cabinet to apply to each project. And that my friend, is why software still has bugs.

An Artful Environs

In an earlier post I discussed the vagaries of herding the cats of developers. Here I want to chat a bit about motivation of creative developers in a more diffuse sense -- about how to create the environment that allows for quality and valuable software to "materialize."

This is nearly a diametrically opposite approach to the conservative management of a formally enforced SDLC. And it only tangentially has to do with software. First, consider in general what motivates an artist: a desire to be creative, a desire for recognition, a desire to be challenged, and a desire to stand out amongst their peers. Maybe they seek glamour, or grunge, or just to make an aesthetic statement. Maybe they want to promote a socially redeeming view.

Considering the amount of demanding self-discipline, focus, sensitivity, and work that this requires, how do you create a comfortable supportive space in a world of abrupt changes, insecurities, hidden stakeholders, control freaks, and conflicting priorities?

Mostly it helps to be honest and open in your approach. You don't have to necessarily build a consensus, but it does help to lay all of your cards on the table: acknowledge both the staff needs and the strategic challenges as well as how you plan to enable the former through the onslaught of the latter.

Yeah that by itself seems like a considerable undertaking, but the mere process of discussing it with your staff and your superiors will get you on the same page. At least it will settle the expectations toward how much of the developers art you can bear under the freedom of the muses versus how much of their work must yield to the dogma of the Methodology.