Software Engineering Rant

Criticizing Those Who Came Before Us 

Why is it so easy for us to criticize the developers that came before us? The newest generation of developers have written numerous blogs, articles and books to inform us how traditional approaches to software simply do not work, yet we continue to make the same mistakes that traditional software had. We try everything from overworking programmers to producing thick rims of documentation that no one follows or uses.

Still, after years of practicing software development many businesses continue to believe that developers must deliver a constant stream of code, thinking that this is of more value to their business than taking the time to create a set plan or a real design. Holding to this practice causes businesses to steadily miss out on major profitability.

This is 2021 and projects are still scrapped because of poor communication that is mistaken for poor management, unforeseen technical roadblocks and incorrect requirement gathering. Pushing developers to follow these practices causes projects to be full of dead code, hacks, bugs, and either under engineered or over engineered systems and subsystems. They continue to find that strict single phase waterfall life cycles fail, yet they follow in the same footsteps, repeating the same mistakes. Many managers have only a shallow understanding of industry buzz words like agile and enterprise design, and just as many programmers are incapable of, or uncomfortable with having direct communication with management, this causes complete disfunction.

It is both naïve and egotistical of us to think that our current process of software development is not fundamentally flawed. The view that we are better than our previous generation is not limited to software development and certainly is not a new concept.

As we continue to adopt change and grow, it is inevitable that future generation programmers will have just as many reasons why the methods we have used are inadequate.

So...What are we doing wrong?

Constantly Re-inventing Software Development

We are constantly trying to improve the way we develop software, but too many times instead of stabilizing the tools we have, someone creates a new system. These new systems bring on a different set of problems that developers are not familiar with. Where we should be able to stay focused and improve the systems we have someone comes along with an idea that focuses on one methodology, pushing for a complete change in the way software development works requiring more training and research.

Software Engineering is a Doomed Occupation

Despite years of progression on the subject, software engineering is not a profitable career path. Instead it seems to be something that developers practice in secret for fear of perception that there is lack of productivity. Software Engineers and Project Managers are just looked at as overhead. The documentation and structure they add to the project is only looked at as nice to have at best and a nuisance at worst.

The title of Software Engineer is often bestowed upon a developer with no more meaning to the company or the organization than a person who has been coding for a long time. Engineers study how to make sense of large systems, they make decisions as to how the problem should be broken into smaller pieces, and have the ability to reduce maintenance and modification cost that still make up the majority of the cost of software systems today.

However, software engineering is not recognized or valued for its accomplishments. The perception among software vendors instead is that they don't have time to wait for an engineered solution that instead they just want a few developers to do what they are told. 

Business managers, therefore take the responsibility, with no prior education or practical understanding of how software should work. And, actually we have begun sending business majors to to become certified Software Project Managers. They usually enter a software development team with a bunch of memorized terminology without having a firm understanding of what they are studying.

The Blue Collar Programmer

Whether you call yourself a Software Developer, Software Engineer, Lead Developer, System Architect. Most companies only see a programmer, a coder, a technical resource. And although you liken yourself closer to an architect who designs large buildings and structures, when you produce code you are no different than any other producer in a blue collar occupation.

The pay may be comparable or better than a structural engineer, architect or even a lawyer. When your job is to produce the product you are a low level worker and details of design are just a side note to your job. The pay may be better than say a brick layer or plumber but the company still views you the same way.

Nothing highlights the attitude more than the growing and now shrinking fad of outsourcing your programming to the lowest bidder. After all, programming is no different than any other blue collar, low skill labor. The client will reason to himself that there is a certain amount of work that will need to be done, there is a cheap source of labor and an expensive source of labor. Why would you choose the later when the company can scarcely tell the difference in the end result.

Even worse, in my humble opinion, is the absence of responsibility of the final solution. Clients are more than happy with solutions that were done quickly. Without regard for long term consequences, they glad except poor solutions that seem to do the job until they are actually used. Some are so poor in fact that the buttons and windows are there but there is no actual implementation or it was never tested.

Upper management rarely holds a programmer responsible for  poor quality work.  

The Laziness Virtue is Becoming a Hindrance

For years I have proudly proclaimed that laziness is a virtue to a programmer. By laziness I really mean that I don't want to do more work than is necessary. If I can find an api, web service or write a simpler method that reduces my code or find a way to completely eliminate the need for what I am writing than I am happy.

However, the development community as a whole has adopted this to the extreme. We are constantly attempting to pawn our work off onto open source SDKs, third party vendors and 4th generation tools. So much so that programmers no longer understand simple algorithms, data structures and design. Business programming can now be likened to being a general contractor, managing other works to make a bigger systems.

The problem with this is that we have not really made progress. Simple problems that would be obvious for us to write a custom function for 10 years ago are now available on the open market. Managers don't want the developers wasting 2 hours on writing anything they could buy or use for free.

Third party solutions often come with an equal amount of problems. Very few of them are truly stable enough to reduce our work load. What they are really good at is appearing to work. After being put in a real world test they more often than not seem to fail. And in order to find solutions to the problem with the third party tool, developers are forced to study at detail the inter working of it. It cost 2 hours of our time to download and reference a third party solution to our problem and costs the company more time to make it "really work" then it would have cost had the developer remembered how to implement a trivial algorithm.

As a solution this conundrum, businesses adopt large swiss army knife frameworks. These frameworks are often too large for the vendor to reasonably test and end up having more flaws than the tons of smaller solutions available. 

The Failure of 4th and 5th Generation Languages

Like the search for the holy grail, developers are in constant search for a next generation programming language that can increase productivity. However, our attempts at implementing them have been hindered by the instability of existing tools, and by a fundamental flaw in the logic that says we need a 4th or 5th generation toolset.

The idea that 3rd generation tools such as Eclipse and Visual Studio and programming languages like C++, C# and Java have made us more productive than languages such as pascal, ADA and ansi C is misguided at best. What these toolsets do is attempt to guide us into making a well designed solution.

However, software design in itself is not promoted in the industry. As much as the tools can help you design software, they can not design it for you. In order to design a system you must understand fundamental design methods. Methods that have remained the same since pre-1970's and have since only changed in terminology and not in meaning.

Replacing Software Design are Canned Patterns and Solutions

After decades of attempting to push fundamental software design, we have given up on designing systems and instead have taken to finding a canned set of patterns that solves problems for us. We have then pushed this on to the "great unwashed" developers of the past who simply followed traditional design methods based on logic and engineering principles.

Admittedly, these patterns in our solutions offers an improvement over non-designed systems. However, they are most often used to replace any kind of forethought in a project. Lets do a 3-tier system and let the system play out how it plays out. Make sure to put in factories and dependency injection, that usually makes better systems.

The next step inevitable, is that our toolsets push a set of patterns into your solution. In an attempt to improve code for you, frameworks such as Ruby on Rails and Microsoft MVC have made it fairly difficult to not use these patterns when using their tools. Just as C++ and C# attempt to improve code by forcing the Object Oriented Paradigm on its developers these toolset often have no effect on developers that do not understand design principles in the first place.

Non-Technical Problems are Solved with Technical Solutions

Solutions to the problems the industry faces are not hard to find. At the moment there is no lack of frameworks, toolsets, higher level programming languages, patterns or methodologies. In fact they may do too much for us.

When it comes to 3rd party tools, what is needed is a correct implementation to begin with. 3rd party vendors, are capable of producing high quality well tested, well documented frameworks. But so far they have not found it profitable. We as developers seldom hold them to a high standard. We have been more than willing to put up with any bugs and flaws in their systems.

Instead of 4th and 5th generation languages that write code for us. Would it not be more productive to give us meaningful error messages? And, instead of generating sql statements and the programming code to go with it. Could we make the database api and sdks work the way they should? Is it not more profitable to the developer to remove poorly designed classes and non-working methods in the current api then to create canned scripts to deal with generating the obscure code.

Furthermore, would it not be more profitable to simply teach fundamental design principles than to blame our poorly written systems on a lack of a new concept, pattern, tool, and or software methodology. Like sheep, the industry blindly falls for the latest snake oil. RAD, Agile, Extreme Programming, SCRUM, DevOps and any other buzz word meant to fix the current ailments all seem to address a problem that already has a viable solution.

However, the real solution to the problem is somewhat hard to swallow and is actually non-technical in its nature. The problem lies between the keyboard and the computer chair. Developers and software managers for that matter, that are properly trained and have a discipline about their work will produce high quality, easily maintained systems.