How do I Deal With Major Mistakes?

Every professional, engineer or not, feels extremely distraught when they mess up. However, while some professions have the luxury of only having their failure visible at a limited level of exposure, most of the time software engineers that mess up tend to get a large amount of negative attention very quickly. The last time I screwed up majorly, a Vice President at my organization (a really big one) noticed. And he wasn't happy about it. My boss wasn't happy about it. My team wasn't happy about it. I REALLY wasn't happy about it. Really, there was just lots of unhappy going around, because it was an expensive blooper that cost a good bit of additional development and operational overhead to correct. While I can't put a literal dollar value on what it cost my company, I can estimate the number was probably somewhere in the $10K (or more) range, mostly as lost opportunity cost or developer hours spent fixing it. This wasn't my first screw up, but (as of now) it's certainly my most expensive one. And yet, when the proverbial "pile" started growing, I kept my cool. Getting worked up and angry at myself could have made an already bad situation much worse - so instead I just hunkered down and figured out how to fix it.

My biggest mistake in this situation was in trusting a deployment pipeline to give us the right answer about the deployment of the application we were working on.  A critical bug was discovered in production, and we had assumed we fixed all of the edge cases that caused the bug and deployed confidently on this assumption. It turns out there were more edge cases. Lots of them. And our deployment pipeline had no knowledge of them, so it happened again. After we confidently said it was fixed. Ouch.

So, several things had to happen after my major mishap. First, and foremost, though - I took ownership of it. Several people were either directly or indirectly involved in the writing of this code and the decision to deploy it, but at the end of the day I had to have the ultimate final say-so that we'd gotten the solution good enough to face our (internal) customers, and I signed off on the solution. So, I found my boss and told him exactly what happened, as honestly as I could, about how this code could get into production in a still broken state. In order to maintain respect within my team and organization, taking responsibility for the error was the best way to show that my professional integrity, though sullied, was far from failing completely.

Next, we worked as efficiently as we possibly could to get the issue fixed. Thankfully it was a job that ran once a month, so between runs we had a little bit of time to clean the egg off of my face.

After that, we went through the process that lead to the broken deployment.  We laid out all of the decision points that could have lead to the failure, and determined where I could make some changes in my own methods for reviewing software to help prevent errors like this in the future. What this basically amounted to was learning where the deployment pipeline wasn't reliable, and finding out how to deal with this in any further deployment - with a manual checklist.

At the end of it all, once I'd cleaned the egg off of my face (with my team's help) and grudgingly accepted the major ding in my quarterly goals review, the most important thing of all was to review the bad time as a learning experience.

Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience? - Thomas John Watson Sr. - First CEO at IBM¹

It's possible to learn from success - at least as far as knowing the things that you can repeat to keep being successful. But the kind of learning that you gain from a failure is akin to the kind of learning you do when you put your hand on the hot stove for the very first time. It stays seared into long-term memory, and helps to propel greater things. We all need to stand on the shoulders of our past mistakes, knowing that we're strong enough to make different mistakes in the future. At the end of the day, if you work somewhere that you respect, and that has respect for your talent - you'll get the opportunity to prove you can be trusted, and you'll also get the opportunity to prove that sometimes - mistakes are just really expensive training exercises.

---

1. If anyone has a source where this quote was captured originally I'm happy to cite it here.  This is just a citation of someone else that re-used it. http://www.inc.com/murray-newlands/30-quotes-to-remember-when-recruiting-for-your-startup.html

Should Code Have "Come Back to this Later" Comments?

We did this today.

    # HACK: Fix this later, this is awful
  ssh $USERNAME@$TARGET '
    for file in $(grep -rl some-string-we-need-to-replace /some/directory/to/do/work/in)
    do sudo sed -e "s/some-string-we-need-to-replace/the-string-to-replace-it-with/g" -i $file
    done'

I'm not particularly proud of writing code of any kind that has these kinds of statements in it. I tend to repeat the mantra "If we don't have time to do it right, when will we have time to do it over?" in my head somewhat repeatedly. I've even known to say it out loud on occasion. And yet, here I am staring at a bash comment with the big green word HACK in it. I plan on doing the walk of shame later, for now I'm going to explain why I feel like there's a few of important rules to assume when you're writing code at work.

1. You're never going to come back to code that you write "I need to come back to this later" in the comments. While this may not always hold true, in general if the software "works" as is, you're going to have a very difficult time convincing your boss/customer/whoever that you need to go back and fix it if they don't see anything broken to begin with.
2. Any time you do write "TODO", "FIXME", "HACK", or your favorite word that is supposed to make you fix it later, you need to make your boss/customer/whoever understand that you do need to take the time to go back and fix it later. This is usually very difficult.
3. Sometimes real deadlines in the release environment make it so that we have to cut corners and just get something deployed. We don't have time to do it right, right now, because we have a customer waiting for it. Just be ready to do it over when something unexpected happens because of your TODO, and be ready to get a good bit of egg on your face if it does happen in the wild. Be VERY selective about how and when you use TODO statements in your code. Production code with TODO in it doesn't seem like it belongs as production code (to me).
4. There's an exception where you're doing something like "TODO: Need to make sure to remember that when we're implementing feature AWESOME-1234 that this is the right place to put controller/domain/dao/magic logic." TODO reminders about future new features are OK, most of the time.

We all have to write code we're immediately sickened by when our fingers type it. Hopefully we all find ourselves in roles where we don't have to do it all that frequently. But, it does happen (see: http://www.commitstrip.com/en/2015/05/29/always-stuck-somewhere-in-my-head/) - so we need to be careful about how we exercise our powers of evil getting things done. Otherwise we may end up writing "TODO: Find new customers because the old ones don't trust us any more" - and nobody wants to have to write that.

Should I Have a Facebook Account?

Poll Results As Of 6-3-2015

In order to find out what people think of my question, I put up a poll on Google+. I ditched Facebook some time ago, and thus far I have to say that I find myself with more time for blogging and other productive interests since I don't have one. I also find myself not compulsively checking my phone every 2 minutes to see if <so and so> ate <that really cool designer food stuff>.

All kidding aside, my professional/personal network is pretty solid, and I'm not entirely certain I'd make it any stronger by adding Facebook back into the equation. Since I haven't been on it in a few years, I honestly have no idea what kinds of feed grooming tools and other signal/noise toggling features they've built in to the service to make it more useful for the customers.  I can say that without much doubt, Facebook provides their development staff with all kinds of amazing things: http://www.businessinsider.com/facebook-vs-google-best-employer-2013-11, and I support any company that treats developers like rock stars. However, if I get back onto Facebook I want to maximize it's potential for helping me propel my career, network, and life goals.  Here's the things it has to offer to be worth it for me:

• More readers on this blog
• Stuff that I want to read, not just random nonsense that I don't care about
• Connectivity to people I care about, not to people who I barely know

I really don't know if I can curate a Facebook account well enough to accomplish those things up there. I suppose it's probably possible, but I really want to protect myself from being consumed in it again, I've really enjoyed my Facebook hiatus thus far and I don't want to find myself coming back to this blog in 3 months saying "Well, I quit Facebook, again." Garth Brooks and Michael Jordan can retire more than once, but I'm not sure if it'd do me any good ;).

So, what do you think - honestly? Can Facebook really be a valuable tool, or is it more of a time waster that makes Mark Zuckerberg a ton of money?

Which Programming Language Should I Learn?

It's college graduation season, so it got me to thinking a little bit about the important infancy of our careers. For many of us, that infancy starts in a university classroom where we don't get to pick which language we're learning, the professor(s) pick for us. When I was in school my first class (and first class I wrote) was in C++.  Every subsequent class after that point was in Java or C (except my programming languages class, which was in... a bunch).  My senior project was C# 1.5. So, even in the short amount of time (relatively speaking) I was in school - I got exposed to 4 languages while learning the craft.

Let me make something very clear - I do NOT think the right way to think about computer science or software engineering is to learn the syntax of a specific language. In some ways, programming and carpentry go nicely together in this regard. A carpenter can use a circular saw, a band saw, a handsaw, a jigsaw, and many other kinds of saws that I know nothing about because I'm not a carpenter. But if you were to ask a carpenter "What's the best saw to use to build these kitchen cabinets" he might look at you sideways, because a saw is a tool and he'll probably use several different ones building your kitchen cabinets. Languages are tools. Saws are tools. Let's make this point VERY clear.

Now that I got that out of the way - once you have a fundamental understanding of what a call stack is, how arrays work, what data structures are, etc. etc., THEN we can talk about programming languages.  On my current project/product here's our choice of tools/languages:

• PHP
• AngularJS
• Ruby (Chef Scripts)
• MySQL (Not really a programming language, but each flavor of SQL is a bit different too)

Is this the right set of tools for everyone? Absolutely not. But for my company in our current situation it's the set of tools that makes the most sense for the solution we're building.  In order to best know the direction in which most companies are moving for writing their software, there's several places to look to know what the most popular technologies in use are.

1. ThoughtWorks technology radar - this thing does a great job of showing whats gaining momentum, what's losing momentum, and also explaining why you should care. Broken down into 4 categories (Techniques, Tools, Languages/Frameworks, Platforms) it really helps to zero in on industry patterns.
2. TIOBE Programming Community Index - if you want to know if your favorite programming skill is in demand right now, this is a pretty nice place to look
3. Indeed.com Job Trends Graph - What I really like about this is that you can plot it in relative mode or absolute mode. Relative mode is great for showing how strongly something new may be growing, even if it's current market share may be minimal. Here's a couple of neat relative graphs for your viewing pleasure. Just to get an idea of why relative and absolute are different, switch graph modes at the top right of the graph - you'll see what I mean.
1. AngularJS vs. jQuery
2. Java vs. Python 
3. Java vs. PHP

I'd love to hear if you have any specific places you like to go to keep track of emerging trends in the world of engineering. It's nice to know what's coming our way, so that we can remain better prepared for whatever's coming next.

What Am I Reading?

A funny thing about our jobs, we do two things for a living. We write and we read. Yes, we get paid to read. In fact, the more we write, the more we have to read about what we are writing. As the author of this blog I produce content once a week. As a consumer of content I read multiple articles, multiple times a day. I also read books on occasion. While I could list the things I read, look for individual in depth reviews of the most important stuff I'll make individual posts. Instead I want to write a little bit about how I decide what I find important enough to read. This generally falls into two categories.

1. Learning how to best implement code/configuration x on platform y
2. Theory and best practices

So, if you want to know how I learn to implement something, it's pretty easy to explain. If I can't ask an expert in my company, I ask the internet experts at StackOverflow.com. Usually my questions are already sufficiently answered on there. When they are I hit the up vote button. Every great once in a while I have a question about something that nobody has answered before. Even then, StackOverflow to the rescue. Really, most of my technical implementation research comes from any of the family of StackExchange sites. It's a wonderful platform for people like us to share our expertise and pitfalls with one another.

The more subtle and somewhat complex reading list I have revolves less around actual technical implementation and more around architectural/developmental theory. Figuring out how I want to best expand my horizons in these areas - I'm still working that out. Here's a couple of things I do to find articles that I want to save for later:

• I follow other content producing developers and companies around on Twitter, they tend to have useful things to say somewhat frequently.
• I follow content producing developers and companies around in Google+
• Because of the links I read, my phone makes recommendations about a few things that might be of interest to me
• Oddly, I don't currently have an RSS feed reader that I rely on. I've played with having one in the past, but I ended up just ignoring everything it captured, so I stopped using it
• For things I read that I find myself regularly referring to I use Evernote capture and switch to article mode
• I have a close knit group of colleagues that recommend content to me

So, long story short, for the most part I look for the right kinds of people/companies through my social media channels. I ditched Facebook some time ago, mostly because my signal/noise ratio was heavily out of balance. I may venture back into RSS reading again some day, but honestly my social media interaction provides me with enough rich content that I'm not sure I'd find additional time to try to track more things via RSS. What do you think? How do you decide what's worth your time to read? I'd love to know how my blog got onto your reading list, so leave a comment to let me know!

Who Do You Look Up To?

As software engineers, we have the blessing of a vast peer network we can tap into. While many other industries also have great peer networks they can interact with, we as engineers get to interact in places like Stack Overflow, and we get to find other people that love doing work in code as much as we do. Though this kind of network provides us with a collective knowledge base, one of the things that it doesn't provide really needs to come from something more interactive and direct. I recently started reading Soft Skills by John Sonmez, and on a whimsy reached out to him on Twitter. As it turns out, John likes interacting with fellow geeks quite a bit, so I like interacting with him personally.

Yet, even still - while John's advice provides me great value, he doesn't fill a critical role that I think every engineer (or, really, anyone who works for a living) needs. While it's great to have interactive peers and people smarter than you influencing your forward career path, I think having a someone to call a mentor makes a huge difference as you through your professional growth. In my case, I consider my professional mentor this man: Jack Bader.  As far as technology professionals go, if you measure success as loving your job and getting paid really well to do it, Jack has quite a bit of it. So, in that regard I feel like Jack can teach me a great many things about the path I'm on for my career.  And, as far as I define mentors, a mentor is someone that you can email, or send and IM, or a text message, and likely get a favorable response within the next several hours. A smart colleague is someone you can tweet or email and hear back from in a few days. A mentor is someone that basically sits on retainer and does so willingly and excitedly because they like interacting with you, and feel like your success is their success.

Not only do I feel like it's important to have a professional mentor, but I also feel like the another crucial way to advance your career is to also act as a professional mentor. Though relatively early in my career (I've only been a professional for 8 years) - I've also been blessed with the opportunity to mentor a current (almost) graduate from my alma mater, Drury University. He was gracious enough to write about it and give me some major props. (OK, so I'm bragging a little, but seriously if someone wrote about you the way that Zach wrote about me, you might brag a little too.) Here's what Zach Glossip had to say about mentorship - and I think he said it more eloquently than I ever could.

I think we have a responsibility as professionals to the future generations of professionals that come after us. While we may not necessarily worry about what happens to our past employers or our last employer once we retire, as responsible global citizens we can teach other professionals how to be responsible global citizens too.  For engineers, I think this means raising other engineers to want to interact with their peers, and to want to write their code so that anyone can understand it later (hopefully by following some of the design patterns that our industry relies so heavily on).

Mentoring the future generation of engineers contains a significant number of topics, some of which I'm really not going to be the best equipped to act as the teacher on. What do you think we need to teach the engineers of the future? Software and technology are perched to take a strong social, political, and economical foothold, growing rapidly as we move into the future. How can we engage future generations to use it in the best ways we can?

“Boba Fett? Boba Fett? Where?” (Star Wars Day)

Image Credit: http://www.starwars.com/may-the-4th

Geek culture has adopted today as Star Wars Day.  For those of us who fit into geek culture, we tend to say silly things like "May the 4th Be With You" most of the day. And in general the responses we get back are good natured too.

As a fair warning, I don't really intend on answering the question in my title today. Actually, it's not really a question to answer, it was more of a rhetorical panic from Han Solo, but it seemed like it was appropriate to use as a title/question today.

Thinking about Star Wars today also got me to thinking about something else. George Lucas created an empire when he created Star Wars a few decades ago. That empire started as a vision and as an idea. The first time he showed the film to critics, they hated it. At least some of this was due to the fact that it didn't have a John Williams score or any real foley effects in it.  From anecdotes I've heard from other mega fans, apparently the first time it was screened Storm-troopers fired cap guns (based on the sounds they made and the evident lack of any "blaster bolt"). But, Mr. Lucas had a vision - and he didn't give up on it just because he ran into some opposition.

Setbacks, (or as we engineers like to call them, "bugs") always need to be treated as an opportunity to do a few things.

1. Fix the problem. This one should be obvious - if someone reports a bug, take the time to fix it.
2. Fix the process that didn't catch the problem. This one isn't quite as obvious. 
• The first step in this should probably be writing an automated test of some kind that reproduces the production problem. Then you can always be assured that new code won't cause a regression and recreate the problem that someone found in production.
• The second step in this is something I'm still debating with myself, as I don't think there's a clear right or wrong answer as to how to better prevent production bugs. More time in QA? Better code review time? Better beta testing? I don't think that I even need a right answer. The really important thing that any production bug should cause is the desire to review the development process, and ensure it's as good as it can be within its constraints.

Bugs aren't the only setbacks we face as engineers, but they sure can be some of the most annoying ones. Especially when we get paged at 3AM and told our system has crashed because someone forgot to wrap a database call in a try/catch, so connection pools exhausted rapidly... <cough>. But, as long as we continue to see these kinds of challenges as an opportunity for doing it better tomorrow, then I think the Force will be with all of us. Even on days other than May the 4th. And just maybe we'll all be lucky enough to catch a glimpse of Boba Fett.