Learning and Knowledge Retention

This is a guest post by Joe Wilding, the CTO and Co-Founder of Boom Supersonic.

✧✧✧

I was asked by someone recently: “How do you know so much about your field?”  My short answer was, “I read a lot.” To that he replied: “Yeah, but how do you retain all of that knowledge?” I didn’t have a crisp answer at the time.  But, as I have thought about that question since, I have come to realize I have developed a pattern over the years which has allowed me to retain much of the knowledge I have read.

While I have to admit this method requires additional effort, I’m personally convinced it’s required for long-term retention.

The Method

The method consists of two elements. The first is ensuring that you deeply understand the content when you first read the material. The second element is making the content sticky by refreshing your memory of it in a deliberate recurring process.

Deeply learning the topic

There are many ways to fully understand a topic when first exposed to it. If the topic is simple enough, the act of reading it, watching a video, or hearing it explained may be sufficient. For more complex topics, other tactics may be required. For me, they all come down to forming some sort of a model of the concept that makes sense to me. This model can be mental, or something that you actually sketch or turn into a diagram. Good books or other sources will do this for you, but not always.

I prefer to understand how the concept works based on the fundamental governing principles, whether that be physics, math, psychology, economics, etc. If math is involved, I do not gloss over the formulas. I pay attention to the inputs, the units, and the exponents on each variable. I try to deduce why each variable is there, and why others are not. I try to get a feel for how the answer would change based on different values of the inputs. If it is a topic i really want to understand I will enter the formula into a spreadsheet, plot it, and watch how the results change with different inputs. This “live feedback” method can increase your understanding tremendously and very quickly.

I also tend to formulate an understanding of the topic such that I can explain it to someone else. Often, I will literally do that: either out of necessity, or because I am typically surrounded by others who love to learn. It is very powerful to express a concept in your own words and to be prepared to answer questions or explain the parts that are not obvious.

Making it stick

A very unfortunate drawback of the human brain is that the knowledge it contains tends to fade over time. This is particularly true for concepts that are learned and then not accessed again before it is evolved into long-term memory. This means that all of the time and effort you put into learning a new topic could be lost if you don’t take action to make it stick.

This is less of an issue if the learned topic is something you will be using frequently for an extended period of time (such as in your daily job). However, much of what I read is a little more obscure, or something I will need only on infrequent occasions. To ensure this knowledge is not lost, I employ a method I read about many years ago called the Super-Memo Model, developed by the Polish researcher Piotr Woźniak.

The following graph shows how this works:

The graph shows that as a new topic is learned, but then not used again, the brain starts to lose that information on a decaying curve called the “curve of forgetting”. Nearly all of the knowledge on a topic can be lost in a few months. For example, try to remember something you might have heard on the news or read in a paper from a few months ago.  If you didn’t have a direct connection to it, you probably can’t.

The happy side of this story is that if you refresh your memory of the topic, not only do you quickly get back to the 100% status, the rate of decay on the curve also decreases. If you can remember to do this three or four times, the decay curve becomes very flat and the information will be accessible nearly forever.

At this point you might be saying: “Great, I have to read everything four times if I really am going to learn it?!” Not at all. If you fully understand the topic the first time—in the way I talk about above—the refresh effort can be very quick. You just have to have a method of making the information quickly available and developing the discipline to actually go back and review it.

My favorite method for easy future accessibility is to take notes when I am first learning the topic. I summarize the key concepts, keep the sketch or diagram if there is one, hang onto the spreadsheet, and list all the references on where the knowledge came from. After that, it is just going back and rereading it a few times in the future. You could schedule these in your calendar, but I usually just keep the notes file on my desktop and then go back and reread it from time to time until I feel like it is fully committed to long term memory. At that point, I usually file it away in a folder for future reference.

There are other methods for reviewing material, including: reading other sources on the same topic, teaching it to others, or using the knowledge on a recurring basis. It doesn’t really matter what the method is. It is just important to refresh your memory routinely until you’ve really mastered it..

I’ll admit that this method requires effort, and I certainly don’t use it for everything I read. But if I have a topic that I really want to master long term, I have found that this method works every time.

Improving your estimates

Estimating most projects is necessarily an imprecise exercise. The goal of this post is to share some tools I’ve learned to remove those sources of error. Not all of those tools will apply to every project, though, so use this more as a reminder of things to consider when estimating, rather than a strict checklist of things you must do for every project. As always, you are the expert doing the estimating, so it is up to your own best judgement.

Break things into small pieces

When estimating, error is generally reduced by dividing tasks into more and smaller pieces of work. As the tasks get smaller, several beneficial things result:

  • Smaller tasks are generally better understood, and it is easier to compare the task to one of known duration (e.g., some prior piece of work).
  • The error on a smaller task is generally smaller than the error on a small task. That is, if you’re off by 50% on an 8 hour task, you’re off by 4 hours. If you’re off by 50% on an 8 day task, you’re off by 4 days.
  • You’re more likely to forget to account for some part of work in a longer task than a shorter one.

As a general rule, it’s a good idea to break a project down into tasks of less than 2 days duration, but your project may be different. Pick a standard which makes sense for the size of project and level of accuracy you need.

Count what can be counted

When estimating a large project, it is often the case that it is made up of many similar parts. Perhaps it’s an activity which is repeated a number of times, or perhaps there’s some symmetry to the overall structure of the thing being created. Whichever way, try to figure out if there’s something you already know which is countable, and then try to work out how much time each one requires. You may even be able to time yourself doing one of those repeated items so your estimate is that much more accurate.

Establish a range

When estimating individual tasks (i.e., those which can’t be further subdivided), it is often beneficial to start out by figuring out the range of possible durations. Start by asking yourself: “If everything went perfectly, what is the shortest time I could imagine this taking?” Then, turn it around: “If everything went completely pear-shaped, what shortest duration I’d be willing to bet my life on?” This gives you a best/worse-case scenario. Now, with all the ways it could go wrong in mind, make a guess about how long you really think it will take.

Get a second opinion

It’s often helpful to get multiple people to estimate the same project, but you can lose a lot of the value in doing so if the different people influence each other prematurely. To avoid that, consider using planning poker. With this technique, each estimator comes up with their own estimate without revealing it to the others. Then, once everyone is finished, they all compare estimates.

Naturally, there are going to be some differences from one person to the next. When these are small, taking an average of all the estimates is fine. However, when the differences are large, it’s often a sign that there’s some disagreement about the scope of the project, what work is required to complete it, or the risks involved in doing so. At this point, it’s good for everyone to talk about how they arrived at their own estimates, and then do another round of private estimates. The tendency is for the numbers to converge pretty rapidly with only a few rounds.

Perform a reality check

Oftentimes, one is asked to estimate a project which is at least similar to a project one has already completed. However, when coming up with a quick estimate, it’s easy to just trust to one’s intuition about how long things will take rather than really examining specific knowledge of particular past projects to see what you can learn. Here’s a set of questions you can ask yourself to try to dredge up that knowledge:

  • The last time you did this, how long was it from when you started to when you actually moved on to another project?
  • What is the riskiest part of this project? What is the worst-case scenario for how long that might take?
  • The last time you did this, what parts took longer than expected?
  • The last time you did this, what did you forget to include in your estimate?
  • How many times have you done this before? How much “learning time” will you need this time around?
  • Do already you have all the tools you need to start? Do you already know how to use them all?

There are loads of other questions you might ask yourself along these lines, and the really good ones will be those which force you to remember why that similar project you’re thinking of was harder / took longer / was more expensive than you expected it to be.

Create an estimation checklist

If you are planning to do a lot of estimating, it can be immensely helpful to cultivate an estimation checklist. This is a list of all the “parts” of the projects you’ve done before. Naturally, this will vary considerably from one kind of project to the next, and not every item in the checklist will apply to every new project, but they can be immensely valuable in helping you not forget things. In my personal experience, I’ve seen more projects be late from the things which were never in the plan, than from things which took longer than expected.

✧✧✧

Estimation is super hard, and there’s really no getting around that. You’re always going to have some error bars around your estimates, and, depending upon the part of the project you’re estimating, perhaps some considerably large ones. Fortunately, a lot of people have been thinking about this for a long while, and there are a lot tricks you can use, and a lot of books on the subject you can read, if you’d like to get better. Here’s one I found particularly useful which describes a lot of what I’ve just talked about, and more:


Software Estimation: Demystifying the Black Art

Meditation for Practical Skeptics

I have been meditating on and off since I was a teenager. When I mention it to others, though, they’re often somewhat surprised since I don’t really seem like “the type”. I’m not into eastern religions, energy fields, chakras, or any other form of mysticism. In fact, I’m a very practical, secular, engineer. To me, meditation is a purely practical, secular activity which helps me gain more control over my body and mind.

It’s worth saying what I actually mean by meditation, as there are a lot of different things which fit under that heading. I specifically mean the activity of sitting quietly while keeping one’s thoughts focused on a single idea. Most often, this is just following the rhythm of one’s own breathing, but lots of things could be a good focus.

This is a lot harder than it sounds, though, as you will find a constant stream of distractions trying to draw your attention away. This could be an itch, being a bit hungry, thinking about what you’re going to do next, or worrying about something on your mind. It might even be just some weird random thoughts which keep popping into your head. Whatever it is, it’s surprisingly difficult to sit for any length of time without thinking about anything in particular.

In fact, the distractions are an expected (and even essential) part of meditation. The essential skill is to be able to retain some part of your mind which isn’t distracted, and use that to notice that you’ve become distracted. Then, stop briefly to merely notice the distraction, and allow your attention to glide back to the focus of your meditation. With practice, this gentle re-direction of thought becomes very fast and natural. Once you get really proficient at it, it becomes possible to keep your focus for longer and longer between distractions.

This ability to focus is, for me, one of the two primary benefits of meditating. The more I practice this skill, the more I can quickly and gently turn my attention away from something I don’t want to think about at that moment, and back to something I do want to think about.

The most common time this comes up is when I’m trying to get some work done. This skill allows me to redirect my attention back to my task at hand whenever I get distracted. This allows me to more readily slip into that “flow” state where complex work becomes easy (e.g., writing some difficult bit of code, or a blog post). Or, if its a day where my brain seems all over the place, it at least gives me a tool to try to corral it back where it belongs.

The second time this ability to redirect my thoughts becomes most useful, though, is when I’m feeling stressed about something. My stress often takes the form of endlessly circling around a difficult issue which is outside of my control. I will keep coming back to the issue and imagining worse and worse outcomes until I’m at the breaking point of anxiety and tension. I find that the skills built by meditation allow me to acknowledge the stressful thought, and dismiss it by replacing it with a more productive one.

The second primary benefit I get from meditating is the mental reset it offers. Each morning, just after I’ve finished washing up and getting dressed for the day, I sit down for ten minutes and meditate. For me, this creates a sense of absolute calm: mentally and emotionally. In fact, I’d say it goes further than that: it creates an anchor. No matter how much my day gets stormy and tries to toss me about, meditating helps me carry over some of that sense of calm, cheerfulness, and rationality underneath.

✧✧✧

If you’re interested in trying meditation (and I highly recommend it), I recommend downloading the Headspace app. It comes with a free set of guided, 10-minute, meditations lead by an expert. The meditations are completely free of any mystical elements, woozy music, or gimmicks. Just a gentle voice walking you through the exercise. You can stick with the free material as long as you want, or sign up for a subscription (I have) to get access to other exercises. I’ve enjoyed it quite a lot, and I’ve gotten a lot of benefit from it.

Being Approachable: Even as an Engineer

I work with a lot of engineers, and we aren’t always the most approachable bunch.  Between the abstruse jargon of our fields, the deep concentration our jobs require, and the occasional individual who lacks practice socializing (ehem), we can be rather prickly.  I freely admit, as a younger person, I was particularly unapproachable: even compared with other engineers.  However, over time, I’ve learned a lot.  In fact, someone even called me “outgoing” a few days ago.  Given my history, I found it so odd an experience that I started reflecting on what’s changed.

Body Language

The first thing which comes to mind, and probably the simplest, is a conscious use of body language.  When someone approaches me at work, I have learned to deliberately take my hands away from the computer, and square up my shoulders to face the person.  This has the effect of turning me completely away from my computer, and turning my eyes and face directly at the person talking to me.  This unambiguously shows my visitor that they have my complete attention.

The second piece of welcoming body language I’ve learned is to smile and give a warm greeting.  Perhaps its a little obvious, but enough other people I work with don’t do this that it clearly isn’t.  By smiling, I not only make myself seem more approachable and cheerful, but it actually does make me more cheerful.  Even if I’ve been dealing with something stressful or frustrating when the visitor approaches me, smiling almost forces a reset of my mental state so that I’m ready to be welcoming and helpful.

The last bit of body language I’ve changed is to adjust my position so that we share a comfortable distance and eye-line.  If my visitor is standing, I’ll stand.  If my visitor takes a seat, I’ll stay seated.  Either way, I’ll shift so that we have approximately the same posture.  Of course, I don’t pop out of my seat… that would just be weird.  But if a conversation seems likely to be extended, I’ll make both of us more comfortable by matching up with the posture of my visitor.

Active Listening

Another skill I’ve been working on for a long while is active listening.  Briefly, this is when you try to quiet your own thoughts when listening to another person, and deeply absorb what they’re saying.  Then, when they’re finished, acknowledge what they’ve said before adding in your own ideas.  This especially applies when brainstorming or debating with another person.

I think we all find it much, much more natural to be half-way listening to what someone else is saying, and half-way preparing our own response in our head.  However, resisting the urge to do this makes it much more likely that I’ll actually hear and understand what the other person has to say, and that, in turn, will make it much more possible for me to have a sincere and honest conversation with that person.  Over time, as people come to expect this from me, they feel much more comfortable approaching me with whatever question, request, or even disagreements.

Offering to Help

Another major change in my own behavior is that I’ve come to persistently offer to help other people I’m working with.  Even when my time or ability to help is very limited, I try to offer what I can.  Occasionally, this will lead me to over-commit myself (although using GTD helps a lot with knowing my limitations), but often the person either doesn’t need my help (but is appreciative of the offer), or only needs something fairly simple.  Either way, when people come to trust that I’ll always be ready to offer what help I can, they come to feel much more comfortable approaching me with a question, or request.  Even when I can’t always offer much help, they know I’ll freely offer what I can.

Being Optimistic & Action-Oriented

When I think back on all the people I’ve most enjoyed working with, this, above all, is the trait they all shared. By this, I don’t mean being some kind of ridiculous Pollyanna. Instead, think this attitude is contained in the question: “So, what are our options?” When faced with some kind of difficulty, whether its some difficult technical problem, or a difficult people problem, this question re-orients the discussion away from the uncertainty, stress, anxiety, guilt, and/or sadness the situation creates, and toward a positive future where things are better. Also, by asking for options, you start a positive process of working together, instead of attempting to assign blame for the difficulty.

✧✧✧

I expect in writing this, I’ve left out a whole bunch of things I’ve learned to do better over the years. I’m equally certain that I’ve left out a bunch of things I haven’t figured out yet. Got some of your own? Leave them in the comments below.

Being Helpful While Still Saying “No”

A while back, I was a team lead at Amazon.  At the time, Amazon was organized into lots of small teams, and each lead was expected to run their team pretty autonomously.  When a team wanted to tackle a project which required the help of some other team, the lead of the one team would generally go talk to the lead of the other team and ask for their help.

This all seems very sensible on the surface, but it frequently happened that certain teams who were at the cross-roads of a lot of systems would be inundated, and, in frustration, would flat-out reject requests.  Naturally, this aggravated the people making the requests, and it turned into an unpleasant spiral.

I had the good fortune not to be running one of those teams, but my team certainly got plenty of requests which weren’t feasible for us to accept.  However, instead of flatly refusing requests, I employed a technique taught to me by one of my mentors at Amazon:

Instead of refusing what you can’t do, offer what you can do.

In one particular incident, I recall the lead of another team calling a meeting with me and a few of his most senior engineers.  They had a scheme in mind which I knew my team wouldn’t have time to handle.  However, taking this advice to heart, I heard him out.  In the end, I told him that my team definitely didn’t have to do what he was asking, but I offered instead to help him brainstorm other options.  At the end of our meeting, we’d managed to work out a different approach which was less work for his team, and none at all for mine.  By offering up a little over an hour of my own time, the lead of the other team walked away happy and thinking well of me despite my refusal of his initial request.

✧✧✧

Of course, this advice has served me well in many aspects of my life since then.  Whether it’s a friend needing a favor, or my spouse wanting help with a project, or anything else, the key insight I’ve taken away is to look for ways to turn a “no” into a “yes”, even if its not the “yes” the person was looking for.

There are two extremely helpful side effects I’ve noticed with this approach.  The first is that it’s a lot easier to offer an alternate “yes” than to say “no”: especially to someone you like.  Using this approach helps me protect my own time instead of allowing my feelings of compassion or guilt trick me into over-committing and regretting it later on.  The second is that it helps me clearly communicate that I do want to help the person, even if I can’t do the exact thing they’re asking.  So, rather than stressing a relationship by a flat refusal (or even more so by a failure to deliver later on), you strengthen it (or at least break even) by showing your eagerness not to refuse them: at least in some capacity.

Spectrums of Sexuality

A little while ago, I came out as bisexual.  The process of coming to grips with my own sexuality took a lot of thinking on my part, and I did a lot of reading on the subject as well as watching YouTube videos of other people who had already come out.  Early on in my research, I ran across the notion that sexuality is a spectrum from heterosexual to homosexual (with bisexual being in the middle).  

This idea of a spectrum has been studied at some depth already.  A sex researcher named Kinsey established a scale from 0–6 to rate people on a spectrum of exclusively heterosexual (0) to exclusively homosexual (6), with bisexual people in between.  A later researcher named Klein expanded upon Kinsey’s work with a much more complex system, but one which still focuses on the straight/gay spectrum (although with a great deal more richness).

It strikes me though, that this is hardly the only axis of sexual expression. While various people have suggested this already (Klein, in particular), I haven’t yet run across any research on the subject.  On top of that, as I’ve been thinking and reading, I’ve come to prefer thinking about these various axes less as spectrums between two opposing points, but rather as a level of interest in a certain kind of activity.  With that approach, the following ranges stand out as being more-or-less independent of one another:

  • sexual interest in a different gender
  • sexual interest in the same gender
  • romantic interest in a different gender
  • romantic interest in the same gender
  • desire to be the active partner during sex
  • desire to be the passive partner during sex
  • interest in performing various acts on a partner
  • interest in receiving various acts from a partner
  • desired frequency of sex
  • desired length of a sexual encounter
  • strength of preference for monogamous relationships

As long a list as this is, it doesn’t even touch on gender expression, and even then I’m sure there are other axes which haven’t even occurred to me (feel free to comment below if you can think of others).  My point, really, is that one can think of most aspects of sexuality as a matter of degree: not a matter of either-or.

I arrived at this concept by observing and reading about lots of different individuals and then working backwards.  So, for the purposes of this essay, I think it would be helpful to work forwards again by considering some specific examples.

One might imagine the worst stereotype of a straight 20 year old male, for example.  This hypothetical individual has high sexual interest in the opposite gender, no sexual interest in the same gender, low romantic interest in anyone (i.e., a desire to form a permanent bond with another person), high desire to be the active partner during sex, low interest in being the passive partner (and then only for certain acts), and a desire for frequent, but relatively short sexual encounters with many partners.

On the other hand, one could also imagine a married lesbian woman in her 40’s.  She may have low (but not zero) sexual interest in men, high sexual interest in women, high romantic interest in women (and none for men), a strong desire to be the passive partner during sex, and a preference for extremely extended (but infrequent) sexual encounters only with her spouse.

Or, just to illustrate that these needn’t be either-or, consider a bisexual woman in her 20’s.  She may have moderate sexual interest in both the same and different genders (though slightly more for men), moderate romantic interest in either, alternating desire to be the active or passive partner (depending upon the partner), and a desire for frequent sexual encounters of moderate duration.

One could, naturally, multiply these examples endlessly (e.g., gay men, swingers, asexuals, …).  And, for any given example, one could conceive of a person who is the same in all the axes: except the one where they differ completely.  Where do you fit on these various ranges?  If you’re with someone, where does your partner?

✧✧✧

There is, of course, a huge variety of permutations when thinking about sexuality as a set of orthogonal ranges.  In fact, I think that’s how real people actually experience their own sexualities.  Moreover, I think taking this perspective helps eliminate the excessively nit-picky labeling which has become rampant without denying the truth of any one individual’s experiences.  With so many possible “labels” to stuff people in, the entire exercise becomes futile.  In the end, we’re all unique in our likes and dislikes, so each person should be judged on their own merits: not which end of what scale they might fall on.

Being the “second” parent

About 16 years ago, when my wife, Rachel, and I were contemplating having our first child, we talked a lot about how we would share the responsibilities for parenting.  In the end, she decided that she’d like to leave full-time hospital nursing to be able to spend full-time with our future child.  A few years after he was born, he was diagnosed as having high-functioning autism.  Needless to say, the job of parenting went from pleasantly challenging to downright daunting.

At the same time, I was building my career as a software engineer / manager.  It was (and still is) very demanding work which often requires a fair bit of overtime, and leaves me with very little time for anything else: even being a father.

So, the dynamic in our family is that my wife has taken about 75% of the burden of parenting, while I’ve taken on the rest.  This includes a lot of things: physical care (esp. during younger years), playing / supervising play, driving to appointments & activities, food preparation (and actual feeding, at least early on).  To be fair to myself, I’ve always shouldered my share of those tasks when I’m home.  But, the reality has always been that I’ve worked full-time out of the house, and I simply haven’t been physically present to do those things nearly so often as Rachel has.

Which leads me to my point.  In our family, I’m the de facto “second” parent.

Being the “second” parent is not remotely the same thing as being an absentee, uninvolved, or disinterested parent.  It also does not mean that I don’t carry my own weight in household chores or coping with parenting challenges.  But there are a few important differences.

First, it is completely true most routine decisions are made without my direct input.  I’ve always felt welcome to comment on them or suggest improvements, but my wife nearly always handles these out without any need for me to weigh in.

Second, my wife and I routinely set time aside (our date night) to talk over those decisions which aren’t routine.  Whether this is making some choice about schooling, correcting some nascent behavior problem, helping our son get started with a new activity, or altering some routine around the house, we always have a time and manner to make sure we address them together.  Often, these are situations where my wife will raise the issue, and I’ll give my feedback: leaving the final call in her court.  Most of the time, though, we come to a consensus on the spot.

Third, I have to be more deliberate about spending time with my son.  Since my calendar is often pretty full, it’s frequently the odd moments here and there which I need to capitalize on.  As one would expect, the exact form this takes has changed over the years.  Recently, he has begun to share my interest in fantasy & science fiction, movies, video games, and a number of other things.  So, recently, these odd moments have taken the form of late-night conversations which range all over, but most often start with one of those shared interests.

Finally, I’m often the member of the tag-team who is “tagged in” when things get intense.  Being on the autism spectrum, our son’s tantrums have always been intense, and they still can be.  By the point when my wife is getting frustrated and starting to lose patience, I’ll jump in and take over the conversation.  Both because of my particular personality, and especially because I’m “tagging in” fresh, I’m able to remain calm and detached from his intense emotions.  In these cases, I sit with him and help him calm down by gently refusing to react to his intense emotions (sometimes over the course of an hour or more), and by helping him talk though the issue.  These conversations often get very philosophical and talk about ethics, character, and principles for addressing the situation at hand.

✧✧✧

It’s easy to dismiss the “second” parent.  In fact, the disconnected, bread-winning father figure, by now, is beyond clichéd.  It’s also not how things have to be.  Whether the “second” parent is the mother or father—and I know at least one family where dad stays home while mom works outside the house—both can be full partners as “the parents”.  It takes some deliberate choices of who’s going to step forward, and how often, and for what things, but that doesn’t mean it isn’t an equal partnership in keeping a home and raising a family.  So, all of this is really just to say: while the roles are different, there really isn’t any “second” parent.

My favorite books on software engineering

I started to teach myself programming when I was 8 years old.  It was a brand new Macintosh Plus, and I was totally hooked on Hypercard.  I mostly learned by tearing apart the examples it came with, and figuring things out myself.  And, while this was fun, it really didn’t teach me a lot about software engineering.

I personally look at the difference between programming and software engineering as the difference between building a dog house and building a real house.

You can learn how to build a dog house on your own: completely self-taught.  You can build a dog house over a weekend or two of dedicated effort.  You can build a dog house on your own.  The consequences of completely screwing up your dog house are pretty minimal.  All that said, it’s not actually easy to make a super nice dog house, so there’s some real effort involved if you’re going to do it well, and there are some spectacular dog houses out there.

However… building a real house is an entirely different kind of endeavor.  To even start, you’re going to need to seek out some professional education.  You certainly will need a lot more than a few weekends.  You’re also going to need a team of specialists to help.  And, of course, the consequences of screwing up a real house will be very expensive: if not actually dangerous.

So, when I went off to school to learn how to go from being an avid programmer to a professional software engineer, I had a lot to learn.

I may talk about my schooling at some future point, but it was actually the reading I did on my own, at the recommendation of some of my early mentors, that taught me the most.  So, I want to pay it forward, and recommend a couple of those books I found most formative.

Code Complete, by Steve McConnell is where I was first introduced to most of the essential concepts of large-scale software engineering.  As each concept was introduced, it was accompanied with lots of examples in code, and excellent, in-depth explanations.  It is no exaggeration that this book completely changed how I wrote code. If you only ever read one book on coding, make it this book.

Design Patterns, by Erich Gamma et. al. is the foundational book for the entire concept of design patterns.  In short, straight-forward chapters, it teaches some of the most fundamental and re-usable building blocks of complex software systems.  The patterns described in this book have become the very language of software design.

Refactoring, by Martin Fowler outlines the most essential skills in changing existing code. Whether it’s to add a new feature, fix a bug, or anything else you’ll want to learn and follow the procedures outlined in this book.

✧✧✧

To be fair, these are three among a great number of amazing books on software engineering, so the fact that I don’t list a particular book here doesn’t mean it’s not valuable.  There are a lot of other books I have on my shelf and have read cover-to-cover (some of them multiple times).  However, of all the books in my collection, I view these three as having been most personally influential on the way I write and think about code.

Git 201: Creating a “fixup” commit

This post is part of a Git 201 series on keeping your commit history clean.  The series assumes some prior knowledge of Git, so you may want to start here if you’re new to git.

✧✧✧

As I work on a branch, adding commit after commit, I’ll sometimes find that there’s a piece of work I forgot to do which really should have been part of some prior commit.  The first thing to do—before fixing anything—is to save the work I’m doing.  This might mean using a work-in-progress commit as described previously, or simply using the stash.  From there, I have a few options, but in this post I’m going to focus on using the “fixup” command within git’s interactive rebase tool.

The essence of the process is to make a new commit, and then use git to combine it with the commit I want to fix.  The first step is to make my fix, and commit it.  The message here really doesn’t matter since it’s going to get replaced anyway.

> git add -A
> git commit -m "fix errors"

This will add a new commit to the end of the branch, but that’s not actually what I wanted.  Now I need to squash it into the older commit I want to fix.  To do this, I’m going to use an “interactive rebase” command:

> git rebase -i master

This is telling git that I want to edit the history of commits back to where my branch diverged from master (if you originally created your branch from somewhere else, you’ll want to specify that instead).  In response to this request, git is going to create a temporary file on disk somewhere and open up my editor (the same one used for commit messages) with that file loaded.  It will wind up looking something like this:

pick 7e70c43 Add Chicken Tikka Masala
pick e8cc090 Remove low-carb flag from BBQ ribs
pick 9ade3d6 Fix spelling error in BBQ ribs
pick b857991 fix errors

# Rebase 1222f97..b857991 onto 1222f97 (       5 TODO item(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
#
# These lines can be re-ordered; they are executed from top to
# bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

All of these commits are those which I’ve made since I cut my branch:  ordered from oldest (on the top) to the newest (on the bottom).  The commented out parts give the instructions for what you can do in this “interactive” portion of the rebase.  Assuming that the fix is for the Chicken Tikka Masala recipe, I’d want to edit the file to look like this:

pick 7e70c43 Add Chicken Tikka Masala
fixup b857991 fix errors
pick e8cc090 Remove low-carb flag from BBQ ribs
pick 9ade3d6 Fix spelling error in BBQ ribs

When I save the file and quit my editor, git is going to rebuild the branch from scratch according to these instructions.  The first line tells git to simply keep commit 7e70c43 as-is.  The next line tells git to remove the prior commit, and replace it with one which is the combination of the prior commit and my fix-up commit, b857991.  The other two commands tell git to create two new commits which result in the same end state as each of the old commits, e8cc090 and 9ad3d6.

As a bit of an aside…  Why does git have to create new commits for the last two commands?  Remember that commits are immutable once created, and that part of the data which makes up the commit is the parent commit it was created from.  Since I’ve asked git to replace the parent commit, it will now need to create new commits for everything which follows on that same branch since each one now has to have a new parent: all the way back to the commit I replaced.

At the end of all this, if we were to inspect the log, we’d see:

> git log --oneline master..

6a829bc3 Add Chicken Tikka Masala
29dd3231 Remove low-carb flag from BBQ ribs
0efc5692 Fix spelling error in BBQ ribs

In essence, everything appears to be the same, except that the original commit includes my fix.  However, looking a little closer, you can see that each commit has a different hash.  The first one is different because I modified the “diff” portion of the commit (i.e., I added in my fix) and therefore a new commit was needed (commits are immutable, so adding the fix required making a new one).  The other two needed to be re-created because their parent commit disappeared, and therefore new commits were needed to preserve the effect of those two commits, starting from my fixed-up commit as the new parent.

✧✧✧

There is one caveat I have to warn you about when using this method.  Any time you rebase a branch, you’ve changed the commit history of the branch.  That is, you’ve thrown out a whole bunch of commits and replaced them with a completely new set.  This is going to mean loads of difficulties and possibly lost work if you’re sharing that same branch with other people, so only use this technique on your own private working branches!

Git 201: Keeping a clean commit history

This series is going to get very technical.  If you’re not already familiar with git, you may want to start over here instead.

✧✧✧

Most people I’ve worked with treat git somewhat like they might treat an armed bomb with a faulty count-down timer.  They generally want to stay away from it, and if forced to interact with it, they only do exactly what some expert told them to do.  What I hope to accomplish here is to convince you that git is really more like an editor for the history of your code, and not a dark god requiring specific incantations to keep it from eating your code.

As I’m thinking of this as a Git 201 series, I’m going to assume you are familiar with all bunch of 101-level terms and concepts: repository, index, commit, branch, merge, remote, origin, stash.  If you aren’t follow this link to some 101-level content.  Come on back when you’re comfortable with all that.  I’m also going to assume you’ve already read my last post about creating a solid commit.

As you might expect, there’s a lot to be said on this topic, so I’m going to break this up into multiple commits… err posts.  To make things easy, I’m going to keep updating the list at the end of this post as I add new ones in the series.  So, feel free to jump around between each of them, or just read them straight through (each will link to the next post, and back here).

✧✧✧