Jekyll2023-12-05T11:27:21+00:00https://www.gregbeech.com/feed.xmlGreg BeechBlog and CV of Greg Beech
Greg Beechgreg@gregbeech.comNo more roadmaps2023-12-01T00:00:00+00:002023-12-01T00:00:00+00:00https://www.gregbeech.com/2023/12/01/no-more-roadmaps<p>Product managers spend aeons creating and rearranging them. Delivery managers obsess over their dates and dependencies. People in “the business” plan financials, campaigns, and more around them. But engineers—the people tasked with actually delivering them—mostly moan about them. The engineers are closest to being right, because roadmaps are little more than detrimental fantasy.</p>
<p>We’ll talk about why, and what I suggest you do instead, but first I want to be clear this isn’t merely a hypothetical opinion piece. At the start of this year I proposed a complete ban on roadmaps at Zego and while lots of people were worried about the implications on the way they work, fatigue with the failed promises of roadmaps led to everyone agreeing to give it a try.</p>
<p>The topic of roadmaps never came up again. Nobody missed the time lost creating or maintaining them. Nobody missed the pain of trying to change what was baked into them. Nobody missed planning around dates that never materialised anyway. Roadmaps just went away, and nobody missed them.</p>
<p>So what happens when roadmaps just go away? What do you do instead?</p>
<p>The biggest concern most organisations will have about ditching roadmaps is how they’ll plan when there are no dates, for example how can a new marketing campaign be planned if the marketing team don’t know the projected launch date. How can financial projections be made if key rollout dates aren’t known? The solution, in the immortal words of Maximus Decimus Meridius, is “we’ve got a better chance of survival if we work together”.</p>
<p>That’s it. That’s the answer. You work together. It really is that simple.</p>
<p>When planning and running projects, don’t have only engineering involved. Involve all the people and teams across the company you’ll actually need for a successful launch. Instead of keeping them at arms length and just updating them on the launch date, get them involved in planning and catch-ups, work with them on requirements, and see what solutions they come up with. Make them part of the project. When you’re working on it, they’re working on it, and you go live when <em>all</em> teams are in a state to do so, not when engineering decides <em>they</em> are ready.</p>
<p>Okay, sure, that’s the shorter-term stuff. But what about the longer term planning? What about things that would normally be planned months in advance?</p>
<p>Ignore them.</p>
<p>If you’re not actively working on a project right now, then don’t bother planning things around its launch because the date won’t be when you think, and it might not even happen. Business priorities shift—particularly once you’ve got rid of roadmaps so they’re <em>allowed</em> to shift—and what seems important now might not seem so relevant in a couple of months time. Anything you do in anticipation has the potential to be wasted effort, so don’t do it. Focus on things that are actually happening soon.</p>
<p>How soon is soon? I’d suggest about six weeks is the sweet spot. I don’t really have any good rationale for this other than six weeks <em>feels</em> like a good sort of timeframe to have an idea of what you’re doing. It’s long enough to deliver some pretty serious value without having to shoddily hack a temporary solution together, but short enough that you won’t be too swayed by the sunk cost fallacy if you realise it’s not working out or you want to go in a different direction.</p>
<p>Sometimes, though, you do need to plan longer term. You’re always going to have a small number of projects that need to hit a particular date, for example launches that have a significant impact on a financial model, or improvements to meet regulatory requirements, or upgrades to meet a deprecation date. This doesn’t necessitate you throw the baby out with the bathwater; you can treat these as special cases where you work backwards from the dates and ring-fence people to work on them, while taking the shorter-term planning approach with everything else, and keep much of your agility.</p>
<p>Agility. That’s the huge benefit you get when you ditch roadmaps. You start to get some business agility back because you’ve no longer got the next six-to-twelve months of deliverables baked into a Gantt chart, but instead you can constantly reevaluate what’s most important to the business and whether your plan for the next few weeks is delivering the most value, in the most efficient way, and change it if not.</p>
<p>I know that’s probably not the definition of agility you’re used to, but that’s because I’m talking about <em>real</em> agility, not the <em>Capital-A</em> Agility from the Deloitte playbook. I’ve got a lot more to say about agility, but it’ll have to wait for future posts because we need to get back on track denigrating roadmaps.</p>
<p>Anyway, to get the benefits of agility, it’s essential these six-ish week periods where you’ve got plans aren’t treated as iterations where you do six weeks and then plan the next six weeks. Instead, you need to treat it as a sliding window where the next six weeks from now is always being evaluated, current projects are being assessed to see if they’re on track and delivering the right value, and ideas are moving into and out of scope. This should be a continuous process, where decisions are made daily or—at most—weekly about changes to the current plans.</p>
<p>You’ll find you change plans a lot more than you thought you would, because you’re always assessing them against the value they can deliver.</p>
<p>That essential project you thought would take eight people two months, but now it’s better understood you realise it’s going to take twelve people and another three months, does it still deliver sufficient value for the new effort? Are there tactical solutions you could take instead which might not be as satisfying from a technical or product perspective, but which deliver sufficient value sooner for much less effort? If you’re evaluating what you’re working on constantly you’ll be much more likely to ask these questions—and act on the answers—than if you’re just shuffling out end dates on a fixed roadmap.</p>
<p>Those little features and fixes that never got onto the big static roadmaps because they were too small or insignificant, can you now get them done when you see they could easily be shipped in a couple of days and add incremental value, or ease the frustrations of customers?</p>
<p>Those unplanned events like realising your database needs attention because you’re maxing out your IOPS at peak, or a security report which you need to spend a few days investigating and patching, or the engineer with critical knowledge on a project who called in sick and will be out for a fortnight. These change your immediate ranking of what has value and what can be delivered, so your plans change to accommodate them, and it doesn’t phase you because you’re doing that all the time anyway.</p>
<p>You’re no longer entrenched in the routine of delivering something because the roadmap says that’s what you have to deliver. You’re always looking at what can deliver the most value in the best way.</p>
<p>A lot of companies go through the phase of thinking they need to introduce OKRs because they realise they’re busy all the time but nothing that really matters seems to have changed much over the last year or so. I’ve been through that myself. I’ve introduced OKRs myself. But it doesn’t work if you still have roadmaps because you’ll always be coerced back into the mindset of thinking in terms of deliverables rather than value.</p>
<p>If this sounds familiar to you—and trust me, I know this sounds <em>very</em> familiar to a lot of you—then don’t try and fix your lack of ability to deliver value with sticking plasters like OKRs. Fix the root causes instead. Like roadmaps.</p>
<p>No more roadmaps.</p>Greg Beechgreg@gregbeech.comProduct managers spend aeons creating and rearranging them. Delivery managers obsess over their dates and dependencies. People in “the business” plan financials, campaigns, and more around them. But engineers—the people tasked with actually delivering them—mostly moan about them. The engineers are closest to being right, because roadmaps are little more than detrimental fantasy.Don’t hire2023-10-12T00:00:00+00:002023-10-12T00:00:00+00:00https://www.gregbeech.com/2023/10/12/dont-hire<p>Most job specs are awful. They have a “main responsibilities” section filled with vague generic bullet points listing things you’ll never actually do, followed by an interminable disjointed list of “essential” skills and experience that will never actually be needed. Have you ever wondered why this is?</p>
<p>Spoiler: It’s because the role isn’t necessary, and the company shouldn’t be hiring for it.</p>
<p>Hiring is the worst thing you can do. I don’t mean hiring bad people, I mean hiring any people. Hiring people makes your company bigger, which means less consensus, more communication overhead, greater division of knowledge, more time with your best people mentoring or managing instead of doing, more time in meetings, and so on. Your delivery speed drops and you start tracking DORA metrics and hiring Agile Coaches or Scrum Masters or Delivery Managers to try and fix things. Which it won’t. It compounds the problem, because the root cause is that you hired people.</p>
<p>“But we’ve got more work than we can cope with.”</p>
<p>I hate to be the one to break it to you, but hiring people doesn’t make this better; it actually makes it worse. The amount of work you need to do doesn’t stay the same as you get more people, it goes up because you have more people so the <em>expectation</em> goes up. However, due to all the problems with more people you deliver sublinearly as you add them, so the amount of work you need to do appears to grow superlinearly. The solution to having too much to do is to do less, not to hire more people.</p>
<p>“But we need to grow the business.”</p>
<p>So grow it with the people you have. The great thing about not hiring people is that you need to grow it less. Let’s say, hypothetically, it’s going to cost you £120k/annum to hire a new person, and you make 3% margin on whatever it is you’re selling, which is pretty common across a range of industries. If you hire that person, you now need to grow the business by an additional £4M/annum just to cover the cost of hiring them. Unless you actually know how that person is going to contribute that £4M, and then some, don’t hire them.</p>
<p>“But I know how they’re going to contribute that £4M.”</p>
<p>Okay, great, in that case go and hire somebody.</p>
<p>No, really. Forget what I said above. If you know what the new person needs to do for you and how they’ll contribute value you otherwise wouldn’t get, go and hire them.</p>
<p>Thankfully, your job spec won’t be one of those crappy ones with a ton of vague generic responsibilities and interminable disjointed skills, because you actually know what they’re going to be doing, and you know which skills are really needed and which can be learned on the job. As a result, it’ll be much more likely attract the types of candidate you’ll want.</p>Greg Beechgreg@gregbeech.comMost job specs are awful. They have a “main responsibilities” section filled with vague generic bullet points listing things you’ll never actually do, followed by an interminable disjointed list of “essential” skills and experience that will never actually be needed. Have you ever wondered why this is?Deriving domain-driven design2021-12-03T00:00:00+00:002021-12-03T00:00:00+00:00https://www.gregbeech.com/2021/12/03/deriving-domain-driven-design<p>If you thought about it for long enough, you would invent domain-driven design. This is a talk I gave at DevLab ‘21 about deriving domain-driven design from first principles to demonstrate that it’s an intrinsic property of well architected systems.</p>
<p><strong>Watch the recording on YouTube:</strong></p>
<div><iframe width="560" height="315" src="https://www.youtube.com/embed/gd9kEH8x76s" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe></div>Greg Beechgreg@gregbeech.comIf you thought about it for long enough, you would invent domain-driven design. This is a talk I gave at DevLab ‘21 about deriving domain-driven design from first principles to demonstrate that it’s an intrinsic property of well architected systems.Strengthen your types2021-06-28T00:00:00+00:002021-06-28T00:00:00+00:00https://www.gregbeech.com/2021/06/28/strengthen-your-types<p>“Static typing” and “strong typing” are frequently conflated, as for many people a statically typed language implies that programs written with it must also be strongly typed. That’s notionally true as the variables themselves have types, not just the values, but I’d argue that most statically typed programs are actually fairly weakly typed.</p>
<p>That’s a bold claim, so I’ll spend the next few minutes justifying it.</p>
<p>To begin, we need to examine what we mean by strong and weak typing, and where better to start than JavaScript? Here are some vexing expressions taken from the infamous <a href="https://www.destroyallsoftware.com/talks/wat">wat talk</a> which produce clearly nonsensical results:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[]</span> <span class="o">+</span> <span class="p">[]</span> <span class="c1">// ""</span>
<span class="p">[]</span> <span class="o">+</span> <span class="p">{}</span> <span class="c1">// "[object Object]"</span>
<span class="p">{}</span> <span class="o">+</span> <span class="p">[]</span> <span class="c1">// 0</span>
<span class="p">{}</span> <span class="o">+</span> <span class="p">{}</span> <span class="c1">// NaN</span>
</code></pre></div></div>
<p>You can read the <a href="https://medium.com/dailyjs/the-why-behind-the-wat-an-explanation-of-javascripts-weird-type-system-83b92879a8db">detailed explanation of what’s happening</a> if you like, but it comes down to JavaScript’s surprising implicit type conversions at runtime. To demonstrate that this is a consequence of weak typing rather than dynamic typing we can contrast the results with another dynamically typed language, Python, which doesn’t coerce types at runtime. Here the results are either sensible or the runtime raises a TypeError:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[]</span> <span class="o">+</span> <span class="p">[]</span> <span class="c1"># []
</span><span class="p">[]</span> <span class="o">+</span> <span class="p">{}</span> <span class="c1"># TypeError: can only concatenate list (not "dict") to list
</span><span class="p">{}</span> <span class="o">+</span> <span class="p">[]</span> <span class="c1"># TypeError: unsupported operand type(s) for +: 'dict' and 'list'
</span><span class="p">{}</span> <span class="o">+</span> <span class="p">{}</span> <span class="c1"># TypeError: unsupported operand type(s) for +: 'dict' and 'dict'
</span></code></pre></div></div>
<p>Wikipedia says <a href="https://en.wikipedia.org/wiki/Strong_and_weak_typing">there is no precise definition of what constitutes weak or strong typing</a> but hopefully those examples are sufficient to convince you that one of its definitions is a good baseline for considering a language weakly typed:</p>
<blockquote>
<p>A weakly typed language has looser typing rules and may produce unpredictable or even erroneous results</p>
</blockquote>
<p>Assuming we can broadly agree on that definition, let’s get back to the business of showing that most statically typed programs are also weakly typed. Quite fantastically, we don’t need anything more complex than Hello World’s <code class="language-plaintext highlighter-rouge">greet</code> function to do so! Here’s what that might look like in Scala 3:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">greet</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="s">"Hello "</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="s">"!"</span>
</code></pre></div></div>
<p>This is Scala so it’s statically typed, and at first glance it appears strongly typed because the parameter type is declared to be a <code class="language-plaintext highlighter-rouge">String</code>, the return type is also declared to be a <code class="language-plaintext highlighter-rouge">String</code>, and it clearly does return a <code class="language-plaintext highlighter-rouge">String</code>. We know the <code class="language-plaintext highlighter-rouge">+</code> operator will always be <code class="language-plaintext highlighter-rouge">String</code> concatenation in this context, so the function works as you’d expect:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">greet</span><span class="o">(</span><span class="s">"Alice"</span><span class="o">)</span> <span class="c1">// "Hello Alice!"</span>
</code></pre></div></div>
<p>However, we can still call this function in ways that produce unpredictable or erroneous results. These might not be as unpredictable or erroneous as some of the JavaScript expressions — we’re not going to get <code class="language-plaintext highlighter-rouge">NaN</code>, for example— but they definitely aren’t the results we would like.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">greet</span><span class="o">(</span><span class="s">" Alice "</span><span class="o">)</span> <span class="c1">// "Hello Alice !"</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">""</span><span class="o">)</span> <span class="c1">// "Hello !"</span>
<span class="nf">greet</span><span class="o">(</span><span class="kc">null</span><span class="o">)</span> <span class="c1">// "Hello null!"</span>
</code></pre></div></div>
<p>As such, it appears we need to modify our definition of weakly typed. It’s not just whether the language itself is weakly typed, but whether the program written in the language is weakly typed.</p>
<blockquote>
<p>A weakly typed <strong>program</strong> has looser typing rules and may produce unpredictable or even erroneous results</p>
</blockquote>
<p>Here we demonstrably have a program with sufficiently loose typing rules that unpredictable or erroneous results are produced. Ergo, this program is weakly typed.</p>
<p>There are two key weaknesses in the parameter type.</p>
<p>Firstly it’s not really constraining the parameter type to be a <code class="language-plaintext highlighter-rouge">String</code> because <code class="language-plaintext highlighter-rouge">null</code> is not a <code class="language-plaintext highlighter-rouge">String</code> and passing <code class="language-plaintext highlighter-rouge">null</code> is permitted. By default in Scala 3 the <code class="language-plaintext highlighter-rouge">String</code> type actually means <code class="language-plaintext highlighter-rouge">String | Null</code>. However, by supplying the <code class="language-plaintext highlighter-rouge">-Yexplicit-nulls</code> compiler option we can strengthen the <code class="language-plaintext highlighter-rouge">String</code> type to mean exactly <code class="language-plaintext highlighter-rouge">String</code> so the final line of code won’t compile:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">greet</span><span class="o">(</span><span class="kc">null</span><span class="o">)</span> <span class="c1">// [E007] Type Mismatch Error: Found: Null, Required: String</span>
</code></pre></div></div>
<p>Secondly we’re not constraining the contents of the string to be valid, so we can pass in things like the empty string, or strings with leading or trailing whitespace. This problem could be solved without changing the types by adding branching logic to the greet function; from what I’ve seen over the last twenty years or so, this is the approach most people would use to solve it in most languages:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">greet</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span>
<span class="k">val</span> <span class="nv">trimmed</span> <span class="k">=</span> <span class="nv">name</span><span class="o">.</span><span class="py">trim</span>
<span class="k">if</span> <span class="n">trimmed</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&&</span> <span class="nv">trimmed</span><span class="o">.</span><span class="py">nonEmpty</span> <span class="n">then</span>
<span class="s">"Hello "</span> <span class="o">+</span> <span class="n">trimmed</span> <span class="o">+</span> <span class="s">"!"</span>
<span class="k">else</span>
<span class="s">""</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">"Alice"</span><span class="o">)</span> <span class="c1">// "Hello Alice!"</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">" Alice "</span><span class="o">)</span> <span class="c1">// "Hello Alice!"</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">""</span><span class="o">)</span> <span class="c1">// ""</span>
</code></pre></div></div>
<p>This looks better. Unfortunately we’ve moved the problem of unpredictable or erroneous results to the return type because the function’s type of <code class="language-plaintext highlighter-rouge">String</code> implies it will always return a greeting, but the empty string isn’t a valid greeting.</p>
<p>You might argue I’m picking nits here, but when it comes to using this function there’s nothing to indicate to the caller they may need to deal with this case so there’s a fair chance no greeting being displayed will be raised as a bug somewhere down the line, and it’ll be nontrivial to work out that it was an empty string that somehow got into the system causing it. Half a day gone.</p>
<p>We can fix this by strengthening the return type to an <code class="language-plaintext highlighter-rouge">Option[String]</code> which tells the caller that <code class="language-plaintext highlighter-rouge">greet</code> may not be able to construct a greeting if the input is invalid, i.e. it is a <a href="https://wiki.haskell.org/Partial_functions">partial function</a> rather than a total function. Now when the function is used, the caller explicitly has to handle the failure case and decide what to do. Half a day back.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">greet</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span>
<span class="k">val</span> <span class="nv">trimmed</span> <span class="k">=</span> <span class="nv">name</span><span class="o">.</span><span class="py">trim</span>
<span class="k">if</span> <span class="n">trimmed</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&&</span> <span class="nv">trimmed</span><span class="o">.</span><span class="py">nonEmpty</span> <span class="n">then</span>
<span class="nc">Some</span><span class="o">(</span><span class="s">"Hello "</span> <span class="o">+</span> <span class="n">trimmed</span> <span class="o">+</span> <span class="s">"!"</span><span class="o">)</span>
<span class="k">else</span>
<span class="nc">None</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">"Alice"</span><span class="o">)</span> <span class="c1">// Some("Hello Alice!")</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">" Alice "</span><span class="o">)</span> <span class="c1">// Some("Hello Alice!")</span>
<span class="nf">greet</span><span class="o">(</span><span class="s">""</span><span class="o">)</span> <span class="c1">// None</span>
</code></pre></div></div>
<p>This function now arguably meets our definition of strongly typed, because whatever you pass into it, the result is predictable and it is hard to subsequently make erroneous use of it. So, we’re done, right?</p>
<p>No. Unfortunately not.</p>
<p>This function might now be hard to misuse, but to make it that way we’ve broken the single responsibility principle. It has the dual responsibilities of understanding what makes a name valid, and also formatting a greeting. This might not seem like too much of a problem, but over time it means other functions dealing with names will have to reimplement the same logic, might do it slightly differently, and in a long-lived codebase you’ll end up with numerous different implementations of the same basic concept and won’t know which is correct (if any).</p>
<p>The reason it’s having to break the single responsibility principle is because the types are <em>still</em> too weak. So let’s strengthen them again and instead of name being a <code class="language-plaintext highlighter-rouge">String</code> introduce a proper <code class="language-plaintext highlighter-rouge">FirstName</code> type and move the validation into its smart constructor <code class="language-plaintext highlighter-rouge">fromString</code> to ensure that only valid instances can be constructed; this approach is sometimes called “making illegal states unrepresentable”:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">types</span><span class="k">:</span>
<span class="kt">opaque</span> <span class="k">type</span> <span class="kt">FirstName</span> <span class="o">=</span> <span class="nc">String</span>
<span class="k">object</span> <span class="nc">FirstName</span><span class="k">:</span>
<span class="kt">def</span> <span class="kt">fromString</span><span class="o">(</span><span class="kt">name:</span> <span class="kt">String</span><span class="o">)</span><span class="kt">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">FirstName</span><span class="o">]</span> <span class="k">=</span>
<span class="k">val</span> <span class="nv">trimmed</span> <span class="k">=</span> <span class="nv">name</span><span class="o">.</span><span class="py">trim</span>
<span class="k">if</span> <span class="n">trimmed</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&&</span> <span class="nv">trimmed</span><span class="o">.</span><span class="py">nonEmpty</span> <span class="n">then</span>
<span class="nc">Some</span><span class="o">(</span><span class="n">trimmed</span><span class="o">)</span>
<span class="k">else</span>
<span class="nc">None</span>
<span class="k">import</span> <span class="nn">types.</span><span class="o">*</span>
<span class="k">def</span> <span class="nf">greet</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">FirstName</span><span class="o">)</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="s">"Hello "</span> <span class="o">+</span> <span class="nv">name</span><span class="o">.</span><span class="py">toString</span> <span class="o">+</span> <span class="s">"!"</span>
<span class="nv">FirstName</span><span class="o">.</span><span class="py">fromString</span><span class="o">(</span><span class="s">"Alice"</span><span class="o">).</span><span class="py">map</span><span class="o">(</span><span class="n">greet</span><span class="o">)</span> <span class="c1">// Some("Hello Alice!")</span>
<span class="nv">FirstName</span><span class="o">.</span><span class="py">fromString</span><span class="o">(</span><span class="s">" Alice "</span><span class="o">).</span><span class="py">map</span><span class="o">(</span><span class="n">greet</span><span class="o">)</span> <span class="c1">// Some("Hello Alice!")</span>
<span class="nv">FirstName</span><span class="o">.</span><span class="py">fromString</span><span class="o">(</span><span class="s">""</span><span class="o">).</span><span class="py">map</span><span class="o">(</span><span class="n">greet</span><span class="o">)</span> <span class="c1">// None</span>
</code></pre></div></div>
<p>The above code might benefit from a little explanation. The <code class="language-plaintext highlighter-rouge">opaque type</code> line defines <code class="language-plaintext highlighter-rouge">FirstName</code> as being a <code class="language-plaintext highlighter-rouge">String</code> but because it’s <code class="language-plaintext highlighter-rouge">opaque</code> that fact isn’t known outside the container <code class="language-plaintext highlighter-rouge">types</code>, so if you try to call <code class="language-plaintext highlighter-rouge">greet("Alice")</code> then you’ll get a compilation error that <code class="language-plaintext highlighter-rouge">FirstName</code> was expected but <code class="language-plaintext highlighter-rouge">String</code> was found. However, inside the <code class="language-plaintext highlighter-rouge">types</code> object the equivalence of <code class="language-plaintext highlighter-rouge">FirstName</code> and <code class="language-plaintext highlighter-rouge">String</code> can be used to return a <code class="language-plaintext highlighter-rouge">String</code> as a <code class="language-plaintext highlighter-rouge">FirstName</code> after validation.</p>
<p>Here <code class="language-plaintext highlighter-rouge">FirstName.fromString</code> has the single responsibility of understanding what makes a first name valid, and <code class="language-plaintext highlighter-rouge">greet</code> has the single responsibility of formatting the greeting. Note that <code class="language-plaintext highlighter-rouge">greet</code> can go back to returning a <code class="language-plaintext highlighter-rouge">String</code> rather than an <code class="language-plaintext highlighter-rouge">Option[String]</code> because the input must always be valid, so it’s a total function rather than a partial function. We could go further than this and return a <code class="language-plaintext highlighter-rouge">NonEmptyString</code> or even a <code class="language-plaintext highlighter-rouge">Greeting</code> (and we should!) but I think you get the idea by now so that is left as an exercise for the reader.</p>
<p>These improvements might not appear to be much of a benefit in this little sample of code, but when additional functions need a first name they can reuse the same type, and won’t need to implement any argument validation, or negative tests for invalid input. The functions will have less branching which makes them easier to reason about, and the code will likely have fewer bugs as a consequence.</p>
<p>There are myriad other benefits to strengthening your types. The obvious one is that it makes the code more self-documenting which is a major benefit for readability and maintainability, especially in long-lived codebases where the original authors may no longer be around, or may be busy with other things.</p>
<p>It also helps to prevent trivial mistakes in code. If you had a method expecting a last name or an email address then if the types are all <code class="language-plaintext highlighter-rouge">String</code> you can pass one where another is expected and the compiler can’t help you. When you use distinct <code class="language-plaintext highlighter-rouge">FirstName</code>, <code class="language-plaintext highlighter-rouge">LastName</code>, and <code class="language-plaintext highlighter-rouge">Email</code> types then it’s not possible to mix them up.</p>
<p>A less obvious benefit is that it helps to enforce good architectural practices. If all your business logic deals with strong types rather than strings or integers then it forces validation of input up to the boundary layers where it should be, not merely by convention, but by necessity because you cannot call lower layers of code without doing so!</p>
<p>Returning to the claim that most statically typed programs are pretty weakly typed, I don’t have proof but I’ll take a bet that majority of the code written by the majority of people reading this looks much more like the initial version of the function with its weak <code class="language-plaintext highlighter-rouge">String</code> parameter and <code class="language-plaintext highlighter-rouge">String</code> result types than the final version with its strong types. That’s fine; I’m in that majority.</p>
<p>But there’s always time to make amends.</p>
<p>Any time you find yourself needing to validate arguments, give a function multiple responsibilities, or make a function partial rather than total, consider whether you can strengthen your types to remove those problems.</p>
<hr />
<p><em><a href="https://zego.engineering/strengthen-your-types-7ed3e4cc4e9f">Originally posted on the Zego Product Engineering blog</a></em></p>Greg Beechgreg@gregbeech.com“Static typing” and “strong typing” are frequently conflated, as for many people a statically typed language implies that programs written with it must also be strongly typed. That’s notionally true as the variables themselves have types, not just the values, but I’d argue that most statically typed programs are actually fairly weakly typed.Feature flags2020-11-02T00:00:00+00:002020-11-02T00:00:00+00:00https://www.gregbeech.com/2020/11/02/feature-flags<p>Introducing feature flags helps to separate release from deployment, and allow changes to be partially rolled out on a percentage basis and/or to target groups of users. It cannot completely separate these two things as some changes such as database migrations are all or nothing. However, it can significantly reduce the risk of deployments.</p>
<p>The trouble with feature flags is that they sound so simple: They’re just an on/off switch, right? Engineers tend to underestimate how much complexity there is in implementing them well, so they’ll either spend too little time choosing an off-the-shelf system and later find out it won’t scale with the company, or spend hundreds of engineering hours building it themselves (and probably still find out it won’t scale with the company).</p>
<p>We’ve just rolled out <a href="https://launchdarkly.com/">LaunchDarkly</a> at <a href="https://www.zego.com/">Zego</a> after evaluating a number of open-source and SaaS offerings. This post contains the criteria we used to select a provider so we can be confident they’ll scale with us. It’s probably not an exhaustive list, but it’ll give you a starting point.</p>
<p>There are some table stakes requirements like multiple environments (production, staging, development, etc.), targeting groups using attributes, partial rollout of flags, archiving flags, and client libraries for all your backend and frontend languages. Pretty much every feature flagging system will support these. However, there are many nonobvious requirements that are also essential.</p>
<h2 id="must-haves">Must haves</h2>
<ul>
<li><strong>In-memory evaluation (performance, reliability)</strong> - It’s likely that requests will pass through a number of feature flag gates during execution; network calls to evaluate features increase latency as even a couple of milliseconds for each of a number of feature flags adds up, and reduces reliability as those calls may fail or be slow.</li>
<li><strong>Second-level persistent cache (reliability, startup time)</strong> - When applications start they need to load feature flags, and going to the source takes time. There must be a second-level persistent cache (e.g. Redis, DynamoDB) that applications can load initial configuration from which improves startup time, means apps can start even if the SaaS service is down, and in emergencies lets you edit settings manually.</li>
<li><strong>Rollout stickiness (sanity)</strong> - When flags have a partial rollout it’s essential that when given the same parameters the decision is the same each time, because a request may pass through a gate multiple times, or a view may make multiple requests, and the decision needs to be consistent otherwise vexing bugs will occur.</li>
<li><strong>Rapid updating of flags (incident management)</strong> - When we want to turn things off because they have gone wrong, being able to change the configuration quickly is important. Flags should be updated within, say, 30 seconds either by polling or event-driven (less is better).</li>
<li><strong>Explaining why a result was returned (debugging)</strong> - As feature flags and target groups get more complex, which they will, being able to see why a result was returned for a user is essential. This is even better if it can be done <em>before</em> the flag setting is changed (e.g. a UI that lets you see what the result would be for a given input).</li>
<li><strong>History of changed flags (compliance, debugging)</strong> - It’s useful to be able to see who changed which flags, and may be important for some security cases. Bonus points if there’s a feed of changes, ability to see changes in a time window, or publishing changes to places like Datadog. Extra bonus points if comments can be added to changes.</li>
<li><strong>Don’t send server-side flags to clients (competitive advantage)</strong> - Feature flags are used to gate new features, so broadcasting your feature flags to clients (i.e. web UI or mobile) can be used by competitors to predict what features you will launch and reduce your advantage; it must be possible to share some but not all flags with clients.</li>
<li><strong>Single-Sign On (security)</strong> - Incomplete offboarding of people who have left a company is one of the biggest security risks, so the management tools should have SSO for users to ensure they are offboarded automatically. Even if you’re not going to use it initially to save costs (SSO tends to be Enterprise Plan $$$) you will need it as you grow.</li>
</ul>
<h2 id="should-haves">Should haves</h2>
<ul>
<li><strong>Partial updates (network)</strong> - Over time it’s likely you’ll get to get to hundreds or even thousands of feature flags. If updates are loaded from a single file, as some services do, then this can cause significant network usage especially from clients or when not centralised. Ideally updates should only update the things that have changed.</li>
<li><strong>Good documentation (ease of use)</strong> - Growing companies onboard people frequently and if the documentation is bad then people won’t read it and they’ll make mistakes. It’s not essential as they can learn from examples, but it really helps.</li>
<li><strong>Target groups (ease of use)</strong> - It’s possible to create properties to represent target groups, e.g. whether they are an employee, but it can make configuration easier if it’s possible to create target groups of common properties, and be able to target them. Bonus points for being able to combine Boolean logic on conditions and groups.</li>
</ul>
<h2 id="nice-to-haves">Nice to haves</h2>
<ul>
<li><strong>Tracking of result distribution (debugging)</strong> - Being able to see a breakdown of feature flag results can make it easier to ensure that your rollout percentages and target groups are doing what you expect. This can be done manually with Datadog or similar but it’d be nice to have built-in.</li>
<li><strong>Scheduled flags (ease of use)</strong> - Turn features on or off on a schedule rather than having to do it manually. This can help when marketing or other teams want to launch at a particular date and time, especially if you’re targeting users in Australia and you don’t want to be up at 3am.</li>
<li><strong>Flag groups (ease of use)</strong> - Sometimes feature flags are related and a number of them can be used to control rollout or behaviours of larger features. Ideally flags should be able to be grouped or tagged or have some other way to organise them and find related flags.</li>
<li><strong>Open source clients (debugging, bug fixing)</strong> - The clients will inevitably have some bugs and so it’s useful for them to be open source which will allow us to fork and patch them if necessary. This isn’t essential as hopefully we won’t need to do it.</li>
</ul>
<p>The other flagging system we evaluated which met the requirements was <a href="https://split.io">Split</a> but we went with LaunchDarkly because it’s a bit nicer to use, the documentation is much clearer, and it has a number of small and even more nonobvious features which weren’t part of our evaluation criteria but which are nice to have. I’d be happy to use it though. Everything else came up a long way short.</p>Greg Beechgreg@gregbeech.comIntroducing feature flags helps to separate release from deployment, and allow changes to be partially rolled out on a percentage basis and/or to target groups of users. It cannot completely separate these two things as some changes such as database migrations are all or nothing. However, it can significantly reduce the risk of deployments.Modelling composite types2020-02-22T00:00:00+00:002020-02-22T00:00:00+00:00https://www.gregbeech.com/2020/02/22/modelling-composite-types<p>A composite type is one that can have different possible types of value, for example contact info might be either a phone number or an email address. Other common names for this—depending on programming language and context—are sum types, tagged unions, disjoint unions, discriminated unions, coproducts, or variant types (not the same as the old COM concept).</p>
<p>These are easy to model in functional languages that use algebraic type systems, because they can directly represent composite types. However, in object oriented languages there is no ideal way to model them. There is another approach which is neither algebraic nor object-oriented which might be the best of both worlds though.</p>
<p>To explore this, we’ll model a contact info type which can be either a phone number or an email, show how it’s used, and then evolve it to also add a website URL alternative.</p>
<h2 id="algebraic-modelling">Algebraic modelling</h2>
<p>For the algebraic type system I’m going to use F# as it has particularly clean syntax for domain modelling. The contact info can be modelled as a discriminated union where <code class="language-plaintext highlighter-rouge">|</code> indicates a choice between the type constructors, i.e. a <code class="language-plaintext highlighter-rouge">ContactInfo</code> is either a <code class="language-plaintext highlighter-rouge">Phone</code> containing a string, or an <code class="language-plaintext highlighter-rouge">Email</code> containing a string.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="nc">ContactInfo</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Phone</span> <span class="k">of</span> <span class="kt">string</span>
<span class="p">|</span> <span class="nc">Email</span> <span class="k">of</span> <span class="kt">string</span>
</code></pre></div></div>
<p>To use this we pattern match over the type. The following code defines a function called <code class="language-plaintext highlighter-rouge">contact</code> and extracts either the number or address string in the pattern match, then takes the appropriate action (let’s assume that <code class="language-plaintext highlighter-rouge">call</code> and <code class="language-plaintext highlighter-rouge">message</code> are functions that somehow exist and know what to do).</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">contact</span> <span class="n">contactInfo</span> <span class="p">=</span>
<span class="k">match</span> <span class="n">contactInfo</span> <span class="k">with</span>
<span class="p">|</span> <span class="nc">Phone</span> <span class="n">number</span> <span class="p">-></span> <span class="n">call</span> <span class="n">number</span>
<span class="p">|</span> <span class="nc">Email</span> <span class="n">address</span> <span class="p">-></span> <span class="n">message</span> <span class="n">address</span>
</code></pre></div></div>
<p>If you’re not familiar with F# then this code might look strange because there are no types and no braces! F# uses spaces for function application so the first line declares a function named <code class="language-plaintext highlighter-rouge">contact</code> which takes an argument named <code class="language-plaintext highlighter-rouge">contactInfo</code>. It doesn’t need any explicit types because it can infer from the pattern match branches that the argument must be of type <code class="language-plaintext highlighter-rouge">ContactInfo</code>.</p>
<p>To evolve the type and add a website, we just add another case. It doesn’t matter that the contained type is a <code class="language-plaintext highlighter-rouge">Uri</code> rather than a <code class="language-plaintext highlighter-rouge">string</code> because the type constructors <code class="language-plaintext highlighter-rouge">Phone</code>, <code class="language-plaintext highlighter-rouge">Email</code> and <code class="language-plaintext highlighter-rouge">Website</code> are independent of each other and don’t need to have any shared interface.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">open</span> <span class="nc">System</span>
<span class="k">type</span> <span class="nc">ContactInfo</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Phone</span> <span class="k">of</span> <span class="kt">string</span>
<span class="p">|</span> <span class="nc">Email</span> <span class="k">of</span> <span class="kt">string</span>
<span class="p">|</span> <span class="nc">Website</span> <span class="k">of</span> <span class="nc">Uri</span>
</code></pre></div></div>
<p>When we add this new <code class="language-plaintext highlighter-rouge">Website</code> type constructor we’ll get a compiler error as the <code class="language-plaintext highlighter-rouge">match</code> is now not exhaustive, so we need to add a case to any matches throughout the codebase. That’s just a matter of adding another line.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">contact</span> <span class="n">contactInfo</span> <span class="p">=</span>
<span class="k">match</span> <span class="n">contactInfo</span> <span class="k">with</span>
<span class="p">|</span> <span class="nc">Phone</span> <span class="n">number</span> <span class="p">-></span> <span class="n">call</span> <span class="n">number</span>
<span class="p">|</span> <span class="nc">Email</span> <span class="n">address</span> <span class="p">-></span> <span class="n">message</span> <span class="n">address</span>
<span class="p">|</span> <span class="nc">Website</span> <span class="n">url</span> <span class="p">-></span> <span class="n">browse</span> <span class="n">url</span>
</code></pre></div></div>
<p>That was pretty neat. Adding a different type didn’t require any existing lines of code to be changed, and the compiler told us everywhere that would be affected. The downside is that this type can’t be externally extended, i.e. only the author of the type can add a new variant, but that isn’t an issue for business domains which are inherently closed.</p>
<p>Now let’s see how we can model this concept in an object-oriented language.</p>
<h2 id="object-oriented-attempt-1-tagging">Object-oriented attempt 1: Tagging</h2>
<p>The approach I see most often used for this situation in object-oriented languages is to add an enum tag to the class indicating the type of value. This is probably because corresponds to the way you’d store the data in a relational database (a column for the type and a column for the value) so it’s easy to use with object-relational mappers.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">enum</span> <span class="n">ContactMethod</span> <span class="p">{</span>
<span class="n">Phone</span><span class="p">,</span>
<span class="n">Email</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">ContactInfo</span><span class="p">(</span><span class="n">ContactMethod</span> <span class="n">method</span><span class="p">,</span> <span class="kt">string</span> <span class="k">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Method</span> <span class="p">=</span> <span class="n">method</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="n">Value</span> <span class="p">=</span> <span class="k">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">ContactMethod</span> <span class="n">Method</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Value</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In use this looks broadly similar to, if somewhat more verbose than, the functional approach where the type is matched and then the value is used.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">Contact</span><span class="p">(</span><span class="n">ContactInfo</span> <span class="n">contactInfo</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">Method</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">ContactMethod</span><span class="p">.</span><span class="n">Phone</span><span class="p">:</span>
<span class="nf">Call</span><span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">Value</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">ContactMethod</span><span class="p">.</span><span class="n">Email</span><span class="p">:</span>
<span class="nf">Message</span><span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">Value</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>However when we come to evolve it to add website with a <code class="language-plaintext highlighter-rouge">Uri</code> value we have a problem because the value is defined as a <code class="language-plaintext highlighter-rouge">string</code>. Generics don’t help here because that would prevent us from doing things like putting multiple contact infos in a list if the type was different (or needing to use existential types in the list).</p>
<p>We’ll have to settle for storing the value as a <code class="language-plaintext highlighter-rouge">string</code> and then converting it to a <code class="language-plaintext highlighter-rouge">Uri</code> when it’s read with a different accessor.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">System</span>
<span class="k">public</span> <span class="k">enum</span> <span class="n">ContactMethod</span> <span class="p">{</span>
<span class="n">Phone</span><span class="p">,</span>
<span class="n">Email</span><span class="p">,</span>
<span class="n">Website</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">ContactInfo</span><span class="p">(</span><span class="n">ContactMethod</span> <span class="n">method</span><span class="p">,</span> <span class="kt">string</span> <span class="k">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Method</span> <span class="p">=</span> <span class="n">method</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="n">Value</span> <span class="p">=</span> <span class="k">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">ContactMethod</span> <span class="n">Method</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Value</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">Uri</span> <span class="n">ValueAsUri</span> <span class="p">{</span>
<span class="k">get</span> <span class="p">{</span>
<span class="c1">// Will throw an exception if Value isn't a valid URI</span>
<span class="k">return</span> <span class="nf">Uri</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">Value</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In use this feels more unpleasant as we now have to remember to call the correct accessor based on the tag. This relationship between logical type and accessor isn’t enforced by the type system so we’ve sacrificed some type safety, and the code is no longer exception-safe even though that isn’t obvious from looking at it.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">Contact</span><span class="p">(</span><span class="n">ContactInfo</span> <span class="n">contactInfo</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">Method</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">ContactMethod</span><span class="p">.</span><span class="n">Phone</span><span class="p">:</span>
<span class="nf">Call</span><span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">Value</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">ContactMethod</span><span class="p">.</span><span class="n">Email</span><span class="p">:</span>
<span class="nf">Message</span><span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">Value</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">ContactMethod</span><span class="p">.</span><span class="n">Website</span><span class="p">:</span>
<span class="nf">Browse</span><span class="p">(</span><span class="n">contactInfo</span><span class="p">.</span><span class="n">ValueAsUri</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In general, this kind of tagged data approach is not very future-proof in object-oriented languages. We could just about get away with the evolving requirements here as the data is almost the same shape, but what if the requirement was to add a phone type (e.g. home, mobile, etc.) to the phone number? There’s nowhere to store it.</p>
<p>Let’s see if we can do better.</p>
<h2 id="object-oriented-attempt-2-inheritance">Object-oriented attempt 2: Inheritance</h2>
<p>For our second attempt we’ll use the proper object-oriented approach of inheritance and subtype polymorphism. I’ve used an abstract base class here, but an interface could be used instead without changing any of the modelling discussion.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">abstract</span> <span class="k">class</span> <span class="nc">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Value</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Phone</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">Phone</span><span class="p">(</span><span class="kt">string</span> <span class="k">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Value</span> <span class="p">=</span> <span class="k">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Email</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">Email</span><span class="p">(</span><span class="kt">string</span> <span class="k">value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Value</span> <span class="p">=</span> <span class="k">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Unfortunately, the above code makes one of the most common mistakes of object-oriented modelling which is exposing the object’s data instead of its behaviour. We shouldn’t be switching on the type of the class and reading the <code class="language-plaintext highlighter-rouge">Value</code> property, but should instead call a method on it and allow the object to respond correctly based on its runtime type.</p>
<p>Let’s change the interface to hide the data and expose the desired behaviour instead. This now works nicely as we can call the <code class="language-plaintext highlighter-rouge">Contact()</code> method on any instance.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">abstract</span> <span class="k">class</span> <span class="nc">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">abstract</span> <span class="k">void</span> <span class="nf">Contact</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Phone</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="kt">string</span> <span class="n">number</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">Phone</span><span class="p">(</span><span class="kt">string</span> <span class="n">number</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">number</span> <span class="p">=</span> <span class="n">number</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Contact</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Call</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">number</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Email</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="kt">string</span> <span class="n">address</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">Email</span><span class="p">(</span><span class="kt">string</span> <span class="n">address</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">address</span> <span class="p">=</span> <span class="n">address</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Contact</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Message</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">address</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We can also cleanly add a <code class="language-plaintext highlighter-rouge">Website</code> class with a <code class="language-plaintext highlighter-rouge">Uri</code> rather than <code class="language-plaintext highlighter-rouge">string</code> value because the type of the value isn’t exposed in the interface, and much like the functional approach we are forced to implement the behaviour as it’s part of the class contract. This is how object-oriented design is supposed to be done. It’s unfortunate that many of the languages have standards (e.g. Java Beans) or features (e.g. auto-implemented properties) that encourage programmers to do the wrong thing by default.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">System</span>
<span class="c1">// other code the same as before</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Website</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Uri</span> <span class="n">url</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">Website</span><span class="p">(</span><span class="n">Uri</span> <span class="n">url</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">url</span> <span class="p">=</span> <span class="n">url</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Contact</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Browse</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">url</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>All good then? Not quite. Unfortunately, subtype polymorphism is only viable when you can actually modify the classes themselves when you need to add behaviours. It can also easily lead to very large classes that have high coupling and low cohesion as everything related to the class ends up in there (how many barely related methods do your <code class="language-plaintext highlighter-rouge">User</code> or <code class="language-plaintext highlighter-rouge">Order</code> or similar classes have, for example?).</p>
<p>Subtype polymorphism is a great approach for functionality that is intrinsic to the type, but for other things you might want to do with it (e.g. converting it to data transfer objects for rendering in APIs or UIs) another approach is necessary.</p>
<h2 id="object-oriented-attempt-3-visitor-pattern">Object-oriented attempt 3: Visitor pattern</h2>
<p>We’ll start off with subtype polymorphism again, but this time we will expose the data because the visitor pattern does dispatch based on type. Note, however, that there are no shared properties or behaviour and so <code class="language-plaintext highlighter-rouge">ContactInfo</code> becomes effectively a marker interface.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">abstract</span> <span class="k">class</span> <span class="nc">ContactInfo</span> <span class="p">{}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Phone</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">Phone</span><span class="p">(</span><span class="kt">string</span> <span class="n">number</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Number</span> <span class="p">=</span> <span class="n">number</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Number</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Email</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">Email</span><span class="p">(</span><span class="kt">string</span> <span class="n">address</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Address</span> <span class="p">=</span> <span class="n">address</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Address</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Rather than having each specific visitor implement the boilerplate for the visitor pattern, we can implement a visitor base class and then allow specific visitors override the methods that handle the concrete types. This code uses C# 7.0’s feature of aliasing variables after <code class="language-plaintext highlighter-rouge">as</code> checks so there is no need for an additional cast.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">public</span> <span class="k">abstract</span> <span class="k">class</span> <span class="nc">ContactInfoVisitor</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">ContactInfo</span> <span class="n">contactInfo</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">contactInfo</span> <span class="k">is</span> <span class="n">Phone</span> <span class="n">phone</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Visit</span><span class="p">(</span><span class="n">phone</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">contactInfo</span> <span class="k">is</span> <span class="n">Email</span> <span class="n">email</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Visit</span><span class="p">(</span><span class="n">email</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">NotSupportedException</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">protected</span> <span class="k">abstract</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Phone</span> <span class="n">phone</span><span class="p">);</span>
<span class="k">protected</span> <span class="k">abstract</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Email</span> <span class="n">email</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">ContactVisitor</span> <span class="p">:</span> <span class="n">ContactInfoVisitor</span> <span class="p">{</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Phone</span> <span class="n">phone</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Call</span><span class="p">(</span><span class="n">phone</span><span class="p">.</span><span class="n">Value</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Email</span> <span class="n">email</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Message</span><span class="p">(</span><span class="n">email</span><span class="p">.</span><span class="n">Value</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We can now evolve this to add the website. Unfortunately again we don’t get any compiler errors when we add the <code class="language-plaintext highlighter-rouge">Website</code> class saying that it isn’t handled, so we need to remember to update our visitor base class in lockstep. Fortunately with the abstract methods we will get errors in the derived visitors when we add the method to the base class, so there’s <em>some</em> safety there at least.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">System</span>
<span class="c1">// other entity classes as before</span>
<span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">Website</span> <span class="p">:</span> <span class="n">ContactInfo</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">Email</span><span class="p">(</span><span class="n">Uri</span> <span class="n">url</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">Url</span> <span class="p">=</span> <span class="n">url</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">Uri</span> <span class="n">Url</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">abstract</span> <span class="k">class</span> <span class="nc">ContactInfoVisitor</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">ContactInfo</span> <span class="n">contactInfo</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">contactInfo</span> <span class="k">is</span> <span class="n">Phone</span> <span class="n">phone</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Visit</span><span class="p">(</span><span class="n">phone</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">contactInfo</span> <span class="k">is</span> <span class="n">Email</span> <span class="n">email</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Visit</span><span class="p">(</span><span class="n">email</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">contactInfo</span> <span class="k">is</span> <span class="n">Website</span> <span class="n">website</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Visit</span><span class="p">(</span><span class="n">website</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">NotImplementedException</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">protected</span> <span class="k">abstract</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Phone</span> <span class="n">phone</span><span class="p">);</span>
<span class="k">protected</span> <span class="k">abstract</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Email</span> <span class="n">email</span><span class="p">);</span>
<span class="k">protected</span> <span class="k">abstract</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Website</span> <span class="n">website</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Finally we can use this to provide external extensibility to the class in a safe way.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">ContactVisitor</span> <span class="p">:</span> <span class="n">ContactInfoVisitor</span> <span class="p">{</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Phone</span> <span class="n">phone</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Call</span><span class="p">(</span><span class="n">phone</span><span class="p">.</span><span class="n">Number</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Email</span> <span class="n">email</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Message</span><span class="p">(</span><span class="n">email</span><span class="p">.</span><span class="n">Address</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">Visit</span><span class="p">(</span><span class="n">Website</span> <span class="n">website</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Browse</span><span class="p">(</span><span class="n">website</span><span class="p">.</span><span class="n">Url</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Look back at the visitor pattern again though. We’ve exposed disparate properties on the entity classes, dispatched on their concrete types, and handled each case individually. It should be evident that the visitor pattern as used here is just a poor facsimile of the functional language’s built-in discriminated union support, with a lot more boilerplate and a little less help from the compiler.</p>
<p>Unfortunately with traditional object-oriented languages we haven’t found an ideal approach for modelling composite types.</p>
<h2 id="ad-hoc-interface-implementation"><em>Ad hoc</em> interface implementation</h2>
<p>This brings us neatly around to another form of modelling which is neither algebraic nor object-oriented. It’s the approach used in Go’s interfaces and Rust’s trait objects, as shown below. Note that in Go you wouldn’t tend to declare the interface along with the structures, but only when you need to make them implement a common behaviour.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="s">"net/url"</span>
<span class="c">// these types get defined up-front</span>
<span class="k">type</span> <span class="n">Phone</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">Number</span> <span class="kt">string</span>
<span class="p">}</span>
<span class="k">type</span> <span class="n">Email</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">Address</span> <span class="kt">string</span>
<span class="p">}</span>
<span class="k">type</span> <span class="n">Website</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">URL</span> <span class="n">url</span><span class="o">.</span><span class="n">URL</span>
<span class="p">}</span>
<span class="c">// the interface and implementation can be defined later anywhere else</span>
<span class="k">type</span> <span class="n">ContactInfo</span> <span class="k">interface</span> <span class="p">{</span>
<span class="n">Contact</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">Phone</span><span class="p">)</span> <span class="n">Contact</span><span class="p">()</span> <span class="p">{</span>
<span class="n">call</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">Number</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">e</span> <span class="o">*</span><span class="n">Email</span><span class="p">)</span> <span class="n">Contact</span><span class="p">()</span> <span class="p">{</span>
<span class="n">message</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">Address</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">w</span> <span class="o">*</span><span class="n">Website</span><span class="p">)</span> <span class="n">Contact</span><span class="p">()</span> <span class="p">{</span>
<span class="n">browse</span><span class="p">(</span><span class="n">w</span><span class="o">.</span><span class="n">URL</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This <em>ad hoc</em> interface implementation for disjoint types is also supported in Python. That probably isn’t too surprising as Python can’t decide what type of language it wants to be, so it chucks a bit of every paradigm into the mix. It takes the approach of defining a ‘base’ method and registering additional methods as handlers, using metaprogramming rather than being a language intrinsic. This ‘base’ method is effectively the interface defintion.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span>
<span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">singledispatch</span>
<span class="c1"># these types get defined up-front
</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Phone</span><span class="p">:</span>
<span class="n">number</span><span class="p">:</span> <span class="nb">str</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Email</span><span class="p">:</span>
<span class="n">address</span><span class="p">:</span> <span class="nb">str</span>
<span class="o">@</span><span class="n">dataclass</span>
<span class="k">class</span> <span class="nc">Website</span><span class="p">:</span>
<span class="n">url</span><span class="p">:</span> <span class="nb">str</span> <span class="c1"># No URL type in Python :-(
</span>
<span class="c1"># the implementation can be defined later anywhere else; no interface needed
</span>
<span class="o">@</span><span class="n">singledispatch</span>
<span class="k">def</span> <span class="nf">contact</span><span class="p">(</span><span class="n">_contact_info</span><span class="p">:</span> <span class="nb">object</span><span class="p">):</span>
<span class="p">...</span>
<span class="o">@</span><span class="n">contact</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="n">Phone</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__contact_phone</span><span class="p">(</span><span class="n">phone</span><span class="p">:</span> <span class="n">Phone</span><span class="p">):</span>
<span class="n">call</span><span class="p">(</span><span class="n">phone</span><span class="p">.</span><span class="n">number</span><span class="p">)</span>
<span class="o">@</span><span class="n">contact</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="n">Email</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__contact_email</span><span class="p">(</span><span class="n">email</span><span class="p">:</span> <span class="n">Email</span><span class="p">):</span>
<span class="n">message</span><span class="p">(</span><span class="n">email</span><span class="p">.</span><span class="n">address</span><span class="p">)</span>
<span class="o">@</span><span class="n">contact</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="n">Website</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__contact_website</span><span class="p">(</span><span class="n">website</span><span class="p">:</span> <span class="n">Website</span><span class="p">):</span>
<span class="n">browse</span><span class="p">(</span><span class="n">website</span><span class="p">.</span><span class="n">url</span><span class="p">)</span>
</code></pre></div></div>
<p><em>Ad hoc</em> interface implementation, or <em>ad hoc</em> polymorphism as it’s more commonly known, is a more powerful approach than traditional object-orientation’s subtype polymorphism. (That’s right sports fans, I actually said that there’s a design decision in Go that isn’t terrible. Just one though. Don’t get excited.)</p>
<p>If you’re into functional programming then you’ll recognise this as being conceptually similar to typeclasses, but this post is already running long and I’d need to introduce yet another language that supports them to demonstrate, so I’m going to call it a day.</p>Greg Beechgreg@gregbeech.comA composite type is one that can have different possible types of value, for example contact info might be either a phone number or an email address. Other common names for this—depending on programming language and context—are sum types, tagged unions, disjoint unions, discriminated unions, coproducts, or variant types (not the same as the old COM concept).The engineering career triangle2020-01-19T00:00:00+00:002020-01-19T00:00:00+00:00https://www.gregbeech.com/2020/01/19/the-engineering-career-triangle<p>Engineering career progression is often described as a ‘ladder’ or ‘track’, where you can also switch into a parallel management track. However, this doesn’t adequately illustrate where the tech lead role sits, or why the staff+ engineer levels have at least as much in common with management as they do with coding. To be able to explain this we need to separate the concepts of management and leadership. The result is the engineering career triangle.</p>
<p><img src="/assets/img/engineering-career-triangle.svg" alt="Engineering Career Triangle" /></p>
<p>There’s quite a lot of information encoded in this diagram which I’ll explain in more depth, but the key points are that the black dots indicate a position, solid arrows show career progression within a track, and dashed arrows show a switch between engineering and management tracks.</p>
<p>Job titles vary hugely in engineering so I’ve tried to pick a set that are fairly cohesive and representative of what I’ve seen at a range of companies. It probably won’t exactly match what your company uses.</p>
<h2 id="management-vs-leadership">Management vs leadership</h2>
<p>Management and leadership are often conflated, but they’re orthogonal. Management responsibilities include performance assessment and feedback, dealing with interpersonal conflicts, and overseeing the planning and delivery of projects. Leadership is more about defining the vision for the organisation, establishing and defending culture, and inspiring people to work towards common goals.</p>
<p>Perhaps the most significant difference, however, is that management implies authority whereas leadership does not. In other words, you have to do what your manager tells you, but it’s up to you whether you follow a leader. That said, good managers are also good leaders, and rarely need to flex their authoritarian muscles.</p>
<p>The reason this distinction is important is that engineers are not managers, but as people in either track become more senior their focus shifts more onto leadership. Consequently the tracks which look quite different at lower levels start to converge, and include many of the same responsibilities at higher levels.</p>
<h2 id="levels">Levels</h2>
<p>In many engineering organisations there isn’t a defined equivalence between engineering positions and management positions. The career triangle makes it obvious by putting equivalent levels in the same horizontal band. Level defines, among other things, the compensation people receive so a senior engineer will be paid similarly to a standard-level engineering manager.</p>
<p><img src="/assets/img/engineering-career-levels.svg" alt="Engineering Career Levels" /></p>
<p>I’ve labelled them from 3 through 8 which might seem somewhat arbitrary, but is based on the approach taken by Deliveroo, which was in turn largely copied from Facebook and Google. However, Microsoft use 59 through 69 for these levels, and Apple use 2 through 6, so this is far from a universal convention even among the FAANG companies.</p>
<p>If you want to retain senior engineers, and have them stay as engineers rather than switching to management, then it’s important to have this kind of equivalence defined. Without it there’s always doubt as to whether managers are treated better (higher compensation, etc.) and whether engineers are really second-class citizens.</p>
<h2 id="responsibilities">Responsibilities</h2>
<p>We can roughly divide positions into segments based on their primary responsibility. Note that although the areas of these segments are roughly equal in the diagram, in reality that will be far from the case. In a typical organisation around 80% of people will be in the green segment, with around 10% in each of the yellow and purple ones.</p>
<p><img src="/assets/img/engineering-career-responsibilities.svg" alt="Engineering Career Responsibilities" /></p>
<p>This illustrates why the transition from senior engineer to staff engineer shouldn’t really be seen as a promotion, but more as a change of role. Focus shifts away from a single team and working on one or two products, writing production code most days, to working across multiple teams and often not writing production code for weeks at a time. It’s a transition that many engineers don’t <em>want</em> to make because what they really enjoy doing is working in a team and writing production software.</p>
<p>Working backwards from this, the rationale behind having two levels of senior engineer becomes clearer. There can a long time between mid-level (“Engineer”) and staff—perhaps ten or fifteen years—even if people actually want it, and without this distinction it can be hard to show progression or give people who have already made senior realistic goals to aim for.</p>
<p>This should also make clear that being a tech lead is not a promotion, but a different role with more management responsibility, for example organising the work team members are doing. The more senior tech lead roles are deliberately shown closer to the management line, as it’s not uncommon at this level that they will start taking on a bit more of the people management side of things.</p>
<p>As a counterpoint, the junior manager role is shown closer to the engineering line. This is because they typically only manage a small number of people while they learn the ropes, and so because they have more free time and technical skills they will often still be somewhat hands-on. As they get more experienced and manage more engineers, or other managers, being hands-on is no longer practical.</p>
<h2 id="switching-tracks">Switching tracks</h2>
<p>I’ve shown the four most common points at which people switch tracks, but it is quite possible to switch tracks at any point in your career. It gets less common as people become more senior because they are more likely to know which track they want to be on, and less likely to have the skill required for the other track, but there are some people who oscillate between the two.</p>
<p>In general if you’re moving from an engineering position to a management position, as a new manager, you’ll drop a level or possibly even two because you won’t be as skilled at management as you are at engineering. Going back the other way it’s more likely you’ll stay at the same level if you’ve maintained technical skill, but if you need to re-skill that may not be the case.</p>
<p>It’s worth noting that in many countries, the UK included, your salary cannot legally be reduced even if you change level. This makes it relatively risk-free to try switching tracks if you want to. Most good companies will support you doing this, and switching back if it doesn’t suit you. The tech lead role is a great way of dipping your toe in the water.</p>Greg Beechgreg@gregbeech.comEngineering career progression is often described as a ‘ladder’ or ‘track’, where you can also switch into a parallel management track. However, this doesn’t adequately illustrate where the tech lead role sits, or why the staff+ engineer levels have at least as much in common with management as they do with coding. To be able to explain this we need to separate the concepts of management and leadership. The result is the engineering career triangle.All about identifiers2019-12-10T00:00:00+00:002019-12-10T00:00:00+00:00https://www.gregbeech.com/2019/12/10/all-about-identifiers<p>Identifiers aren’t usually considered very interesting. Indeed, popular frameworks such as Django or Hibernate find them so uninteresting that they ship with the default of auto-incrementing integers so users don’t need to be concerned with them. Unfortunately using sequential identifiers is a bad idea, and if you don’t think about your identifiers up front then fixing things later can be very time consuming. Here’s most of what you need to know about identifiers.</p>
<h2 id="dont-use-sequential-integers">Don’t use sequential integers</h2>
<p>Before we get into anything else, let me convince you that you should not use sequential integers as your primary identifier, whether 32-bit or 64-bit. There are three main problems with sequential integers.</p>
<p>Firstly, they make life difficult if you want to generate them in multiple places such as a different service if an entity is being migrated, or in different shards, because the sequences will overlap with each other. If you’re using 64-bit integers this issue can often be mitigated by partitioning the keyspace, but this kind of partitioning can be manual and error-prone, leading to accidental overlaps. If you’re using 32-bit integers then you’re out of luck unless your rate of growth is really very small.</p>
<p>Secondly, sequential integers can impact application security because they facilitate enumeration and thus make <a href="https://www.owasp.org/index.php/Testing_for_Insecure_Direct_Object_References_(OTG-AUTHZ-004)">insecure direct object reference (IDOR)</a> attacks easier to mount, and the consequences of insufficient access control more severe. No matter how dilligent you are, mistakes will be made in access control. While this is to some extent security by obscurity, it’s still a valuable part of defence in depth.</p>
<p>Another way security can be impacted by integer identifiers is in the aftermath of database failure. Because backups lag some way behind real-time, when a database is restored from a backup the latest id will be lower than the previously highest issued id which can result in things like <code class="language-plaintext highlighter-rouge">UserId</code>s being reissued to different people. If these integers have already been used elsewhere (security tokens, caches, third parties, etc.) then the person with the post-restore <code class="language-plaintext highlighter-rouge">UserId</code> may be able to access information related to the person who held it pre-restore. This isn’t just a theoretical case; I have worked at a company where this happened.</p>
<p>Lastly, they leak business information which could be valuable to competitors, who can derive things like how many orders/day you’re doing from sequential order numbers. The workaround of partitioning the keyspace can make this leak even worse because if your partitions relate to business units or markets then competitors can obtain a more detailed breakdown. Trust me, your competitors <em>will</em> do this.</p>
<p>Instead I’d recommend using randomly generated identifiers with effectively zero probability of collision such as <a href="https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)">v4 UUIDs</a>. However, note that while most mainstram v4 UUID generation algorithms are cryptographically secure and thus the UUIDs are unguessable under any circumstances, this is not required by the specification. UUIDs are required to be <em>unique</em> but not <em>unguessable</em>. As such, if the identifiers must be unguessable then it may be safest to use a cryptographically random generator directly.</p>
<p>Another option you might want to consider is <a href="https://en.wikipedia.org/wiki/Partial_sorting">k-sortable</a> identifiers which are essentially a random string prefixed with a specially formatted timestamp so that the identifiers are roughly sortable by time both lexicographically and in binary. However, again beware that this leaks information as the timestamps could be extracted by interested third parties.</p>
<p>Before ending this section, a word on performance. These larger random identifiers work well in many data stores, but for some such as SQL Server which uses clustered primary keys they can cause major problems from effects like page splits. In this case you may want to create the table with an auto-incrementing integer primary key which is a <a href="https://en.wikipedia.org/wiki/Surrogate_key">surrogate</a> and only used <em>within</em> the application but never exposed to any other application or party. This isn’t necessary in Postgres; using a UUID rather than a 64-bit integer key has a negligible effect on performance.</p>
<h2 id="identifiers-can-be-personal-data">Identifiers can be personal data</h2>
<p>When we think of personal data we tend to think of things like name or email. However, any identifier for a person that is accessible to that person or any third party (whether it’s in a user interface, URL, API body, cookie, JWT, etc.) should also be considered personal data. On learning this many people are dismissive, but consider that it falls into the same category as identifiers like passport or driving license number.</p>
<p>If you operate in Europe and are thus required to be compliant with the GDPR then you need to be able to <em>delete</em> any accessible identifier if the person files a <a href="https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/">deletion request</a> (aka “the right to erasure” or “the right to be forgotten”).</p>
<p>In theory if you implement ‘true’ deletion where all of the data related to that person is actually deleted then there should be no problem having the same identifier for both internal and external use.</p>
<p>However, it is common practice particularly with relational databases to implement <a href="https://en.wikipedia.org/wiki/Pseudonymization">pseudonymisation</a> and replace any personal data with anonymised data, because unpicking the foreign key relationships to allow true deletion is impractical. In this case the accessible identifier must also be anonymised; this is clearly not possible if it is the primary key or used in foreign key relationships.</p>
<p>As such, for people you often need at least <em>two</em> identifiers: One that is used only for internal purposes, and one that is exposed to the person. The internal identifier should be used as the key for all internal storage but never exposed in any UI, URL, JWT, etc. which is where the external identifier should be used.</p>
<p>The reason for at least two rather than exactly two is that if you integrate with any third parties then you should provide a different identifier to each of those third parties, or even to different accounts with the same third party if applicable. This is part of defence in depth as it means the third party cannot identify people they don’t have access to, and it makes it harder for them to correlate information. In fact, individually translating the identifier of <em>any</em> resource for third parties is generally good practice.</p>
<h2 id="cross-system-compatibility">Cross-system compatibility</h2>
<p>Identifiers are one of the few things in systems that tend to stay stable, unlike languages and storage or data warehousing technologies, so we want identifiers to work with any future technology choices we may make. In addition, if they are sent to third parties then they may be processed using a completely different set of technologies of which we are unaware. To ensure the highest level of compatibility there are a number of considerations.</p>
<p>The most important consideration is not to create identifiers that differ only by case. Some data stores such as DynamoDB are case sensitive, others such as Postgres have selectable collations which control case sensitivity, and others such as Elasticsearch are <em>sometimes</em> case sensitive depending on both index and query. Having identifiers that differ only by case makes it more likely that they will collide or be conflated, causing hard-to-find bugs that potentially have security or privacy implications.</p>
<p>If the identifiers will not differ by case then it also makes sense to define a canonical case so they can always be compared lexically as well as logically. For example <code class="language-plaintext highlighter-rouge">42A23DAB-67AD-4AD7-B7A9-3DAFF34F6C02</code> and <code class="language-plaintext highlighter-rouge">42a23dab-67ad-4ad7-b7a9-3daff34f6c02</code> are <em>logically</em> the same UUID but are lexically unequal. If identifiers are always represented in a particular case (I prefer lowercase) then systems can convert identifiers to the canonical case and then it doesn’t matter if they use logical or lexical comparison.</p>
<p>On the subject of case, we don’t want to deal with anomalies like the <a href="http://www.i18nguy.com/unicode/turkish-i18n.html">Turkish I problem</a> so restrict identifiers to ASCII characters. As identifiers may end up being used in URLs, headers, cookies, etc. it’s also best to stay away from any character that might have special meaning in those places. A safe character set to stick to is <code class="language-plaintext highlighter-rouge">0-9</code>, <code class="language-plaintext highlighter-rouge">a-z</code>, <code class="language-plaintext highlighter-rouge">-</code> and <code class="language-plaintext highlighter-rouge">_</code>.</p>
<h2 id="labelling">Labelling</h2>
<p>One of the issues with identifiers is trying to work out what they refer to if you don’t have the context, and this gets worse in distributed systems or with data warehousing. A neat solution to this is to include labels in the identifiers.</p>
<p>Stripe uses a fairly basic prefix approach with their API keys (which are really just a form of identifier) using a <code class="language-plaintext highlighter-rouge">pk_</code> prefix for publishable keys, <code class="language-plaintext highlighter-rouge">sk_</code> prefix for secret keys, and then a <code class="language-plaintext highlighter-rouge">live_</code> or <code class="language-plaintext highlighter-rouge">test_</code> environment slug. For example, a live publishable key looks like <code class="language-plaintext highlighter-rouge">pk_live_8hTOJDTo...</code>.</p>
<p>Amazon uses a much more complex approach with <a href="https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html">Amazon Resource Names (ARNs)</a> which have a structured format including details of where the resource is and which account it belongs to, e.g. <code class="language-plaintext highlighter-rouge">arn:partition:service:region:account-id:resource-id</code>.</p>
<p>These two examples also show a different approach to identifier uniqueness scope. In Stripe the resource identifier is globally unique, whereas in AWS it is only necessarily unique within the scope of the partition, service, region and account because the complete identifier is always used.</p>
<p>At Deliveroo we chose an approach somewhere in between these two, using the format <code class="language-plaintext highlighter-rouge">drn:partition:market:resource-name:resource-id</code> which gave us future flexibility to partition the system, a market because all resources are strongly market-affine, and a globally unique resource identifier that can be used standalone in the many existing URLs without ambiguity (as those URLs typically already identify both the market and resource name).</p>
<p>We decided not to include an environment component as it added a generation and validation burden with little practical benefit because environments are strongly separated using other mechanisms. The market is deliberately abstracted from the shards in which that market resides to allow shards to be rearranged as markets grow.</p>
<p>Labelling identifiers makes them much more useful, but be careful not to paint yourself into a corner by encoding aspects of your business that might change in future.</p>
<h2 id="summary">Summary</h2>
<p>Identifiers are more complex to get right than you might think. Here’s a summary of the recommendations in this post which should help:</p>
<ul>
<li><strong>Never</strong> use sequential integer identifiers</li>
<li><strong>Usually</strong> use v4 UUID identifiers</li>
<li><strong>Always</strong> represent identifiers in lowercase</li>
<li><strong>Consider</strong> labelling identifiers with their meaning</li>
<li><strong>Remember</strong> that identifiers can be personal data</li>
</ul>
<p>As always with software development, “never” doesn’t mean never and “always” doesn’t mean always. You’ll find that many of these recommendations are broken in the wild, and often for good reason. However, this is a good base set of guidelines to start from, and you should think carefully before deviating.</p>
<hr />
<p><em>2020-03-24: Updated with additional paragraph about integer identifier security problems in the aftermath of database failure.</em></p>Greg Beechgreg@gregbeech.comIdentifiers aren’t usually considered very interesting. Indeed, popular frameworks such as Django or Hibernate find them so uninteresting that they ship with the default of auto-incrementing integers so users don’t need to be concerned with them. Unfortunately using sequential identifiers is a bad idea, and if you don’t think about your identifiers up front then fixing things later can be very time consuming. Here’s most of what you need to know about identifiers.Deliberate engineering2019-11-07T00:00:00+00:002019-11-07T00:00:00+00:00https://www.gregbeech.com/2019/11/07/deliberate-engineering<p>When building a software project with a given scope there are three main constraints you can trade off: good, fast, and cheap. This is commonly known as the project management triangle. Focusing on one will have an effect on the others. For example, trying to do things cheap will adversely affect good and fast.</p>
<p>Cheap in this context uses salary as a proxy for the skill and experience levels of engineers. Don’t take it too literally as you may be paying over the odds for engineers, so cheap really means projects that are staffed by engineers lacking in skill and/or experience. It could also mean projects that are under-resourced, but that problem is much more obvious so we won’t talk about it here.</p>
<p>The thing many people don’t seem to realise is that you don’t have a free choice with these constraints. Much like the CAP theorem only allows you to trade off between consistency and availability because network partitions are beyond your control, with software projects you can only really trade off between good and fast because cheap is largely defined by your company’s hiring policy.</p>
<p>Some companies have such a poor hiring policy that they are unable to choose either good or fast because there simply is not the skill and experience available. Companies like this tend to have a lot of engineers who have been at the company less than two years; most of the rest have been at the company for over a decade and are just waiting for retirement. Their interviews are <em>ad hoc</em> with no rubric and no hiring bar. This is, sadly, a large number of companies.</p>
<p>At the other end of the scale there are the companies who have relatively small but highly skilled teams who can choose both fast and good. These companies have a high proportion of senior engineers, including some that are well known in the industry, and low staff turnover rates. They have multi-stage interviews with standardised rubrics and a very high hiring bar. If you don’t know whether you work at one of these companies, then you probably don’t.</p>
<p>The remainder are somewhere in between. They have a relatively bottom-heavy pyramid of seniority with lots of junior/mid-level engineers and a much smaller number of seniors. They have structured interviews, but the rubrics may be absent or lacking and interviewers may not be calibrated, so the hiring bar is inconsistent. This group includes most startups and young companies.</p>
<p>This last group of companies is the most interesting. They have the choice between fast and good, but struggle to do both because the people who could do this are too thinly spread.</p>
<p>The trouble is, they often can’t do either fast or good.</p>
<h2 id="why-fast-fails">Why fast fails</h2>
<p>Every company wants to deliver new features fast, because features are thought to be what drives sales, conversion, etc. and so getting them out in front of customers is the most important thing. When the pressure is on to go fast, people want to look and feel like they’re going fast.</p>
<p>The majority of engineers appear to believe the fallacy that any activity other than writing code isn’t “real work” and is just ceremony that slows them down. This means when they want to feel like they’re going fast they’ll forgo—or merely pay lip service to—gathering requirements, writing design documents, doing research, and building proof-of-concepts to test and measure. Instead they jump straight into the code and start building.</p>
<p>They don’t even realise that they’re trading off good. They’re getting peer review on the code and fixing up the issues pointed out there, they have good test coverage, and they may even be extracting reusable components. They feel like they’re moving fast, and they feel like they’re doing good quality work. In the first month or two it can be hard to tell a successful project from a doomed one.</p>
<p>Building software is a bit like building a house. We all know roughly what shape a house is, that it’s built of bricks, and that it has windows and doors. If you were to start laying down bricks and putting in windows then fairly quickly you’ll have something that starts to resemble a house. You might even convice yourself you’ll be finished soon.</p>
<p>Then the rain comes and because you didn’t build foundations one of the walls starts to subside and crack, and you need to spend time propping it up and repairing it. Then you realise you forgot to hook up the gas or water so you have to dig up the floor. Then you find out you’ve got two kitchens but no bathroom. Then you find out that the client wanted a barn, not a house.</p>
<p>By forgoing the requirements and blueprints and foundations the house starts to take shape quickly, but finishing it off takes far longer than originally planned because it keeps needing to be re-engineered. The end result isn’t quite what anybody wanted, and it keeps trying to fall over because it has no foundations.</p>
<p>I’ve lost count of the number of software projects I’ve seen run this way.</p>
<p>For any nontrivial software project, the faster you think you’re starting, the slower you’ll end up finishing. If you finish at all. As the estimates and timelines start to stretch and the priorities of the company evolve, the probability of projects getting cancelled increases significantly. The alternative is that hard deadlines get set and the project gets delivered, but is unreliable and unmaintainable, and consumes months of time after launch to support and re-engineer it.</p>
<p>You go fast to go slow.</p>
<h2 id="go-slow-to-go-fast">Go slow to go fast</h2>
<p>If you want to go fast then you have to understand how to go fast, and as you’ve probably deduced by now, that means starting slowly.</p>
<p>Define the problem statement, the goals, and the non-goals so you can scope the project more tightly. Work out the minimal set of features you can cohesively deliver, which shortcuts make sense, and what tradeoffs you’re prepared to make in non-functional requirements like reliability, cost, or performance. Design the architecture and data model in a document and get feedback from people with experience in the domain and/or the technologies, to resolve potential problems before you’ve coded them.</p>
<p>These activities don’t need to take long. For a relatively straightforward feature or service in a well understood domain using boring technologies you should be able to get the docs written and peer reviewed in a couple of days.</p>
<p>That’s not to say you should timebox it to a couple of days though. For a new service I worked on recently, research, design and prototyping took a few weeks. It was a good job we did it though, because many of our initial hypotheses were invalidated in ways that wouldn’t have become apparent until after it was live. It’s much quicker to correct mistakes on (virtual) paper than it is in code.</p>
<p>Once you’ve done the preparation, you can deliver deliberately and fast. On any nontrivial project, starting slowly will mean you deliver faster than if you just dive in and start writing code.</p>
<p>This doesn’t necessarily mean you’ll end up with good software. If fast is the main priority, as it might be for features of uncertain long-term value, then you will make some design decisions that are not objectively good. For example using a data store with a high hosting cost to reduce implementation time, or omitting reliability features like retries or circuit breakers because downtime is an acceptable risk. What’s important is that you considered these decisions, made them deliberately, and you know the future cost of making the software good, should that prove necessary.</p>
<p>On the flip side if your priority leans towards good, as it might for features that form the core of the business and which you know will need to be evolved over years, then deliberate engineering helps you to ensure that you meet your goals, and also gives you the best chance of delivering it fast.</p>Greg Beechgreg@gregbeech.comWhen building a software project with a given scope there are three main constraints you can trade off: good, fast, and cheap. This is commonly known as the project management triangle. Focusing on one will have an effect on the others. For example, trying to do things cheap will adversely affect good and fast.Tail recursion, fold, and more2019-03-30T00:00:00+00:002019-03-30T00:00:00+00:00https://www.gregbeech.com/2019/03/30/tail-recursion-fold-and-more<p>People new to functional languages often struggle a bit with the concept of recursion, and in particular tail recursion. In this post we’ll look at both recursive and tail recursive functions using the substitution model to show their effect on the stack. We’ll also walk through deriving <code class="language-plaintext highlighter-rouge">fold</code>, <code class="language-plaintext highlighter-rouge">map</code>, and other functions from first principles because it fairly naturally follows on from this.</p>
<p>To start with let’s write a recursive function to sum a list:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">sum</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">list</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Nil</span> <span class="k">=></span> <span class="mi">0</span>
<span class="k">case</span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span> <span class="k">=></span> <span class="n">x</span> <span class="o">+</span> <span class="nf">sum</span><span class="o">(</span><span class="n">xs</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Like most recursive functions we have two cases. The base case of an empty list (<code class="language-plaintext highlighter-rouge">Nil</code>) has a sum of zero, and when this case is reached the function stops recursing. The inductive case breaks the problem down into two smaller problems by defining the sum as the first element plus the sum of the rest of the elements, so each time this case is hit it will recurse by calling itself.</p>
<p>This function works fine for small lists, but unfortunately once we start trying it with larger lists it overflows the stack and errors.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">sum</span><span class="o">(</span><span class="nv">List</span><span class="o">.</span><span class="py">range</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">1000</span><span class="o">))</span> <span class="c1">// 499500</span>
<span class="nf">sum</span><span class="o">(</span><span class="nv">List</span><span class="o">.</span><span class="py">range</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">10000</span><span class="o">))</span> <span class="c1">// java.lang.StackOverflowError</span>
</code></pre></div></div>
<p>To see why this is we can substitute some real values and examine how the function executes. For a four element list the steps would be as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sum([1, 2, 3, 4])
1 + sum([2, 3, 4])
1 + 2 + sum([3, 4])
1 + 2 + 3 + sum([4])
1 + 2 + 3 + 4 + sum([])
1 + 2 + 3 + 4 + 0
1 + 2 + 3 + 4
1 + 2 + 7
1 + 9
10
</code></pre></div></div>
<p>Each time the function executes it stores the current value of <code class="language-plaintext highlighter-rouge">x</code> in its stack frame, and then runs <code class="language-plaintext highlighter-rouge">sum</code> on the rest of the list in a new stack frame. This gives rise to the distinctive triangle shape as the stack grows with the list length, and then unwinds to compute the results. For longer lists more stack frames are allocated and so with a long enough list the stack will overflow. The default stack on the 64-bit JVM is only 1MB so a few thousand frames will typically do it.</p>
<p>Let’s look at an alternative way of writing the function, which accumulates the result as it recurses using a helper function. Here <code class="language-plaintext highlighter-rouge">loop</code> is the recursive function, and <code class="language-plaintext highlighter-rouge">sum</code> is a wrapper to keep the desired method signature that just starts the first loop iteration.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">sum</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">rest</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">rest</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Nil</span> <span class="k">=></span> <span class="n">acc</span>
<span class="k">case</span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span> <span class="k">=></span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span> <span class="o">+</span> <span class="n">x</span><span class="o">,</span> <span class="n">xs</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">loop</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">list</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Using the substitution approach again, we can see that this has a different execution profile because there is no operation after the call to <code class="language-plaintext highlighter-rouge">loop</code>; each function in the stack just returns the value from the recursive call. This pattern of having the recursive call as the last operation the function performs is known as tail recursion.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sum([1, 2, 3, 4])
loop(0, [1, 2, 3, 4])
loop(1, [2, 3, 4])
loop(3, [3, 4])
loop(6, [4])
loop(10, [])
10
10
10
10
10
</code></pre></div></div>
<p>The reason tail recursive functions are important is that due to there being nothing to do after the recursive call, rather than adding a new stack frame for the recursive call the compiler can <em>replace</em> the current stack frame for that call. This is known as tail-call optimisation, and means the stack usage actually looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sum([1, 2, 3, 4])
loop(0, [1, 2, 3, 4])
loop(1, [2, 3, 4])
loop(3, [3, 4])
loop(6, [4])
loop(10, [])
10
10
</code></pre></div></div>
<p>The stack depth is now constant regardless of the list length, so the function succeeds for any list:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">sum</span><span class="o">(</span><span class="nv">List</span><span class="o">.</span><span class="py">range</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">1000</span><span class="o">))</span> <span class="c1">// 499500</span>
<span class="nf">sum</span><span class="o">(</span><span class="nv">List</span><span class="o">.</span><span class="py">range</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">10000</span><span class="o">))</span> <span class="c1">// 49995000</span>
</code></pre></div></div>
<p>One thing we should also do is add the <code class="language-plaintext highlighter-rouge">@tailrec</code> annotation to the <code class="language-plaintext highlighter-rouge">loop</code> function. This is a compiler instruction which tells the compiler that it must perform tail-call optimisation on the function, or fail if it cannot. The Scala compiler will perform tail-call optimisation where possible anyway, so the annotation is more a safety check for the programmer.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">sum</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="o">{</span>
<span class="nd">@tailrec</span> <span class="k">def</span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">rest</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">rest</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Nil</span> <span class="k">=></span> <span class="n">acc</span>
<span class="k">case</span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span> <span class="k">=></span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span> <span class="o">+</span> <span class="n">x</span><span class="o">,</span> <span class="n">xs</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">loop</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">list</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>There are loads of other functions we can write using a similar tail recursive approach. For example, here’s a function to find the maximum value in a list:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">max</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="o">{</span>
<span class="nd">@tailrec</span> <span class="k">def</span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">rest</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">rest</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Nil</span> <span class="k">=></span> <span class="n">acc</span>
<span class="k">case</span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span> <span class="k">=></span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span> <span class="n">max</span> <span class="n">x</span><span class="o">,</span> <span class="n">xs</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">loop</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">list</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This looks very similar to <code class="language-plaintext highlighter-rouge">sum</code> though. In fact the only difference is what we do to the numbers to accumulate them, so as good programmers we should factor that out into a higher-order function, which we call <code class="language-plaintext highlighter-rouge">fold</code>. To make it more general purpose we also factor out the seed value that the accumulator starts with. Note that the function <code class="language-plaintext highlighter-rouge">f</code> is in a second parameter list to help with Scala’s flow-based type inference.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">seed</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)(</span><span class="n">f</span><span class="k">:</span> <span class="o">(</span><span class="kt">Int</span><span class="o">,</span> <span class="kt">Int</span><span class="o">)</span> <span class="k">=></span> <span class="nc">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="o">{</span>
<span class="nd">@tailrec</span> <span class="k">def</span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">rest</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">rest</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Nil</span> <span class="k">=></span> <span class="n">acc</span>
<span class="k">case</span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span> <span class="k">=></span> <span class="nf">loop</span><span class="o">(</span><span class="nf">f</span><span class="o">(</span><span class="n">acc</span><span class="o">,</span> <span class="n">x</span><span class="o">),</span> <span class="n">xs</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">loop</span><span class="o">(</span><span class="n">seed</span><span class="o">,</span> <span class="n">list</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">def</span> <span class="nf">sum</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="mi">0</span><span class="o">)(</span><span class="k">_</span> <span class="o">+</span> <span class="k">_</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">max</span><span class="o">(</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="mi">0</span><span class="o">)(</span><span class="k">_</span> <span class="n">max</span> <span class="k">_</span><span class="o">)</span>
</code></pre></div></div>
<p>This is great for list of integers, but it can be generalised further to work on any kind of list, and to return a value that might be a different type to what’s contained in the list by introducing some generic types.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fold</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">],</span> <span class="n">seed</span><span class="k">:</span> <span class="kt">B</span><span class="o">)(</span><span class="n">f</span><span class="k">:</span> <span class="o">(</span><span class="kt">B</span><span class="o">,</span> <span class="kt">A</span><span class="o">)</span> <span class="k">=></span> <span class="n">B</span><span class="o">)</span><span class="k">:</span> <span class="kt">B</span> <span class="o">=</span> <span class="o">{</span>
<span class="nd">@tailrec</span> <span class="k">def</span> <span class="nf">loop</span><span class="o">(</span><span class="n">acc</span><span class="k">:</span> <span class="kt">B</span><span class="o">,</span> <span class="n">rest</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">B</span> <span class="o">=</span> <span class="n">rest</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Nil</span> <span class="k">=></span> <span class="n">acc</span>
<span class="k">case</span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span> <span class="k">=></span> <span class="nf">loop</span><span class="o">(</span><span class="nf">f</span><span class="o">(</span><span class="n">acc</span><span class="o">,</span> <span class="n">x</span><span class="o">),</span> <span class="n">xs</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">loop</span><span class="o">(</span><span class="n">seed</span><span class="o">,</span> <span class="n">list</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Using this more generalised definition we can get some interesting results, e.g. starting with an empty list and appending to it we can get the list out again, possibly with changes. (NB: Don’t use <code class="language-plaintext highlighter-rouge">:+</code> with <code class="language-plaintext highlighter-rouge">List[A]</code> in Scala because it’s very inefficient; this is a theoretical implementation rather than one optimised for performance.)</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">fold</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="mi">4</span><span class="o">),</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">Int</span><span class="o">])((</span><span class="n">xs</span><span class="o">,</span> <span class="n">x</span><span class="o">)</span> <span class="k">=></span> <span class="n">xs</span> <span class="o">:+</span> <span class="n">x</span><span class="o">)</span> <span class="c1">// List(1, 2, 3, 4)</span>
<span class="nf">fold</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="mi">4</span><span class="o">),</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">Int</span><span class="o">])((</span><span class="n">xs</span><span class="o">,</span> <span class="n">x</span><span class="o">)</span> <span class="k">=></span> <span class="n">xs</span> <span class="o">:+</span> <span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span> <span class="c1">// List(2, 3, 4, 5)</span>
<span class="nf">fold</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="mi">4</span><span class="o">),</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">Int</span><span class="o">])((</span><span class="n">xs</span><span class="o">,</span> <span class="n">x</span><span class="o">)</span> <span class="k">=></span> <span class="n">xs</span> <span class="o">:+</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span><span class="o">)</span> <span class="c1">// List(1, 4, 9, 16)</span>
</code></pre></div></div>
<p>Once again these look remarkably similar with only the transformation applied to <code class="language-plaintext highlighter-rouge">x</code> differing, so we can capture this pattern in another higher-order function, commonly known as <code class="language-plaintext highlighter-rouge">map</code>. It turns out <code class="language-plaintext highlighter-rouge">map</code> is just a specialised form of <code class="language-plaintext highlighter-rouge">fold</code>.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">map</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])(</span><span class="n">f</span><span class="k">:</span> <span class="kt">A</span> <span class="o">=></span> <span class="n">B</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">B</span><span class="o">]</span> <span class="k">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">B</span><span class="o">])(</span><span class="k">_</span> <span class="o">:+</span> <span class="nf">f</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
</code></pre></div></div>
<p>In fact most list functions can be written as a specialised form of fold. You wouldn’t want to write them quite like this as the <code class="language-plaintext highlighter-rouge">:+</code> and <code class="language-plaintext highlighter-rouge">++</code> operators have O(N) performance on lists, and functions like <code class="language-plaintext highlighter-rouge">contains</code> or <code class="language-plaintext highlighter-rouge">find</code> ought to short-circuit. However, it’s interesting to see that <code class="language-plaintext highlighter-rouge">fold</code> is really the base abstraction for many more common operations.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">contains</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])(</span><span class="n">f</span><span class="k">:</span> <span class="kt">A</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="kc">false</span><span class="o">)(</span><span class="k">_</span> <span class="o">||</span> <span class="nf">f</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
<span class="k">def</span> <span class="nf">find</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])(</span><span class="n">f</span><span class="k">:</span> <span class="kt">A</span> <span class="o">=></span> <span class="nc">Boolean</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="nv">Option</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span> <span class="o">{</span>
<span class="nf">case</span> <span class="o">(</span><span class="nc">None</span><span class="o">,</span> <span class="n">a</span><span class="o">)</span> <span class="k">if</span> <span class="nf">f</span><span class="o">(</span><span class="n">a</span><span class="o">)</span> <span class="k">=></span> <span class="nc">Some</span><span class="o">(</span><span class="n">a</span><span class="o">)</span>
<span class="nf">case</span> <span class="o">(</span><span class="n">acc</span><span class="o">,</span> <span class="k">_</span><span class="o">)</span> <span class="k">=></span> <span class="n">acc</span>
<span class="o">}</span>
<span class="k">def</span> <span class="nf">flatMap</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])(</span><span class="n">f</span><span class="k">:</span> <span class="kt">A</span> <span class="o">=></span> <span class="nc">List</span><span class="o">[</span><span class="kt">B</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">B</span><span class="o">]</span> <span class="k">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">B</span><span class="o">])(</span><span class="k">_</span> <span class="o">++</span> <span class="nf">f</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
<span class="k">def</span> <span class="nf">length</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="mi">0</span><span class="o">)((</span><span class="n">n</span><span class="o">,</span> <span class="k">_</span><span class="o">)</span> <span class="k">=></span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">reverse</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">A</span><span class="o">])((</span><span class="n">xs</span><span class="o">,</span> <span class="n">x</span><span class="o">)</span> <span class="k">=></span> <span class="n">x</span> <span class="o">::</span> <span class="n">xs</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">take</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">list</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">],</span> <span class="n">n</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
<span class="nf">fold</span><span class="o">(</span><span class="n">list</span><span class="o">,</span> <span class="nv">List</span><span class="o">.</span><span class="py">empty</span><span class="o">[</span><span class="kt">A</span><span class="o">])((</span><span class="n">xs</span><span class="o">,</span> <span class="n">x</span><span class="o">)</span> <span class="k">=></span> <span class="nf">if</span> <span class="o">(</span><span class="nf">length</span><span class="o">(</span><span class="n">xs</span><span class="o">)</span> <span class="o"><</span> <span class="n">n</span><span class="o">)</span> <span class="n">xs</span> <span class="o">:+</span> <span class="n">x</span> <span class="k">else</span> <span class="n">xs</span><span class="o">)</span>
</code></pre></div></div>
<p>Even these functions can be generalised further to operate on any effect <code class="language-plaintext highlighter-rouge">F[A]</code> rather than just <code class="language-plaintext highlighter-rouge">List[A]</code> which is where libraries like <a href="https://typelevel.org/cats/">cats</a> come in. But we’re already veering way off topic so let’s call it a wrap.</p>Greg Beechgreg@gregbeech.comPeople new to functional languages often struggle a bit with the concept of recursion, and in particular tail recursion. In this post we’ll look at both recursive and tail recursive functions using the substitution model to show their effect on the stack. We’ll also walk through deriving fold, map, and other functions from first principles because it fairly naturally follows on from this.