data is exponential

I made a mistake at work. I thought I had sum-producted two arrays, but I had only passed in one. Because the number wasn't weighted, it was off by an order of magnitude and I told my PM she had to move 8M incremental message sends when she really just has to move 800K. I think both are tall orders, so maybe the conclusion doesn't change very much, but it's the fact that I made a mistake that's getting me.

When I got fired at The Browser Company, Jane said "recent mistakes had really undermined confidence in my work." I'm sensitive to mistakes because of this (even though there was definitely other stuff going on at that company). Jane raised my baseline level of anxiety, and a higher baseline level of anxiety leads to a kind of a mistake doom loop. I'm anxious so things feel more urgent and fast, which means I make more mistakes, which leads to more anxiety. The people who perform best under pressure are people who are again, a little dumb..1

A persistent piece of feedback I've received at work is that I bounce around a lot, or kind of seem frenetic, and all over the place. This happens when I'm anxious I think, but also when I'm excited.

These are dangerous tendancies for someone who does data work. There is a ton of complexity in data, and the complexity of any given task balloons very quickly. An example: my dataset is one column with four values. One of those values is NULL. I’m interested in how these four values relate to another set of values so I drop the nulls from my first column and join to a second column. This second column has three values so I get some output with 9 distinct pairs of values (3 non-null from the first, 3 from the second). Something looks off in this breakdown so I go back and run some cuts on the original column: turns out the NULLs were coming from Android! So my breakdown was accidentally conditioning on not having this weird Android bug. What initially seemed like a simple question answerable via a defensible 2-step procedure turned into a recursive 4 step process at minimum, reliant on a heavy dose of intuition.

The only way out of this, I think, is through. RC said it best: "We learn something by doing it. There is no other way." And as a corollary to that, the way to do something initially is to do it wrong, and slow. This is the way to learn rigor.

I think learning rigor is a noble goal. I would like to learn rigor. I would like to have skills. But I've been angry at work for weeks, maybe even months now. There are structural issues at work I know. I've since found out that the founder of the Browser Company is known to be a bit unpleasant. At Meta, grading based on explicit measures of "influence" implicitly favors people who had the good fortune to be born male or white or both. I don't know that I have many useful yardsticks to measure myself against at work, and that makes me feel powerlessness, especially in the face of these mistakes. I'm angry because I'm not sure I can build a career where being myself is enough.


  1. Not dumb, exactly. I think I said this in that post, but they're just good at quieting thoughts. My mom calls this "forcing herself", my dad says "consciously, as an act of will" — it's all just another name for tenacity. Even if it does make you seem a little head empty.