Performance Feedback


Predator Prey Model (The original uploader was Lämpel at German Wikipedia.)

Here is a phrase that most people hear at some point in their lives:

“I don’t see the results.”

Whether it’s a test score, earnings, or something else, what you’re getting is a form of feedback.

Feedback is defined, more generally, as a system where outputs are routed to inputs. In cases where we are measuring performance, we use something like a test score as a measure of how much effort we should put in to studying a subject.

Feedback can also describe natural systems like the predator-prey model. As the predator population rises, there are fewer prey to go around. Eventually the predator population collapses because there isn’t enough food, and the prey can grow in number again.

Predator-prey describes a “negative” feedback loop, because the population always stabilizes. Many natural and mechanical systems fall into this category. Negative feedback is desirable for predictability.

But with performance feedback we are usually looking for a “positive” feedback loop, where continued practice, development of new techniques and technologies will manifest in better results.

Sports Science

Everyone is interested in performance. Better, faster, stronger, right?

Now, here’s the problem. Most performance feedback is actually really bad - it comes slowly and with little reinforcement. Our general solution to improving performance is random trial and error plus persistence.

Beatmania Rating

One of the compelling things about games is that they can keep breaking down performance into smaller and smaller pieces that are easier to measure, give faster feedback and are more responsive to trial and error methods.

The process of game design itself happens on two levels, though: There’s feedback for players, and feedback for designers.

One of the reasons why players tend to engage in a game is because it gives frequent and encouraging feedback.

Any time someone picks new feedback goals, the potential for a new game exists. You might be able to finish the game, but how fast can you finish it? Can you finish it with restrictions? As you pick out more of these goals, your playstyle changes and becomes more refined to meet the challenge, and new parts of the game are exposed.

MitchFlowerPower: Super Mario Bros. 3 Warpless

Although game designers can think about how to improve feedback for players, feedback is also needed for every other part of the creative process.

The workflow used to create a game is driven in large part by the kinds of feedback used. Just asking what people think or waiting for sales numbers and review scores to come in are simple forms of feedback that immediately come to mind as a way of measuring a game’s quality, but they are slow and only measure a sort of final result. So they aren’t a good option for an individual contributor.

Instead, part of the project planning should include the feedback mechanisms, which might include but aren’t limited to:

Adherence to a set of feedback mechanisms essentially defines how a project will take form over a longer period of time. Although on any given day you might have a random set of ideas, feedback guides you to a more precise set over time.

Bob Ross and 2600 Basic Programming

For media producers, most of the feedback is determined by the tools: A programmer develops the project in terms of lines of code, bugs and features, an artist in terms of images, models, animations.

Being overly tied to the tool is the fallacy, though. A programmer who only programs, or an artist who only draws, will lose track of other goals. It’s in taking a layered approach that mixes multiple feedback methods that the interesting stuff tends to happen.

When you go to do exercise, there are multiple layers of feedback that are so simple that you might not notice them.

Fitness Log

First, there’s the feedback of your body as you perform a technique.

Second, there’s some count of time, weight, repetitions and sets to inform you of how well you did.

Third, there’s a log of your performance each session to gauge if your program is resulting in the desired improvement.

Each layer presents new opportunities for analysis and improvement.

Compositing

Layering is applicable to all kinds of projects: media is often a simple composition of different layers. Layering is one way to explain why companies tend to be orders of magnitude more profitable than individuals: a well-organized team naturally has multiple perspectives and layers of feedback, leading to dramatically higher performance.

Shocken Department Store

It also supports most arguments for diversity. A diverse jungle of opinion where numerous perspectives are forced to reconcile will test ideas more rigorously than a monoculture.

Jungle

When I write articles for this series, my feedback starts by musing on a question as if it were an internet comment thread. This turns into the writing of bullet points in an outline and then gradually fleshing out each bullet point for conceptual coherence, until I have something that explores the question to my satisfaction.

Then to test the writing I record it spoken out loud and revise it to work as a spoken essay. This process originated because I thought I might do the series as videos, but I realized that the format didn’t capture everything I wanted. However, the process of recording added structure that I wouldn’t have otherwise.

Finally, I add supplemental content with images, video, and an interactive part. These reinforce the coherence of the work further. I’m still learning how to create good interactive content for this series, but one of my models for how it can be done is the Exploratorium museum.

Results of the articles in terms of popularity numbers, shares, etc. are trackable with many online tools, but they fail to capture impact. I can know how many people clicked a link, but not how many people I influenced. So I don’t rely on these numbers, although I may want to learn who is talking about the work.

Likewise, results in terms of profit figures, career success, and so on, are not a form of feedback I can access directly. If an article results in getting work I wouldn’t have otherwise, obviously I profited, but that’s not a thing I can dial in knowingly as I build each work.

So there is an element of scale and knowing which scale of feedback is feasible. If I could see a number called “quality” go up and down as I make something, it would help, but it wouldn’t tell me what parts I had to improve. To really know what to do next, I have to have multiple mechanisms for feedback.

Hence, when you hear “I’m not seeing the results,” it’s important to take into consideration what form of feedback the speaker is looking at. If it’s not something you can usefully optimize your performance against, you need to disregard it and find other methods.

But how does one develop better feedback, then?

Earlier, I enumerated some ideas for feedback methods, but let’s try to get at it from a more principled approach.

Traffic Signal

The most basic feedback mechanism you could have is a success or failure signal.

For example, if you have a rule like “the player should never be forced to restart the game from the beginning”, you have a failure signal if that scenario happens.

From a system of many success and failure signals, you can perform deduction and induction. These are the two logical mechanisms by which we can infer new ideas.

Sherlock Holmes

Deduction means putting boundaries on statements, to narrow the possibilities by using logic to rule out propositions, until a final conclusion is within reach: “Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.” The constraints of deduction create facts that build upon each other, making for a tool that generates many surprising results.

Johan Rijpma: Extrapolation

Induction is, correspondingly, the art of generalizing your statements. Induction is inherently less certain than deduction, but allows us to make useful extrapolations: An example of an inductive statement is that if we have rolled six on a pair of dice ten times in a row, our expectation is that our next roll will also be a six.

Dice

Inductive reasoning takes on uncertainty by extrapolating, while deduction constrains it. In the case of the dice roll, deduction would say, “previous results don’t change the probability of the next roll. The chances of it not being a six should always be the same.”

To contrast these two ideas, there is also the concept of abduction, which is our “best available” or “most likely” explanation. Abduction is a starting point, but typically less surprising in terms of generating new ideas and feedback, since it’s a guess. It builds a hypothesis and tells us what we reasonably expect, not what is impossible or what we could strive for. Abduction needs to be backed up by other processes to make a convincing argument. If we keep rolling a six, an abductive statement might be “perhaps the dice are weighted” - a hypothetical which can be confirmed or denied with deduction. Although Sherlock Holmes speaks in terms of deduction, many of his arguments are built on abductive hypothesis.

Applying logical inferences towards creativity suggests focusing on conceptual coherence: Within the set of ideas you’re working with, pick the ideas that best support each other, rather than contradicting each other, to make your reasoning more fluid. This is a tool I frequently use now, and have been itching to write about in more depth. So it’ll be one I cover soon.