58. Writing a Playtest Report
Step 6. Knowledge Sharing
Although it might appear to be time consuming to have to write up a report, there are advantages to doing so:
It converts tacit knowledge that exists inside a few people’s heads into an explicit form that can be consumed by many others.
It is a persistent account of what happened.
Having as many people read the playtest findings as possible allows for a diversity of potential solutions - good ideas can come from anywhere.
Report Format
What you decide to include in your report is up to you, and can be as brief, or as detailed as you feel is necessary. I will outline possible sections below and why I think they are useful, but how much you ultimately decide to use might depend on time available or how quickly the team need the results.
Front Cover or Header
The purpose of this is to clearly state the purpose of the contents.
Name/codename of game.
Meaningful description of playtest, e.g. 2nd Usability Playtest on Tutorial
Version/build of the game that was tested
Date report was written - day, month, year
Report structure
There is no ‘one way’ to structure a usability playtest report, but some of the more common approaches are:
By research question, e.g. take each hypothesis in turn.
By game level/section/beat - report issues depending on sections of the game.
By game feature - report issues as they relate to each feature in the game, e.g. inventory, controls, HUD, menu navigation.
By time (player’s journey) - report issues as the player encountered them.
By issue severity (high to low) - report highest priority issues first.
In addition, to help you find the best structure for your team, you might want to ask them:
Do they have a preferred structure to read through the findings, or if there is a way they are expecting/used to?
What will they do with the report once they receive it? For example, some teams might want to see the issues in their task management software, can you make it easier for them to do that?
Summary and Key Findings
Providing a summary of the key findings is useful for someone who doesn’t need the details, but does need to be aware of an overview of what happened, and the implications - the ‘so what’.
The summary page might inform the reader about:
Purpose - what was the aim of the playtest? Why was this test being done?
Context - put this test in relation to others and/or the development timeline, e.g.
This was the first time the lobby usability was evaluated.
This is the second iteration of FTUE/onboarding playtesting.
Main findings - summarise the answers to the main research questions. If you can visualise this even better (graphical, colour coding etc).
The ‘so what’ - When taken as a whole, what impact do the findings have on the game?
Reporting Findings
This is the main content of your report, documenting what happened (player behaviour), and if possible, why it happened (from sensemaking).
One possible format for writing up each issue is to take one page per issue. By ‘page’ this could be one slide in your presentation software, or one page in a word processor. Although this might look like it creates a long report, it makes reading predictable, as you know there’s one issue per page.
For each issue or hypothesis being documented, aim to cover the following points:
Hypothesis Description - what was the research question the team had?
Hypothesis Outcome - Pass/Fail. I’d suggest to use colour coding to make this stand out. For example, putting a ‘fail’ in red will help make sure no one overlooks these important findings.
Severity - Positive, High, Medium, Low etc how important is this issue to address?
Description of the issue - be factual - describe the player’s actual behaviour (what you observed) or player’s mental model (from interview data). This description should help the reader:
Identify the feature or system being discussed. If there is no name for the issue in question then you might have to use an image and refer to it.
Explain the issue - what is the actual issue with the feature or system identified? This will either be a behavioural issue - the player did or did not do something, or a mental model issue - the player doesn’t fully understand how something works.
Why it occurred - what are the factors that give rise to this issue?
Effect - what happened as a result? Can you link the usability issue (cause) to the gameplay or player experience issue (effect)?
Suggestions - did the player mention anything that might lead the team to a better solution? It’s possible the player mentioned how they expected a system to work, a possible solution they think might work, or even another game which does it well, such information might assist the team in working towards a re-design. Alternatively, the team may have mentioned a possible solution during the playtest that could be captured here.
It’s worth noting that the team will likely be holding separate design sessions to resolve issues from the playtest, you don’t need to find solutions right now, but this information might factor into the re-designs.
Image - a picture of the design element being discussed (to avoid ambiguity). Highlight a section of the image to identify the element if it doesn’t have a name.
Player quote - to help bring the findings to life you may want to include a direct quote from the player that came out of the player interview.
So in summary, when reporting the findings section each page could contain
Hypothesis description - what was being explored.
Hypotheses outcome - pass/fail.
Severity of the issue - High, Medium, Low etc.
Description of the issue - what happened, why, effect.
Suggestions - did the player or team have any thoughts which could guide a potential solution?
Image - can you use an image to show what is being discussed?
Player quote - any comment from the player that brings the issues to life?
You don’t need to mention how many players had an issue. Usability tests are about exploring the range of possible issues that could exist and the effects they have.
Method and Participants
A method section is a brief summary of how the playtest was run. This might include:
Methodology - the methodology section is a brief summary of how you designed the playtest. You could include:
Methods used - player observation and interview.
Number of participants.
Playtime - the length of each gameplay session
Build of the game - which version of the game did they play?
Device/platform - PC, console, mobile device etc
Participant Details - Relevant details of the participants in the playtest. The most important variable is the games they are playing or have played (behaviour). This helps give confidence to the research results, i.e. it’s not the players that are at fault, it’s the design.
Staff involved
You should mention the main author(s) of the report along with their contact details as other team members may want to ask follow-up questions. In addition, it could be useful to list all team members who were present in the observation and interview phases.
Of course, this is a guide. If you feel you don’t need to report this much information you can adjust as you see fit. Bear in mind, if someone who was not present at the playtest reads the report, does it contain enough information which they could learn from? It may not take much more effort to write a report which could have longer-lasting value.
Your report is now done, you have captured all the main findings and now it’s time to let the wider team know what happened. It’s time to share knowledge.
Key Takeaway
The playtest report is a valuable way of sharing knowledge with many others about what happened in a playtest. It also helps with team alignment by creating one shared account of where the game is at.