How to Save Millions With Marketability Testing

Do I need pictures or videos? Well, both, but while pictures are good, video is what attracts mobile gamers the most. The video ad format is considerably superior to photos when it comes to advertising mobile games. It is because your future users want to get a feel for a game before downloading it and learning how to play it. It’s difficult to capture that in a photograph.

And don’t forget to make more than one of each. I know it may feel overwhelming for a starter, but keep in mind that in different campaigns for different users, different creatives will work. And some of them will never work as you planned. Be ready to evaluate and make changes.

The main reasons to run a concept test:

• to determine the game’s IPM and marketability 
• how marketable an established game style is.

The app store platforms are becoming more competitive every day. The best way to gain an advantage in this saturated space is to more accurately identify which concepts will appeal to the widest audience. Testing helps you make these important product development decisions using concrete data, and it plays a critical role in revealing how people make their own decisions and which messages convince them to install

Most developers think of the app stores as the final stage in their global launch plan. By bringing it back to the beginning of the process with concept testing, it gives developers a unique glimpse into how potential audiences will react, if there’s an opportunity for a new app/game, and if it even makes business sense to invest resources into the creative framework to constantly look to find ways to improve creative testing and store fronts.

Step 1: Market research

Market research is the process of looking at the game market through numbers, facts and opinions. It’s about finding the equilibrium between what the audience wants, what your teams want to build and what your teams actually can build, scale and operate. This step will give you ideas about what you can expect while making and operating your game. Thorough market research should also help you to identify the challenges you’ll face in production and marketing allowing you to mitigate them in advance. 

Researching key players in your game’s genre can give you some great insights into what your target audience values in a game. You need to do some homework and answer these questions about the games and the players in the genre you’re targeting:

• How are other games named? Are they memorable?
• How does an IP fit into the game genre?
  • For example, a game that is based on a well-known IP (for example Disney, Harry Potter, Marvel etc.) doesn’t guarantee success if game mechanics is poorly implemented or the game is lacking well-built progression elements 
  • e.g. Disney IP in the RPG game won’t probably match the expectations of the classic RPG target audience. Looking at the recent launches of Disney Sorcerer’s Arena, Looney Tunes™ World of Mayhem – Action RPG, etc.

• Which keywords are they ranking high for? 
• Who wants to play this kind of game? 
• What is the size of the potential audience? 
• Does this audience have significant disposable income? 
  • What is the potential RPI? (Revenue per install) 
• Are players competitive or social? 

Doing an internal analysis could prove to be very useful in getting your game in front of the right audience. Let’s talk more about how to do it.

Target audience

The game industry is booming like no other with thousands of new mobile games launched each day in the Appstore alone. When your own mobile game makes it to the game launch, it will have a lot of competitors fighting for the attention of the same players. 

The obvious strategy here is, of course, to try to make a game that appeals to your target audience and at the same time stands out from the crowd of similar games. But how do you know, who exactly is your target audience and what does it want? 

The usual way to identify your game’s target audience is to use game categories. If you are making an RPG game, you can check out the best performing RPG games in your market, analyze them and try to learn their strengths and weaknesses. It’s natural because you aim to focus on RPG-enthusiasts. Well, you can surely have some success with that logic, but you are also severely narrowing your view and creating unnecessary blind spots. And these blind spots can cost you a lot of potential gamers and daily revenue. Games usually focus on primary and secondary target audiences. 

For example Candy Crush Saga – Candy Crush is a casual browser/mobile Match 3 game. The audience of Candy Crush consists mostly of female gamers and also much older gamers. These gamers focus on Completion (collecting and finishing everything) and very low on almost every other motivation factor. So the primary focus on middle-age female audience that is interested in match-3 genre and cares mostly about achievement over all other aspects of the game, but also aim to entertain male audiences with hard puzzles in the later stage of the game. 

The traditional way of categorizing games has been around for a while and we have grown to accept it. Puzzle games, strategy games, RPG’s, simulation games and the list goes on. 

Game developers often use terms “hypercasual”, “casual”, “mid-core” and “hardcore” to define the nature and focus group of their games.

Determining the target audience

Determining the target audience is key to reaching the loyal and high-profit players, in order to ensure the ROI (return on investment). To effectively determine the games’s target audience, game developers should consider the three main general aspects of target audience grouping: demographics, psychographics and motivation.

Demographic information

Demographic information involves statistical aspects of consumers such as gender, ethnicity, income, qualification and marital status. Demographic information is important because it gives a basic background of the players. This helps game developers to judge on a basic level how to communicate effectively with who they have identified as the target audience. Demographics are key because they provide the foundation of who the game will be targeting.

Psychographic information

Psychographics is the use of sociological, psychological and anthropological factors, as well as consumer behavior, style of living and self-concept to determine how different market segment groups make decisions about a philosophy, person or product. Psychographic information can be utilized by the developers to gain a deeper understanding of the player segments they intend to target, by analyzing the more intimate details of the player’s lifestyle and thinking processes so as to gain an understanding of their preferences. Things like financials, interests, hobbies, and lifestyle will all be filtered to create a target audience that will, in theory, be open to play the game and eventually make a purchase.

Players motivation information

Players play video games for different reasons. Most gamers, from their own gameplay experiences and playing with others online, have some taxonomy of some of these gameplay preferences. Over the past two decades, academic researchers and game developers have proposed many models and frameworks to codify these differences. There is certainly no shortage of models that describe players (from Bartle’s well-known Player Types to LeBlanc’s Aesthetics, from Lazzaro’s Fun Types to Sherry’s Gaming Uses & Gratifications)

Some models have 3 factors, while others have more than a dozen. To the degree that gameplay motivations cluster together, how many core motivation factors are there?

How can we check whether some motivations actually cluster together or are entirely independent? Are Collaboration and Competition related? Are they part of the same motivation group or belong in different groups?

And how do we find a reliable way of measuring these motivations to build knowledge around them?

Despite the large number of mentioned models, there has been a relative lack of quantitative data backing up most of those models. From a statistical point of view, the following issues have typically not been addressed quantitatively. Quantic Foundry developed, based on the motivation definitions or existing survey scales, brainstormed a few items for each of the motivations. For example, for Challenge: “take the time to practice and master a game” and “play the game at the highest difficulty level”.

A basic example of a player profile is: LAURA, female aged 35–40 who live in the US and have a university-level education (demographic), are a sociable extrovert from a top-middle economic class and live an active lifestyle, plays puzzle & adventure games (psychographic), lives in Nashville, Tennessee (geographic) and makes small and frequent purchases without considering the brand (behavioural). She desires to complete every mission, get every collectible, and discover hidden things.

There are many methods of demographic, psychographic, geographic and behavioral data collection. There are quantitative methods, being statistical processes such as surveys and questionnaires, and qualitative methods, being in-depth approaches such as focus groups or comprehensive interviews. There are several companies that specialize in target audience research – e.g. mentioned Quantic Foundry or 12triats. For the game developers that are not able to work with these companies, let’s talk about a way to squeeze the most out of the free tools.

How to define your target audience with free tools

1. Create a list of 3-5 competitors and prepare a spreadsheet to collect their demographics and interests data

Who are your game’s 3-5 biggest competitors? We are looking here to match your game with other games in the same genre, with the same mechanics, done in the same style or art direction, and that have similar pricing or business models.

Spreadsheet is done, we need to find out more about the competitor’s demographics, interests and behaviors.

2. Mine and collect the demographics and interests of those 3 games in your spreadsheet

Demographics are the age, gender, and race of players. Interests are, well, categories or things they’re interested in. And behaviors are activities they do either online or offline.

For example, again using the above mentioned Candy Crush Saga illustration, and having already talked about their demographics, their interests would be things such as other games by King, women’s health brands, and the Hallmark Channel. And their behaviors would be things such as dining out, buying pet products, and clicking mobile ads.

This can be pulled out directly from Facebook Advertising Audience Insights. Log into your Facebook account and head on over to the dashboard for Facebook Advertising

Once you arrive on the Audience Insights page, you’ll notice the general demographics and interests for all of Facebook have already been pre-populated. But we want to narrow that data down by interest. On the left-hand side of the screen you’ll notice a section appropriately named, “Interests”. Start typing “Candy Crush Saga” in that input box

3. Analyze the data, identify the overlaps, and draft a target audience profile for your game

Now that your work is done, it’s time to start analyzing your data for overlaps. You should be able to pretty much eyeball it, especially if you were able to find 3 or 5 solid competing games. The overlap should be pretty clear. However, if the game you’re making is a more unique game, or unlike much that’s out there, then connecting the dots with your data is probably going to be more difficult.

It can be easy to think that people who play games belong to one of two groups – hardcore gamers living and breathing complex highly skilled games playing for 8 hours a day, and casual gamers playing low skill, non challenging games, playing them for 20 minutes a day. These extreme cases are also associated with a range of demographic assumptions and stereotypes. Like hardcore gamers being young and male, and casual gamers being old and female. The reality is that a majority of players fall somewhere between these two extremes on the game genre spectrum.

Take player differences into account when designing your game or adding new features:

Be specific: Is this feature for a certain segment of players, or all players in general?

Be informed: Get feedback from the right segments of players.

Be concrete: Articulate how a feature will change specific aspects of these players’ experience, and why that is important to them. 

For example – Golf Clash added the competitive PvP Meta from Clash Royale into the sports genre. This worked out because it was targeting competitive older (30-50 years old) males, which overlapped with the feature of Clash Royale.

Step 2: Concept testing

Most developers know how to test concepts and do this early on to mitigate the risk of the investment. Since concept testing is extremely cost effective, there is no reason for other developers not to do it as well. The app store platforms are becoming more competitive every day. The best way to gain an advantage in this saturated space is to more accurately identify which concepts will appeal to the widest audience. Concept testing helps you make these important product development decisions using concrete data, and it plays a critical role in revealing how people make their own decisions and which messages convince them to install

Let’s take a look at Pokémon Go. Pokémon Go is a great example of something fresh and original utilizing a great IP. Location-based games existed prior to Pokémon Go, but the idea of going out into the real world to “catch ’em all” is the realization of every Pokémon fan’s dream. And the market data expresses that this was a perfect fit. To date, Pokémon Go remains one of the highest-grossing games on mobile.Google unwittingly planted the seed for Pokemon Go in the year 2016 in one of the many April Fools’ Day jokes the internet company is famous for. This could be understood to some degree as a Pokemon Go concept testing. 

The other benefit of concept testing is its flexibility. To enhance your learnings, you can  use surveys. To obtain player feedback you can create a redirect on the test app store page that directs people to a survey. These questions can cover game mechanics, characters, feature set, core loop, monetisation, spending behaviour, how often players play games and more. These survey questions are invaluable to:

• Find the most loved (or hated) characters
• Identify aspects or features that could drive the best monetization and return on investment 
• Pinpoint the most liked (or disliked) features

That way, you can ensure every detail is perfect before making it available to your target audience.

Concept Testing survey

Concept testing surveys are designed to record how players feel about your ideas. They provide your business with useful pieces of feedback that explain what players like and dislike about your concept. When creating one of your own surveys, your team should consider the tips below for building an effective concept testing survey.

1. Establish a goal

If you’re having trouble coming up with questions, it often helps to establish an overall goal for your survey. Think about the purpose of the test and the specific details you want to learn  from respondents. This will help you create questions that are meaningful and provide unique insights into the players’ perspective.

2. Create a consistent structure

Questions that are related to each other should be grouped together on your survey. This creates a smoother flow making it easier to answer questions. Players won’t have to constantly shift their attention and can produce more detailed responses since they’re focused on one aspect of your idea.

3. Include Likert Scales

Likert Scales ask participants to rank their opinions on a five- or seven-point scale ranging from “strongly agree” to “strongly disagree.” This creates a consistent structure to your survey which makes it easier to analyze results. And, your team can use feedback collection tools with this type of data to automate their reporting. Plus, most people are familiar with Likert Scales making your survey user-friendly for participants.

4. Include demographic questions

Like with any other survey, you’ll want to know who’s participating and whether they fit your target audience. Just because a participant says they disliked your concept doesn’t mean your idea is bad. That participant may not be your ideal customer and therefore, isn’t interested in your offer. This makes it important to include demographic survey questions to determine if your idea will be effective with your customer base.

BONUS TIP

Try to be neutral and don’t fall for the confirmation bias, as this is the downfall of most of the surveys. Even a small change in a question’s wording can affect how people search through available information, and hence the conclusions they reach.

Suggestive question example

What did you like about the game’s tutorial? 

• This is a suggestive question, you will get positive answers

How did you feel about the game’s tutorial?

• This is a neutral question without any suggestions. 

Concept testing survey basic example

• Which types of games are your favourite?

1. 1. 1. Simulation 
      2. Strategy
      3. Action
      4. Arcade
      5. Puzzle
      6. Other (please type genre here)

• Which of the following Train or Rail related hobbies do you have?

1. 1. 1. Collect models
      2. Ride trains for pleasure
      3. Learn the history
      4. Build/Fix/Make Trains
      5. Collect Memorabilia
      6. Take pictures or videos of Trains

• If you have a job in a train company, what would it be?

1. 1. 1. Train driver
      2. Station manager
      3. Train architect
      4. Ticket collector
      5. I wouldn’t like a job in a train company

• In a train related game, history accuracy is important to you.

1. 1. 1. Strongly disagree
      2. Disagree
      3. Neutral
      4. Agree
      5. Strongly agree

• Do many of your friends play Train related games? 

1. 1. 1. No, not at all
      2. Yes, we play co-op together
      3. Yes, but we don’t play together
      4. Yes, we compete with each other

• What is the biggest amount of money you have spent in a mobile game?

1. 1. 1. $0
      2. $1 – 9
      3. $10 – 49
      4. $50 – 99
      5. $100+

• Are you Female or Male?

• How old are you?

How? Process!

Concept testing is a process that almost any business can perform. Let’s discuss some of the ways you can use it to evaluate ideas at your company. Not only in mobile games! 

Example of concept test static banners on Facebook

When testing a theme, try to keep everything else in the picture as close as possible. Notice how each image below has characters in a similar grouping. 

There are many different styles of art: pixel art, low poly, cartoon, serious and more. The game’s art style tells the player a lot about the game. When someone sees Bugs Bunny for the first time; they can tell by the art that the show is intended for kids. It’s important to think about your target audience when considering which art styles to test.

As a general rule, the more casual the game looks, the cheaper it is to acquire users. Just take a look at the hypercasual market! So even within the 4 main categories (hypercasual, casual, mid core & hardcore) there is still a huge range! CPI testing is much better than CTR testing to determine a mobile game’s marketability. 

Speaking about marketability, IPM, or the number of installs generated for every thousand impressions, is a lesser known metric which is often overlooked by marketing teams. But in fact, it’s critical for UA managers looking to evaluate the performance. IPM speaks to the market’s response to a game. For example, if a user acquisition campaign for Game No.1 generates an IPM of 3.5 and a campaign for Game No.2 generates an IPM of 25, we can conclude from this result that your target audience prefers Game 2. This exercise will give you an idea about how the art style/theme resonates with your target audience. This exercise will not tell you how your game will monetise and what your ROAS are going to be. 

Using the winner of the first concept test, each subsequent test will help you determine a different aspect of your mobile game.

Ways to test

My personal preference is using Facebook ads to drive traffic for concept testing. It’s not only because of targeting! But to be honest, you can target almost everybody on Facebook – both age and gender to be sure your art is appealing to your target audience. If you run a test without targeting and get great CTRs on some super awesome looking art, but the game is meant to be for males, then your test results aren’t very helpful. 

Example:

When I worked on TrainStation 2 at Pixelfederation, we tested several visual themes and art styles even before development had started. A sequel to the company oldtimer game TrainStation. 

Target audience – US, Male 35+, train enthusiasts

Concept testing lets studios test out any idea before releasing it to their customer base. While you might think your idea is incredible at the end of the day it’s your players that decide if it is or not. If they don’t like it during testing, they’re not going to like it in mass production. By testing the idea beforehand, your company can avoid launching ineffective and unsuccessful mobile games. Let’s take a look at how you find out if the concept is not resonating well with your target audience

If you already have a game you can get the real Cost Per Installs (CPI), along with Click Through Rate (CTR) & Conversion Rate (CVR). If you don’t have a game you can only get CTR. To be clear, the main KPI is Click Through Rate, because Conversion Rate can be influenced and improved by proper ASO (strategy).

CPI = CPM/IPM

IPM = CTR x CVR

IPM = Number of Installs per 1000 impressions

CVR = Conversion rate (Number of players visited the Appstore and installed the game)

CTR Testing

CTR lets you identify the general interest of your target audience in your game.

• Pick what you want to test (theme, style, palette, characters, etc.).
• Create 2-4 static images that are very different, but focus only on the element you want to test.
• Create ads on Facebook using your creatives. Give them the same copy and send them to the same app (make sure to use an app that doesn’t bias one over another).
• Allow all the ads to get around the same number of impressions and enough to be statistically significant using Split testing tool by Facebook

The most important part of the test is the range of confidence for your results! The smaller the difference is between art assets, the more users you will need in order for the test to be reliable.

CPI Test

CPI gives you a deeper read into how marketable a game is. You can gauge general interest (CTR) as well as how likely people are to actually install (CVR) and most importantly, how much it will cost you to acquire users (CPI).

• Pick what you want to test.
• Create 2-4 static images that are very different, but focus only on the element you want to test.
• Create ads on Facebook using your creatives. Give them the same copy and send them to your app.
• Allow all the ads to get around the same number of clicks, enough to be statistically significant (see above).

It’s important to note that for statistical significance for CPI, your goal is the number of clicks, whereas for CTR it’s the number of impressions.

What to do when you don’t have a game ready, but you need CPI’s?

You can try to calculate the “predicted CPI” (pCPI) with the numbers you already have: 

Clicks x CVR = #installs

Spend/# installs = pCPI

But is this true? You need to have in mind that you run a campaign that was optimised on clicks, not installs! Therefore the real CPI will be different. I saw 2x-2.5x times higher real CPI, when compared to predicted CPI on several concept tests. 

pCPI = $0.28

rCPI = $0.57

In a system where CPIs are low, games can grow fast. If you’re able to spend $1 and make $1.50 back then you should just keep spending more money and grow faster.

Different genres of mobile games have different monetization profiles. Games with the highest monetization profiles are able to spend $50+ per user because of their high LTV. 

Let’s look at Hypercasual genre as an example. It works well because they can decrease the CPI to incredibly low levels. However, the monetization model relies on users watching and engaging with advertising. Getting the balance between a very low CPI while maximising the Advertising spend is where the profit is.

Concept testing test store page specs

App icon

1024×1024

Full App description

5 screenshots each including descriptions of the game’s mechanics/features. Depending on the game format (landscape/portrait)

– 1125 x 2436 pixels (portrait)

– 2436 x 1125 pixels (landscape)

Create at least 3 Reviews of the Game each with 4 or 5 stars 

Reviewer Names – avoid names like John Doe, use real names!

Total Star Rating will range from 4.4-4.8

Send the Mailchimp Code Snippet for email signup (after player clicks on Install button)

Tracking metrics via Google Analytics or Facebook pixel tracking.

Concept testing marketing ad specs

Static Banners

Resolution – 1200 x 628 pixel and 1080×1080

Ratio – 1.91:1 and Square 1:1

Ideally no text in banners

Copytext:

Text – 90 characters

Headline – 25 characters

Link description– 30 character

Video 

Format – .MOV or .MP4

Ratio — 16:9

Resolution – 720p at minimum

File size – 2.3 GB maximum

Ideal length – max 30 sec.

Copytext:

Text – 90 characters

Headline – 25 characters

Link description– 30 character

Main takeaways

Every player is unique and special, with their own motivations for playing any game and developing a personalized approach to its ecosystem. However, it’s nearly impossible to assess and cater to each type of personality with every aspect of your game, so it’s necessary to step back, organize, and plan things out a bit. And that starts by understanding what you’re dealing with in regard to your target audience.

There are many methods of demographic, psychographic, geographic and behavioral data collection. There are quantitative methods, being statistical processes such as surveys and questionnaires, and qualitative methods, being in-depth approaches such as focus groups or comprehensive interviews. There are several companies that specialize in target audience research. For the game developers that are not able to work with these companies, there are ways to squeeze the most out of the free tools.

Most developers know how to test concepts and do this early on to mitigate the risk of the investment. Since concept testing is extremely cost effective, there is no reason for other developers not to do it as well. Going forward, all developers must consider marketability early in the shipping process, and not only after the game is given the green light. Marketability answers the simple question: does the market want to play my game? The app store platforms are becoming more competitive every day. The best way to gain an advantage in this saturated space is to more accurately identify which concepts will appeal to the widest audience. 

Marketability testing helps you make these important product development decisions using concrete data, and it plays a critical role in revealing how people make their own decisions and which messages convince them to install.