Do you have experience with validation of health games or apps? What do you think about the need for validation and methods of validation in this sector? Read on to fill out the survey on validation of health games.

This survey is part of the wrap-up of Growing Games, a support program (2013-2016) to promote the sustainable growth of the Dutch applied games sector. This survey is the next step after the Growing Games consortium published a position paper at the end of 2015 on the state of validation of health games and apps in the Netherlands. That paper called for a more differentiated approach to validation befitting the highly different characteristics of these innovative products.

To further develop a critical view and build upon the position paper, it is crucial to hear what people with recent experience validating health games and apps think about the need for validation and appropriate methods of validation in this sector.

>>> Go to the Validation of Health Games Survey <<<

And remember: it doesn’t matter whether your experience was good or bad, or at a design, research, insurance or governmental organization, or about a game or app for prevention, cure or care; it is greatly appreciated if you share any experience you have had!

Feel free to spread the word in your network, and have others fill out the survey too.


(Image: Coach4Life – Little Chicken Game Company)


Dutch gaming companies are natural allies of researchers and knowledge centers in several fields of interest. Research is a given for applied games, but entertainment games aren’t lacking academic attention either.

Games as a medium have been the subject of research ever since the birth of commercial video games in the 1970s. Academics have published entire libraries’ worth of materials about all possible aspects of games and gaming. Whether it’s about the working of games, their cultural value or technological aspects, gaming is well-represented in academic research.

It should therefore come as no surprise that game companies themselves often have close ties to knowledge institutes, education and individual researchers. The previously mentioned Gamesmonitor 2012 already showed that this is clearly the case: interviews with 77 game companies showed that more than half of them collaborate with research institutes or education programs in the Netherlands, and that a minority has ties to institutes in the US or Europe. These results concern both studios building entertainment games and those involved in developing applied games. Game companies assume different roles in the collaboration, from knowledge receiver or partner for knowledge development to developer of research tools. That collaboration is often focused on education, but research also makes up a large chunk of it. In 2012, collaborations revolved around developing tools to better build games, as well as to develop business models and develop and scientifically validate game concepts.

The latter is mostly represented within the world of medical applied games. Health games have greater responsibility when it comes to proving that a game works the way the developer says it works. A rehabilitation game focused on the recovery of certain muscle groups should not cause excessive strain or train the wrong body parts. Similarly, a game that helps ADHD patients plan their day (see the example following this chapter) should have a proven effect outside the game – if you attain high scores but fail to plan better in daily life, the game is considered a failure.

This is just one of the many areas of research for applied games; think of how faithfully patients attend counseling sessions, skill development in training games, et cetera. For example, the Erasmus MC in Rotterdam collaborated with Amsterdam-based IJsfontein Interactive Media to develop what they dubbed the “abcdeSIM”. ER doctors need to make split-second decisions about which patients to treat first. To do so, they use the ABCDE method: check the patient’s Airway, whether they’re Breathing, and work their way down the alphabet. They use a playful app to train that skill to make quick decisions, but said app has such a proven effect that the hospital has accredited it as an official training method.

This shows how game design can significantly profit from research: it validates the effect and helps convince clients, investors and users. With advantages like that, you think that research is a standard component for medical games, but you’d be mistaken. There is a gap between the wish to have games researched and actually having research done, and that forms a bit of a problem. Building games is a process in which new insights are immediately added to a product; developers work from version to version, and learn from testing between rounds.

That process is at odds with science’s preferred way of studying a research object: as a fixed value, to be observed over a longer period of time. Because of financing hurdles, that “longer period of time” is usually four years – in other words, an eternity in the world of software development. The beta version of any game would be obsolete by the time it would be finished; waiting four years for the study results before being able to build version 2.0 – and waiting another four years for validation research – would be undoable.

While there is a solution to that problem, it requires researchers to start thinking in one-year projects. It’s possible, but that in turn means something in the organizational structure of research institutes needs to change. Another option is to help researchers tag along with game studios during the development process while using more evidence-based research methods. It may not be as prestigious in the world of science, but it would be of great help to game validation. Regardless, the games industry really wants to come to a solution. While we know that many applied games work, we don’t always know why and how they work; something for which scientific help would be greatly appreciated.

Fortunately, science is there to lend a hand. For example, NWO, the Dutch government organization responsible for financing research, is investing in a study on persuasive game design for online treatments in addiction care. How can you develop games in such a way that they stimulate people to change their behavior – preferably for good? Although Ranj Serious Games is involved in that study (as are the Delft University of Technology and the University of Amsterdam), it’s not specifically tied to a single game. This study is not to be confused with research on game addiction such as is being done at the University of Amsterdam; those are effect studies originating from within the division of social sciences.

Similar research is being done at the University of Twente, where in-game characters are taught intelligent “human” behavior, allowing police officers to improve their social skills within a game environment. LOITER, developed especially for that purpose, features “virtual suspects” who react more strongly than in-game characters usually would. Incidentally, the University of Twente isn’t the only university studying the artificial intelligence of non-playable characters; the University of Tilburg is currently focusing on the same subject.

This type of research revolves around games-related computer technology, an area in which the Netherlands is starting to build quite the reputation. Researchers at Utrecht University study digital simulations of crowds: how do you make a group of avatars look realistic while moving? Other researchers focus on the space the crowds move through: automated game design is all about intelligent software generating levels by itself, thereby decreasing the costs for game development.

Meanwhile, TU Delft researchers focused purely on computer graphics and visualizations, combining research on human perception with software development. The premise is that by understanding how we see, games could be using less images per second. That means less computing would be necessary, which leaves game developers with more space to use for other features. Delft has built several of these software packages, which have proven popular with game engine owners: their technology is featured in the Unreal engine as well as Unity.

If you look at it in broad strokes, it would seem that the collaboration between Dutch industry and science is going well. That’s true for the most part, but there are still things to be improved. Research funding is often granted to the validation of applied games and applicable technological research. While that’s not a bad thing in itself, there is an obvious lack of attention when it comes to researching the economic and cultural role videogames play in our society. Who are these gamers, exactly? Why do we play games? And, equally important – especially for a Dutch industry that still has to work on this – how is money being made from games now and in the future?

Another important consideration is the fact that game companies are sometimes asked to contribute to the financing for certain research constructions. This can be a particular problem for smaller studios, who may have to choose between investing in research or in new computers. Some studios are so popular among researchers that they find themselves to be the subject of excessive demand, and have to turn down new projects.

Regardless of issues like these, knowledge institutes provide an important addition to the efforts of game studios. Structurally seeking out scientists and education facilities is a good way for everyone to maintain a fresh perspective. What has proven to work in this young industry, and why? In today’s society, games are not always seen as the interesting medium they are: if you ask the public, they’re still regarded as mere entertainment for teenage boys. However, this view could be changed through scientific validation of the effect of games, technological and scientific insights regarding hardware and software, and economically verifiable revenue models. Games are a young and innovative medium, and there’s lots of growth potential. In order to reach that potential and combat the lack of understanding in today’s society, collaboration with science is key.

Dutch companies earn money through games. The Netherlands has a sizable games industry, but what does it look like?

What is the Dutch games industry? If you ask the average gamer, they’ll probably mention Guerrilla Games, the studio behind first-person shooter franchise Killzone. If you press on, perhaps provide a few hints – Ridiculous Fishing? Awesomenauts? – they might name smaller studios like Vlambeer or Ronimo Games. But if you mention successful studios like WeirdBeard, Abbey Games, Codeglue or Spil Games, chances are most gamers won’t have a clue what you’re talking about.

Is that all? Or are we simply asking the wrong people? Although the abdominal surgeons at UMCG Hospital in Groningen may never have heard of Guerrilla, they’re very familiar with Grendel Games. After all, they train their surgical skills through playing Underground, an applied game developed especially for them. Fire chiefs responsible for operational procedure exercises will know E-Semble, a studio that supplies virtual training simulations. Finally, the dredgers at multinational Van Oord in Rotterdam have come to greatly appreciate V-Step, a company specializing in maritime simulations that help prepare dredgers for hands-on situations.

All in all, it’s safe to say that Dutch game studios aren’t equally well-known across the board. Still, there are about 330 companies currently building, publishing and selling games throughout the Netherlands. That number was taken from the Gamesmonitor 2012, the most recent in-depth study on the industry. However, that number doesn’t include a few dozen companies that don’t consider gaming their core activity, but that still develop games. For example, the Dutch army develops its own revalidation and training exercise games, and there are a number of ad agencies that employ in-house game programmers. And while they significantly add to the success of the industry as a whole, specialized freelancers aren’t included in that list of companies either.

In addition, there are several companies that support the industry, while not developing games themselves: server farms specializing in games (cloud gaming is experiencing significant growth), companies analyzing game marketing data, or academic institutions doing fundamental research on games and gaming.

Those companies that spent more than half their time building games, employed about 3,300 people in 2012. Although there hasn’t been an in-depth study on the industry since, that figure is now obsolete. There are 32 games education programs in the Netherlands, from which several hundreds of new game developers graduate every year. How many of them actually find a job in the industry remains unclear. We also know the annual industry turnover of 2012, which was somewhere between 150 and 225 million euros – again, not counting turnover for game-related parties like the Department of Defense and ad agencies.

It’s safe to say that the creative industry in the Netherlands is thriving, and that there’s money to be made in developing games. In that sense, the Netherlands is no different from other developed Western countries. But what does the Dutch games industry actually look like? There are a number of things that stand out. For such a small country, the industry is quite large; bigger countries like France and Japan have a comparable number of game studios. Perhaps even more striking is the fact that the Netherlands focuses more on so-called ‘serious games’ or ‘applied games’ than other countries do. On estimate, at least half of all Dutch games production consists of games that help people train, learn, and manage change processes. The other half consists of so-called ‘entertainment games’, which ­revolve around the experience of gaming itself. The next few chapters will deal with both of these forms in further detail.

Another thing that stands out is the small scale of most companies. There are about a dozen medium or large game studios in the Netherlands, of which Guerrilla Games is the largest, with 230 employees. However, most Dutch game studios are run by ten people at most. That makes us very different from the industries surrounding us: game studios in Germany, France and Great Britain are generally larger. Size does matter here: you need large teams to develop prominent entertainment titles for game consoles, also known as AAA games. It’s no wonder that Dutch game studios generally specialize in different areas: mobile games, small-scale PC games, browser games, et cetera.

Besides their relatively small size, Dutch studios are often quite young as well. While some companies have been around for over a decade, most studios haven’t reached that anniversary yet. Those newer studios often employ savvy twentysomethings who know that in these times of digital distribution – think Apple’s App Store, or Steam for PC – having one solid idea is all it takes to make it big. There are definite pros and cons to this approach. Like the results of the Gamesmonitor showed, the industry is quick to grow horizontally, if not vertically. In other words, while the number of Dutch studios increases, they don’t show a lot of company growth.

Is that a bad thing? That really depends on your point of view. A pessimist would look at the risks of all these young and eager companies: inexperienced startups entering a dynamic and extremely competitive market. An optimist would see small-scale, unorthodox companies with a creative and innovative approach to games and new media in general, exploring them in ways nobody thought of before. Take an innovative project like Story Wall by Groningen-based Studio Bleep, an app that helps kids in hospitals overcome their fear. Simply use your smartphone to scan a cow printed on the wallpaper, and it will start talking to you on-screen to set you at ease. Another example is entertainment game Bounden, developed by a game studio from Utrecht called GameOven. Two people dance while holding a single smart phone between them, and the game shows them exactly how to move. You won’t get away with simply swaying back and forth, either; the in-game choreographies were created by the Dutch National Ballet.

The Dutch games industry is very innovative, and that innovation extends beyond the games themselves. For example, as a Sony studio, Guerrilla helped develop high-tech hardware for Sony’s Playstation4 console. Software package Game­Maker, popular among game developers, was also developed by a Dutch studio. So was ( )distribute (programming language for do distribute), which helps companies safely share their games with journalists and gamers. Haarlem-based iQU builds analysis software that allows game companies to follow gamers digitally, often throughout several games. It’s a very specialist area, and they’re one of the international market leaders. And then there’s the more fundamental research on IT, artificial intelligence, and other game-related software issues that’s being done at several Dutch universities.

All in all, the Netherlands can boast a great knowledge structure when it comes to computers and IT, and the games industry both adds to that expertise and uses it. You’ll find more on that in the final chapter. A broad IT culture is another vital part. Dutch people are generally highly educated, and the Netherlands offers solid opportunities for technical education. In addition, most of the population owns smartphones, PCs and tablets, and we reap the benefits of a modern glass fiber network and multitude of radio masts for mobile traffic. The Netherlands is one of the most well-developed countries in terms of Internet penetration and use. Of course, this is largely due to AMS-IX, a large Internet hub connecting European and transatlantic cables. That hub is the reason why an American cloud hosting company like Softlayer came to the Netherlands in the first place.

In order to understand the Dutch industry, there are two other important areas besides technology that need to be addressed. First of all, the games industry is part of the creative industry, and as such meets the requirements for financial support under the so-called ‘top sector policy’: government policy meant to stimulate the knowledge economy. That funding is used to improve the creative infrastructure – stimulating a startup culture, partnering the games industry with the business world and academic institutions, or encouraging Research & Development for games.

Secondly, the culture within the industry itself shapes the form the industry takes on. With only 3,300 people, the industry is still very manageable, and its transparent culture means sharing knowledge and insights is no problem at all. Even the smaller studios are part of a well-formed industry network. New companies connect through startup structures like the Dutch Game Garden, an ‘incubator’ for smaller game companies that currently has three locations throughout the country. With its own industry magazine (Control Magazine), annual game awards, trade fairs and consumer events, the dialog between stakeholders remains dynamic as ever. In the end, that’s what makes the Dutch games industry so quintessentially Dutch: it’s a small country, and we need to help each other out wherever we can.

May 13th during a special Dutch Games Association (DGA) Day, organized in collaboration with CLICKNL|Games and Dutch Game Garden, a new online platform for the games industry was launched. Together with the launch of the book gamesNL and the premier of gamesNL the movie, this day marked the beginning of a new strategy for the DGA. The aim is to develop te Dutch games industry to a European top 3 position in the following years.

During three workshops and a lively panel-discussion with members from the games industry and the Dutch government, the main question of the day: ‘How to get to a European top 3 position?’, was discussed. The Dutch games industry is a vibrant industry that consists of people with a great passion for games and gaming and belongs to the European top already on a creative level. Entrepreneurship and business knowledge are getting more and more attention and The Netherlands, as a small country, is performing well in this respect. But the ambition of the DGA is clear: we want to rise above the size of our country and be part of the European top 3. DGA Chairman Horst Streck concludes the panel with a note on collaboration: “It is only through intensive collaboration and sharing knowledge, both inside the industry and from the industry to the outside world, the governement and other sectors, that this industry can grow. The online platform support a knowledge network, collaboration and knowledge exchange. The book and the documentary are there to inform people outside the industry in Holland and abroad.”

Investments: money, knowledge and experience

Simon Usiskin (co-founder iQU): ”The Netherlands have not had a big exit within the games industry which means that private investors don’t have an interest in the sector. On the other hand, because of the success of companies like Rovio and Supercell, the Finnish games industry is seen as an exciting sector for private venture capital. Successful entrepreneurs tend to help each other and invest in each other’s start-ups and reinvest in the next generation of games start-ups and the government agency supports the private investor throughout the process. But I also have to say: the Dutch must be more ambitious and have to develop a more risk-taking culture if we are going to be one of top 3 games countries in Europe; let alone the number one!”

Lennart Sas (Co-Owner Triumph Studios): “Because of Dutch labour laws it is not an attractive option to hire a lot of people. On top of that, our passion is to make great games, not to manage a large team.”

Michiel Sala (CEO Little Chicken): “I don’t see many people with an economical background like me coming to work in the games industry. It’s time for a special games MBA.”

How research can contribute to the support of growth for the industry

The importance of research in the process of supporting growth for the industry was addressed by Berni Good, founder of Cyberpsychologist Limited, in an inspiring keynote. She pointed out the importance of psychological knowledge about the behavior of the gamer in the development of both entertainment and applied games. Michiel Sala (CEO Little Chicken): “Scientific research is important for the development of our sector. I would like to see that researchers would conduct research on questions that are posed by companies, in stead of researchers deciding for themselves what to research.”

After de keynote and the panel-discussion, the book gamesNL was handed to Marjan Hammersma, Director General Legacy & Arts at the Ministry of Education, Culture and Science. After the book was officially handed to her, she also started gamesNL The Movie to conclude the day.