The craft of incentive prize design Getting started with incentive prizes

Getting started with incentive prizes

Public, private, and philanthropic leaders are wrestling with technological, economic, environmental, and societal problems that seem to get more complex each day. Public leaders, moreover, must consider these multifaceted problems with limited resources that often prevent them from developing innovative solutions quickly and effectively. This is why government leaders in particular are turning to incentive prizes to advance their missions through incentives and the ingenuity of the crowd. Leaders who use prizes effectively take a strategic approach. They work with colleagues, partners, and subject-matter experts to carefully select and define problems likely to be solvable through prizes. They collaborate with stakeholders inside and outside their organizations to determine the outcomes they wish to achieve—and then use those decisions to drive a prize design process that will yield specific outputs. Because public organizations must adhere to specific legal requirements, government leaders determine what legal authority will allow them to achieve their desired outcomes. These leaders publicize the challenge, its requirements, and its results in language that will resonate with the audiences they seek to engage. Finally, to realize the full benefits of the prize, leaders initiate legacy activities to provide resources and support to the prize participants who remain engaged after the challenge comes to a conclusion.

Problem definition

Because problem definition involves grappling with a great deal of ambiguity, it is arguably the most difficult part of prize design. It sounds deceptively simple: What problem should the challenge address? Answering that question, however, requires clarity about the outcomes sought and the ways to achieve them as well as a specific problem statement that succinctly describes the fundamental difficulty to be overcome. Designers often initiate these definitional discussions with a diversity of internal and external experts and stakeholders, because they can bring valuable perspectives and ultimately need to be aligned around the final problem statement.

To manage the ambiguity of problem definition, designers often start by developing a clear understanding of the outcomes they seek and the different ways they can achieve them. Because prize design varies, sometimes dramatically, depending upon the outcomes selected, careful definition of these outcomes is critical. These early-stage problem definition discussions help to establish the causal and logical linkages between the specific difficulty to be addressed and the outcomes selected. They help to surface the kinds of challenges (for example, incentive prize, grant, investment) that are best suited to address the problem. These discussions reveal ways in which the designer’s organization may or may not have the legal authorities, resources, skills, and capabilities to address certain facets of its own problems. Finally, by refining their understanding of the outcomes sought and ways to achieve them, designers can explore whether a prize is likely to produce results more effectively than other possible approaches.

A problem that requires years of work to solve or specialized facilities or high capital expenditure may not fit well with certain target participant groups.

Outcome specification establishes the broad set of aspirations, whereas problem statement definition more narrowly frames the need that the prize will ultimately address. Developing a problem statement helps designers craft a need that is not too hard (because no one will win the prize) and not too easy (because the prize will be won too quickly and not necessarily with the optimum solution). Prizes need a problem statement that will be attractive to a broad selection of potential competitors (because greater diversity can lead to more innovative solutions), but not too broad (because an overly broad net can erode submission quality). And, the problem statement must describe a challenge whose scope is appropriate for the types of participants sought: A problem that requires years of work to solve or specialized facilities or high capital expenditure may not fit well with certain target participant groups.

Making these decisions often requires tapping into different types of expertise and devoting a considerable amount of staff time, depending upon the complexity of the problem. Technical experts can be valuable for grappling with the science and technology underlying the problem. Academics and industry representatives can be highly useful for evaluating the time, expertise, and expense needed to solve certain kinds of problems. Designers and strategic thinkers can help refine and reframe problems in ways that are conducive to prize-based solutions. Finally, a gifted facilitator can help to ensure that these different types of professionals have the right conversations and make progress toward a workable problem statement.

All manner of problems may be amenable to prize-based solutions, if defined properly. Consider, for example, the range of problems defined by USAID for its Tech Challenge for Atrocity Prevention. For one of the five components of this challenge, USAID defined the “problem” as third parties who enable or contribute to genocide, consciously or inadvertently. To solve this problem, they sought technologies and innovations that “identify, spotlight, and deter” these enablers. For another component of the prize, USAID identified the unpredictability of genocide as the problem. This led the agency to seek algorithms that could forecast potential hot spots based on socio-political indicators and historical trends.¹

According to Jason Crusan, who directs the National Aeronautics and Space Administration’s (NASA) Center of Excellence for Collaborative Innovation (CoECI), problem definition entails “hav[ing] to deconstruct the problem into bite-sized pieces, and abstract[ing] [each] to understand how it’s just one piece of the larger puzzle.”² Indeed, it can take up to a year to wrestle with problem definition.³During this time, designers typically conduct a detailed landscape analysis, meeting with internal and external experts as well as partners to define and digest the scope of knowledge applicable to the problem and its surrounding issues. As designers begin to prioritize specific areas of the problem for research, they can also begin evaluating what combination of potential solutions may best achieve their desired outcomes.

An example from CoECI emphasizes this point. Every year, fraud, waste, and abuse in the health care industry accounts for hundreds of billions of dollars in losses.⁴ The US Centers for Medicare and Medicaid Services (CMS) wanted to apply new tools to their ongoing efforts to address this challenge. The agency partnered with CoECI, the state of Minnesota, Harvard Business School, and TopCoder to find a more efficient and effective way to help states spot medicaid fraud.

Given the challenges associated with identifying fraud, the partners took time to define the problem, which focused on how current software systems could not effectively screen risk scoring, validate credentials, authenticate identities, or sanction checks. To tackle this problem, they launched the Provider Screening Innovator Challenge, which sought screening software that could help ensure that medicaid funds are not spent fraudulently.⁵ To make sure the overarching challenge would generate a workable solution, the design team broke it into four components and 124 separate challenges. As a result, the partners were able to obtain an ecosystem of solutions based on submissions from more than 1,600 participants from 39 countries. The software applications developed as a result of the challenge series are being compiled into an open-source solution for the state of Minnesota—and perhaps the nation.⁶

Experienced prize designers have learned that incentive prizes are not appropriate for every type of problem and are not a silver bullet even for the right problems.

Push versus pull: Is a prize appropriate?

Problem definition discussions inevitably raise important questions about which approach—a challenge, a prize or some other mechanism—can generate the best solutions. Experienced prize designers have learned that incentive prizes are not appropriate for every type of problem and are not a silver bullet even for the right problems. One valuable way to navigate this strategic choice is to consider the distinction between “push” and “pull” mechanisms, a reference to how different types of rewards, placed at different points in a solution development process, can create unique incentives.

• Push mechanisms include traditional grants and contracts, such as fixed price or time and materials contracts or research and development grants. These provide vendors or grant recipients with payments or incentives to develop and deliver specific services or technologies, in effect paying for the effort involved, but leaving the risk that the product may not meet expectations. Push mechanisms can be used to generate a range of outputs, from purchasing services or technologies that are well-understood to supporting early-stage research and development efforts that have uncertain outputs.
• Pull mechanisms, including incentive prizes, reward participants not for their efforts per se, but for their outputs, such as ideas, prototypes, pilots, or commercial products and services. Leaders use pull mechanisms to encourage participants to experiment with innovative and, sometimes, risky approaches, while paying only for results that meet predetermined rules or specifications. For some pull mechanism prizes, if no one wins, the sponsoring organization is responsible for only its administrative costs.

When should you not use a prize?

Prizes cannot solve every type of problem. Here are a few considerations:

Prizes should not be used when there is a clear, established, effective approach to solve a problem. 

A prize’s strength comes from its ability to incent participants to create novel solutions. Using a prize to create solutions already available in mature markets may simply waste participants’ efforts.⁷

Prizes should not be used when potential participants are unwilling or unable to dedicate time and resources to solve the problem. 

For instance, as appealing as start-up companies may be as prize participants, they are rarely able to shift their commercial focus to a challenge. Prize designers need to understand the risk tolerance and capabilities of their potential participants before committing to the use of a prize that requires their engagement to be successful.

Prizes should not be used when there are only a limited number of participants who can address the problem. 

If the universe of participants is small and known, then a prize may not be necessary. Instead, leaders should use other types of challenges, such as “pay for performance” approaches that issue grants or contracts with milestone-based payment terms. One example of this approach is NASA’s Commercial Orbital Transportation Services program in which industry agreements with certain companies provide for fixed-price payments only when performance milestones are met.⁸

Experienced designers often combine prizes with push mechanisms to achieve their goals more quickly and effectively. For example, in 2013 the Army Research Laboratory ran five prizes that successfully identified new methods for generating energy from a walking hiker and new ways to produce potable water for humanitarian missions. The winning solutions came from individuals from around the globe—many of whom would have not had the opportunity to work with the army through other means. The Army Research Laboratory plans to continue developing these ideas through traditional push mechanisms such as testing at laboratory facilities and future small business funding opportunities.⁹

Despite the fact that extensive consideration may be required to determine the suitability of a prize, this preparatory requirement has not put a damper on experimentation in the past five years. Many agencies, such as NASA, embrace prizes and translate their growing confidence and experience into policies that codify and explain their problem-solving strategies.¹⁰ The White House Office of Science and Technology Policy provides annual progress reports on prize competitions offered by federal agencies, and the Office of Management and Budget offers detailed legal guidance to prize designers. This work can be immensely helpful for less experienced organizations considering similar approaches.

Most experienced designers consider prizes to be just one important problem-solving approach in a larger portfolio that includes challenges and other, traditional approaches as well. In some cases, for example, NASA program managers have folded challenge outputs into grants or in-house R&D efforts. In other cases, the agency uses traditional contract arrangements to implement designs solicited from prizes. NASA’s designers view push and pull mechanisms not in isolation, but in varying combinations custom-designed to achieve their desired outcomes.¹¹

Evaluating legal options

Public sector leaders can’t simply design and execute a prize without first evaluating their legal authority to do so, particularly when it involves paying cash to winners. US government prize designers, in particular, must look carefully at the legal constraints they face. Typically, this involves early liaison with general counsel to avoid unwelcome surprises. For federal agencies, several laws can affect incentive prizes. The most well-known is the America COMPETES Reauthorization Act of 2010, which provided broad authority for every federal agency to conduct prizes in the service of their missions. America COMPETES created a clear, simple legal path for using these tools and complemented other pre-existing agency-specific prize authorities.¹² One key aspect of the prize authority provided by America COMPETES is that federal agencies are able to co-fund prizes (both the prize purse and administration costs) with other agencies as well as private sector and philanthropic organizations.¹³

 Perceptions of faulty evaluation criteria or unfair judging procedures can lead participants to take legal action, especially if the stakes are high.

In 2010, the Office of Management and Budget issued guidance on various legal authorities and provisions, intellectual property considerations, and other issues affecting prizes in a memorandum called “Guidance on the use of challenges and prizes to promote open government.” This memorandum provides prize designers and their legal counsel with a useful starting point for developing their own legal strategies.¹⁴

Building a legal strategy applies to state and city prizes as well, because legal requirements must be considered in light of desired outcomes. For example, designers of the New York City Big Apps Challenge intended to spur the development of tech businesses and therefore opted to let participants retain the intellectual property rights of the apps they created.¹⁵

The conclusion of a prize also poses legal considerations that should be addressed early in the design phase. Perceptions of faulty evaluation criteria or unfair judging procedures can lead participants to take legal action, especially if the stakes are high. Committing to the transparency of the judging process and ensuring that participants can view scoring and selection criteria when they register for the prize can ameliorate such issues.

In the federal context, the Government Accountability Office recently ruled that it did not possess the legal authority to adjudicate a dispute related to a prize offered by the Federal Trade Commission, despite its well-established ability to do so for contracts.¹⁶ This ruling raises important questions about how the federal government will handle prize-related conflicts in the future.¹⁷ It also underscores how important it is for prize designers to build prizes that are highly transparent, with independent judging panels and, for worst-case scenarios, conflict resolution processes.

After reviewing these considerations and engaging in an iterative problem definition process, designers will be ready to begin building a prize.

Read the full report on The craft of incentive prize design.