Designing Challenges

One of the key insights from the past 8 years of the Post-Industrial Design School project was understanding what made the $1K Challenge an effective teaching tool.

We ran the $1K Challenge mostly as part of the Entrepreneurial Design course at SVA IxD, but also at Orbital.  The most recent version of the challenge from our 2018 course was:

Design, launch and complete a crowdfunding campaign that benefits a community you’ve worked with over the course of the semester. The campaign should raise at least $1,000 from 50 different backers.

Breaking down the design of this challenge, there are a couple of key aspects:

• the monetary goal establishes a clear bar
• requiring 50 different backers ensures you're making an honest effort to appeal to a market
• leveraging Kickstarter, an all-or nothing crowdfunding platform, lets us benefit from both the constraints and capabilities of the platform as well as the limitless examples and open resources
• centering it around a community they've already gotten to know earlier in the semester (the $1K Challenge is preceded by two other "starter" challenges that help the student hone in on a community of interest) help de-risk this altogether
• that all of the students are pursuing a similar framework enables them to learn from each other while providing sufficient flexibility in order for these projects to feel like unique to them.
• that this all takes place in public where the "stakes are higher" than a simple homework task graded by the instructor helps establish the right context for the instructors to be coaches.  in effect, it aligns our interests.

My partner Christina Xu and I go into much more detail on these above points here.

What we didn't spend much time on, however, is a more fundamental point regarding the nature of challenges. That is, there are different flavors of challenges.

Here are a few that come to mind:

1. Challenges as Measurement. At a basic level, one example of a challenge is simply a test, like a university mid-term exam.  The point of taking the test is partially to prove what you know, but it's also a mechanism to facilitate the grading of a large number of students.  As an instructor, you need to make the test sufficiently difficult in order to create a wide enough distribution to establish bands for what is an A vs. an A- or a B+ and onward.  The intent behind this is a scalable measurement tool, and the incentive for the participant is to do whatever they need to get a good grade, or alternatively to tank it because they'd rather spend their time elsewhere.
2. Challenges as Incentive and Filter. Then, there are challenges that function more as prizes.  For example X Prize offers challenges in the form of monetary awards for people who are able to accomplish difficult tasks.  Netflix was a pioneer of this at scale: they offered a prize to anyone who could improve the performance of their recommendation algorithm.  Topcoder turned this whole idea into a company.  And, today it's customary for companies to offer bug bounties as a way to ensure the security of their software.  In all of these contexts, you're using challenges to incentivize the behavior towards an outcome you want to see, where the magnitude of the prize is sufficient enough to help you separate the few from the many.  What's interesting about the Netflix example is that many of the top competitors individually had solutions that didn't qualify for the reward.  However they were all incentivized to win the prize money so they ended up combining forces to produce a winning result.
3. Challenges as Catalysts.  Challenges are also often used to help catalyze new outcomes, where the prize is something that's significant enough to combat one's inertia, but where the majority of the participants gain value from just the experience itself.  For example, Startup Weekend brings together people who may not all know each other beforehand to build a startup in 54 hours, which may actually result in new companies as a bonus, but at a minimum help people form new relationships.  Asian American Film Lab's infamous 72 Hour Shootout similarly establishes sufficient constraints an incentives to get people to produce a short film in a compressed period of time.  Nanowrimo runs an annual novel writing challenge in November, where the goal is simply to write 50,000 words. The challenges are meant to catalyze action, and the constraints of the challenge help ensure that the work doesn't go off the rails.  After all, you have a finite amount of time to deliver.
4. Challenges as Onramp. Challenges can also be used as a teaching mechanism, most commonly as a discrete sequence of stair-step challenges that build in difficulty.  This is common in sports where you are people need to demonstrate mastery of fundamental skills before moving on to more complex challenges that build upon those skills.  You also see this in video games, where the onboarding/tutorial experience takes the form of mini-challenges that orient you to the controls, enviroment and affordances of the game before presenting you with more elaborate, complex tasks.  The biggest difference between this type of challenge versus the previous types is that we're really talking about a considered collection.  You need to design effective individual challenges, but you also need to consider how they do or don't fit together in a sequence.

Coming back to the $1K Challenge, it's not such a straight-forward proposition to classify it.  In fact, I'd say that it has changed over the years.

In the early years (2012+), we presented the challenge in Week 1 of the semester.  That seemed like the responsible thing to do, however it had an adverse effect on some of the students, who perceived it to be #1 or #2, rather than #3 or #4.  This meant that unless you were excited and motivated in Week 1, you had more time to work on your project, whereas those who understandably weren't excited about the challenge would be effectively penalized with less time.

In retrospect, this is a perfectly reasonable attitude given that the default educational experience tends to present challenges as achievement gates you have to game in order to move ahead in the world.

Over time, we realized that we needed to make some changes not just in the over all architecture—moving to a model like #4 where we introduced a sequence of small training challenges that precede the $1K Challenge—but also in how the course felt to the student.  These were "smaller" design details, but they were critical to incentivizing the desired behavior.

For example, many of our in-class exercises were deliberately designed to catalyze and normalize acts of mutual help and support amongst the students, which was important to combatting the sense that success at the $1K Challenge was going to be a function of one's individual achievement.

Additionally, we never used anything remotely resembled a leaderboard in the course.  We were also careful about how we presented people's work in relation to others.  Comparison is a slippery slope to competition, which would then incentivize people to work alone.

Third, our goal was learning, so rather than have the crowdfunding campaign end on the last week of class, we had it end 4 weeks prior to the end of the semester.  That gave us ample time to run a proper and rigorous review process.  If we were aiming for #1 or #2, this choice wouldn't have made sense because the participants would be better off with more time.

Challenges are potent tools.  And, when I think back to how we started the course, it wasn't immediately clear what our goals were.  That was something that developed over time through many iterations and cycles of teaching.  

But once we had clarity—which in our case was to teach everyone in the room—it allowed us to be much more precise in terms of what flavors of challenges we wanted to employ and to ensure we had the proper supporting context that signaled how it should be perceived, thus reinforcing the right incentives.