Here’s this video, along with some context and an overview of what we’ve learned.
By the nature of our work, we are heavy consumers of all media, including television. Many of us are also owners and enthusiastic users of the "connected TV" devices, including the ones we tested. At the same time, we understand that we are in a minority when it comes to our own media consumption habits.
While we are confident that the evolution of the TV is set towards its "smartification", we wanted to learn more about how people different from us watch television, how new technologies might change their routines, and what barriers lie on the way of the "connected TV" technology's mainstream adoption.
None of the devices are advertised as cable replacements, and we didn’t set out to test their effectivenss as such. We decided to take the experiment to its extreme and disconnect cable boxes to bring out issues that would’ve remained under the surface had we let the devices co-exist with the families’ existing TV set-ups.
We had plenty of questions going in but no hypothesis to validate. Some of the findings were as surprising to us as they might be to you; others were less unexpected.
We developed and sent out a screener survey to our in-house list. About 60 families volunteered. Of them, we picked five, one for each device.
The five families we selected were fairly diverse in their composition, media habits and levels of technical expertise (and we are extremely grateful to them for letting us disrupt their lives). Everyone had either Netflix or iTunes accounts (some had both), and most had devices other than their cable box already hooked up to their TVs. Every family watched at least 16 hours of television a week (self-reported), watched video online, had a modern TV set, a cable or satellite subscription, broadband access and a wireless set-up.
Each family was modestly compensated for their time upon the experiment's completion; this is a standard practice. Each also received a small allowance to be spent on buying content via their devices. None of the participants was a Hill Holliday employee.
We visited each family twice, to install and then a week later to pick up their box. During each visit, we conducted a one-hour interview, first about their media habits in general and then about their particular experiences with the device over the previous week. Each interview was filmed.
We have also left each family with a Flip camera. Our instructions were simple: whenever you use the device, point the camera at the screen and record and comment on what you are doing.
We have ended up with some 15 hours of video footage, which was edited down to the six minutes of the final video. While we've got plenty of comments about each device, evaluating them individually wasn't our goal. Instead, we looked for themes that were common to all five participating families.
WHAT WE LEARNED
While our sample was by no means representative, the results of our experiment point us toward some real issues that one should consider when thinking about the future of the "connected TV" technologies.
One finding that is probably obvious in retrospect is that TV is invisible until it’s shut off. It's a bit like walking: you are aware of the direction in which you are headed but you don't really focus on the individual steps until you come across an unusual terrain. The exclusively on-demand nature of the devices we tested is just such an unusual terrain that makes you think not only about "where" but also about the specifics of "how".
The devices demand a lean-forward involvement with what has been traditionally considered a lean-back medium, and this requirement proved disconcerting to many when it lasted longer than the usual bursts of involvement with their DVRs or video-on-demand channels.
The Paradox Of Choice
Constantly having to pick what to watch next was daunting not only because it interrupted the usual flow of TV-time activities in the house or required interacting with unfamiliar interfaces but also because of the cognitive load involved in considering all of the numerous content alternatives. "I don't want to have to think about it" was one of the strongest sentiments we've captured in our interviews. As with “the paradox of choice” phenomenon that describes how broadening the range of options leads to a decrease in overall consumption, we saw how families gave up on watching TV altogether when they couldn't decide what it is that they wanted to watch. This problem is serious enough for Netflix to award a million-dollar prize for a better way to tell people what they should watch next; it didn't seem the problem was sufficiently addressed by any of the devices.
People have well-formed expectations about how a TV should work, and the devices didn't seem to confirm well to these mental models. Surfing TV channels is seamless; "tasting" unfamiliar on-demand shows includes picking them from different menu categories and waiting for them to buffer first (and often paying for them up-front). This latency is tolerated in exchange for high-consideration longer-form content but it becomes too much of a friction when all one wants is the "in-n-out" material.
The lack of search spanning multiple video services on a single box was a usability flaw that stood out among other complaints that could be attributed to our users' brief experience with unfamiliar technology. From the users' perspective, there is no reason why they had to search the Netflix, Amazon On Demand and other services individually while looking for a particular piece of content on a single device.
Finally, the devices seemed perfectly suited for certain modes of TV consumption -- viewing the high-consideration content I mentioned above, on occasions when it was available -- but much less so for others. The two families with children found the devices didn't fit how they watched TV at all, which seems like an interesting market opportunity to explore.
And with this introduction, here's the video. I also invite you to watch the panel's reactions and the ensuing discussion. We will also publish our findings about the availability of content across video services and devices. Follow us on Twitter and we'll alert you when they are up.