During a panel at our TVnext summit yesterday, we showed a video with highlights from a recent experiment. For this experiment, we had invited several families to give up their cable and instead use a “connected TV” device for one week following last Christmas.
Here’s this video, along with some context and an overview of what we’ve learned.
PURPOSE
By the nature of our work, we are heavy consumers of all media, including television. Many of us are also owners and enthusiastic users of the “connected TV” devices, including the ones we tested. At the same time, we understand that we are in a minority when it comes to our own media consumption habits.
While we are confident that the evolution of the TV is set towards its “smartification”, we wanted to learn more about how people different from us watch television, how new technologies might change their routines, and what barriers lie on the way of the “connected TV” technology’s mainstream adoption.
None of the devices are advertised as cable replacements, and we didn’t set out to test their effectivenss as such. We decided to take the experiment to its extreme and disconnect cable boxes to bring out issues that would’ve remained under the surface had we let the devices co-exist with the families’ existing TV set-ups.
We had plenty of questions going in but no hypothesis to validate. Some of the findings were as surprising to us as they might be to you; others were less unexpected.
METHODOLOGY
We developed and sent out a screener survey to our in-house list. About 60 families volunteered. Of them, we picked five, one for each device.
The five families we selected were fairly diverse in their composition, media habits and levels of technical expertise (and we are extremely grateful to them for letting us disrupt their lives). Everyone had either Netflix or iTunes accounts (some had both), and most had devices other than their cable box already hooked up to their TVs. Every family watched at least 16 hours of television a week (self-reported), watched video online, had a modern TV set, a cable or satellite subscription, broadband access and a wireless set-up.
Each family was modestly compensated for their time upon the experiment’s completion; this is a standard practice. Each also received a small allowance to be spent on buying content via their devices. None of the participants was a Hill Holliday employee.
We visited each family twice, to install and then a week later to pick up their box. During each visit, we conducted a one-hour interview, first about their media habits in general and then about their particular experiences with the device over the previous week. Each interview was filmed.
We have also left each family with a Flip camera. Our instructions were simple: whenever you use the device, point the camera at the screen and record and comment on what you are doing.
We have ended up with some 15 hours of video footage, which was edited down to the six minutes of the final video. While we’ve got plenty of comments about each device, evaluating them individually wasn’t our goal. Instead, we looked for themes that were common to all five participating families.
WHAT WE LEARNED
While our sample was by no means representative, the results of our experiment point us toward some real issues that one should consider when thinking about the future of the “connected TV” technologies.
One finding that is probably obvious in retrospect is that TV is invisible until it’s shut off. It’s a bit like walking: you are aware of the direction in which you are headed but you don’t really focus on the individual steps until you come across an unusual terrain. The exclusively on-demand nature of the devices we tested is just such an unusual terrain that makes you think not only about “where” but also about the specifics of “how”.
The devices demand a lean-forward involvement with what has been traditionally considered a lean-back medium, and this requirement proved disconcerting to many when it lasted longer than the usual bursts of involvement with their DVRs or video-on-demand channels.
The Paradox Of Choice
Constantly having to pick what to watch next was daunting not only because it interrupted the usual flow of TV-time activities in the house or required interacting with unfamiliar interfaces but also because of the cognitive load involved in considering all of the numerous content alternatives. “I don’t want to have to think about it” was one of the strongest sentiments we’ve captured in our interviews. As with “the paradox of choice” phenomenon that describes how broadening the range of options leads to a decrease in overall consumption, we saw how families gave up on watching TV altogether when they couldn’t decide what it is that they wanted to watch. This problem is serious enough for Netflix to award a million-dollar prize for a better way to tell people what they should watch next; it didn’t seem the problem was sufficiently addressed by any of the devices.
Expectations
People have well-formed expectations about how a TV should work, and the devices didn’t seem to confirm well to these mental models. Surfing TV channels is seamless; “tasting” unfamiliar on-demand shows includes picking them from different menu categories and waiting for them to buffer first (and often paying for them up-front). This latency is tolerated in exchange for high-consideration longer-form content but it becomes too much of a friction when all one wants is the “in-n-out” material.
Usability
The lack of search spanning multiple video services on a single box was a usability flaw that stood out among other complaints that could be attributed to our users’ brief experience with unfamiliar technology. From the users’ perspective, there is no reason why they had to search the Netflix, Amazon On Demand and other services individually while looking for a particular piece of content on a single device.
Finally, the devices seemed perfectly suited for certain modes of TV consumption — viewing the high-consideration content I mentioned above, on occasions when it was available — but much less so for others. The two families with children found the devices didn’t fit how they watched TV at all, which seems like an interesting market opportunity to explore.
And with this introduction, here’s the video. I also invite you to watch the panel’s reactions and the ensuing discussion. We will also publish our findings about the availability of content across video services and devices. Follow us on Twitter and we’ll alert you when they are up.
Ilya runs the R&D / Consumer Insights group at Hill Holliday and studies the relationships people develop with different traditional and emerging media. 
This is a great video! It really emphasizes how far the user experience for connected TVs must go before they can be adopted by the everyday, mainstream user.
I was surprised how nobody made the more specific complaint about about the lack of sporting events. I know there is the ESPN on the XBOX but that has a very weak selection, so I never use it. Perhaps each sports team needs to join and broadcast their own games live cutting out the middle man.Netflix also needs to have the ability to create a playlist of shows/movies and a top priority show list that includes a sports team channel. So that when you log on to Netflix or whatever service you are using the first thing you see is that a new episode of a show on your top priority list is available or that your sports team now has a game in progress. Then when all the new shows on the top priority list are complete, they improve the recommended selection. I can just copy the series I have saved on my DVR and add them to the list (if there was a way to automate this process would be best). Also there needs to be a way for people who can not type to speak to their television and or more interactive way select like using the Kinect.
The number one and most important thing that is preventing me from cutting the cable cord is sports……and the number two reason is Entourage & Boardwalk Empire.
We fired our satellite provider a few months ago, and went to a pc/net based viewing using a custom built xbmc/boxee hybrid (has both installed.. can change what we use based on desired experience).
The ‘burden’ of choice, and the lack of an easy search option can bring rise to frustration, as well as a “give up and just pick something or shut it off” decision.
the reduced noise levels, the removal of the constant distraction, and no longer being forced to accommodate THEIR time schedule.. FAR outweighs the difficulties and growing pains
I recommend everyone try it.. (just make sure you have a good high speed connection)
[...] their connected TV products. Nevertheless this past Friday there were five Boston-based families doing just that. [...]
Very interesting video considering I just bought a RoKu box and am waiting for it to be delivered UPS.
It was very interesting in that people seemed to be in a mild state of cable TV withdrawal. things weren’t what they were use to so they were stressed.
I think my experience will be much better considering I’ve never had cable Tv and don’t want it. I am NOT a passive viewer. I am a very active person so I don’t watch much television. when I choose to do so it is a deliberate act. If I want to watch netflix I pull up a movie. If I want to hear music I pull up pandora. I think I will be just fine with my RoKu choice
This was only half the story, because the experiment did not allow the users to access free over-the-air programming.
I cut the cable about a year ago and access online content through a computer that uses the TV screen as a monitor, but I also watch all the networks, (and can record them) from a set of rabbit ears. I get twenty eight channels digitally that answer the similar complaints from the participants: I know when shows are on, I can easily surf, etc.
This was a really interesting study to me. I feel like I’ve been look for the ‘holy grail’ of internet tv boxes so that I could disconnect cable. A couple of things I’ve discovered, after years of trying old laptops, xbox and various other things: I now have the sony google tv, but this would work just as well with the LG Revue. I also have a ps3. I find that the PS3 serves most of my tv needs with Netflix, Hulu Plus and Vudu (plus Blu-Ray, of course!). I was a little surprised the xbox was chosen in this study versus the PS3. I think the xbox is great, but I returned mine a week after I got it for the PS3 (much to my friends’ dismay), because the online video content is more mature. Yes, it doesn’t have ESPN, and that’s something I’m still trying to solve. But generally, the google tv gives a reasonably easy computer type way to access video if all else fails. So combined, I feel like the two devices complement each other well, and I can always watch the shows or movies I want (ok, 28 days after blockbuster).
The other part of this, at least for me, comcast basic cable was only $4 with the internet package, so realistically, it’s easy to get some live news or live shows (just not in HD).
[...] “I feel really good about Comcast” is not something Roku, Boxee or Google execs would want said by people who’ve tried out their connected TV products. Nevertheless this past Friday there were five Boston-based families doing just that. [...]
Could you please disclose if you currently represent any television providers (Verizon, AT&T, Time Warner, Cablevision, Comcast, or others) in the United States, and whether there was any outside funding for this work?
Sports is the crown jewels of cable and satellite. LIVE sports. the internet is not “good enough” yet for fully live. Can you imagine you get halfway through the superbowl, and “buffering please wait” comes up?
This actually bodes well for Microsoft’s upcoming Windows Media Center – Embedded solutions. They include much of what these devices do, but also embed real Cable TV.
To-date, it’s been niche because it’s been a PC-only solution. With the embedded device route, there may be a real contender there for mainstream usage.
Why wasn’t anyone also given an over the air digital antenna? Most of the issues given by the households would have been easily solved by freely available broadcast HD. LIVE sports, in HD. Passive viewing/flipping around.
The biggest statement I took from the video is the visible lack of adequate broadband, whether it be an issue of accessibility or affordability—another study.
We gave up cable TV, all TV actually and just use the web when we want to watch something. For sports, like the NFL, we listen to the radio.
It has been 4 years now, and the only time we watch normal TV is in a hotel room. This made me realize how much I hate it, always having my attention drawn to what was on the screen instead of where I want it to be. This made me happy to get back home where I view what I want, when I want it.
Very poor and random study. All consumers today do a bit a investigation and comparison before they chose to purchase any device or service, so why must this be a “Made To Fail” experiment. No one had a choice when they received their device and maybe the match was poor? The test subjects all admitted that their TV experience is mainly for “veggin’ out” or “just leave the TV on all day” or my favorite “there’s a movie on and I don’t know why but I like it” attitude. If your happy with cable TV and $1200+ a year is fine, then by all means stay with it. I’m sure that the invention of indoor plumbing was a hassle to people years ago and many would rather continue to throw their feces in the street. Pitiful.
Robert forgot that NBC broadcast the olympics live online using Microsoft Silverlight. And I heard it was pretty good.
What’s with people having WiFi issues? Why didn’t you guys just bring ethernet cables and make sure there wouldn’t be issues from, say, cordless telephones or microwaves?
I watch live sports all the time. Online. And the box is plugged into my tv aka 46″ monitor. There’s no buffering with the online package I buy.
It’s like you people are complaining that all cars are horrible because you drive a Pinto. I’m quite happy with my BMW, thank you. I suggest you get better internet.
Thank you all very much for your comments. We are pretty overwhelmed by the response and all the discussion on Techcrunch, Electronista, Gizmodo, Kotaku and other blogs that have picked up the video.
Let me try to answer the questions I see here.
@Jon – We’ve heard a number of comments about the lack of sporting events. If I remember correctly, one family plugged their cable back in to watch a game during the experiment. For the video, we wanted to focus on themes shared by all families, however.
@Dudley – I think you will enjoy your Roku if it fits your lifestyle; it’s a great device at a great price.
@Tom and @Chad – We were curious to see how the devices would stand on their own and how they would influence people’s TV viewing choices.
@Glenn – Hill Holliday paid for the experiment in its entirety (the cost of the five devices, compensation for the participants, and two Zipcars). The experiment wasn’t commissioned by any third party. I am not aware that we work for any cable business, but I’m double-checking.
@Crutch – One of the limitations of the experiment was the fact that people didn’t grow into their devices organically by purchasing the devices themselves as a response to a recognized need, but were thrown rather abruptly into unfamiliar technology.
I don’t think letting the participants choose their device in advance would’ve made any difference.
@Tom – In several cases, ethernet cables were an impractical solution.
@Seldon – Many of us here are satisfied users of everything from HTPCs to the boxes we tested. It works for us, too. What we wanted to see is what would happen if we gave these devices to people who are not us.
Seems like an interesting survey. The technology is not mature but it offers some insight into viewer habits. My question is: What is the average hourly TV usage in the US? Is this sample made up of viewers who normally watch TV 2 hours a day or 8 hours a day? Are they typical viewers?
I thought the days had passed where people watched of 6 hours of TV a day.
“re: None of the devices are advertised as cable replacements, and we didn’t set out to test their effectivenss as such.”
um, yeah, you did. You cut off people’s cable and gave them these devices. If you didn’t want to test them as cable replacements then you should have just given the devices to people and asked them which they used more at the end of the week, the cable or the device.
While “cord cutting” is very big in the media right now I feel confident that people just don’t drop cable on a whim to get one of these devices. Also, most people don’t drop cable b/c these devices are easier but to save money. People know there is a trade-off, money for convenience.
Some people drop landlines for cell phones, but they have cell phones first before dropping the landline.
Interesting; thanks. I have DirecTV and a Roku box. I’d love to lose some of my $100 p/m DTV bill, but instead added $8 to get Netflix on the Roku… sigh. It is, however, a great combination. For example, we were watching “Damages” (which we missed on the original run) on DTV, when my wife asked if it was on Netflix. Ta Da… and in HD without commercials, too.
If it were not for their exclusive series, I’d be able to drop HBO/Showtime and same some bucks. Getting HD channels like NatGeo, History etc, and not bumping up against my ISP data ceiling is another consideration.
So, at least for me, it turns out that Roku is a great addition… and I’m paying more (instead of less.) Ain’t capitalism wunnerful?
@Cary — According to Nielsen (http://bit.ly/hd6axM), an average American watched about 30 hours of TV a week last year.
@Bob Emmerich — Our goal wasn’t to evaluate the devices, it was to observe people’s behavior. Let me also add by quoting myself: “We decided to take the experiment to its extreme and disconnect cable boxes to bring out issues that would’ve remained under the surface had we let the devices co-exist with the families’ existing TV set-ups.”
The problem is, just like anything else, there’s a learning curve to all these new technology options – even with our phones, unless you sit down and spend a few hours reading the manual, playing with it or going on-line, or maybe finding someone in the store who can show you how it works and set it up. I think all these choices are daunting and yet we crave them because normally all we do is complain about how much we are paying for connectivity and yet there is still very little worthwhile watching. If I could have access to all the things I like to watch at anytime and make them favorites or build a library I can go back to anytime I want to watch them, that would be worth a few dollars but we all are already complaining about the high cost of our cable bills – what does the future hold when the largest cable company in the nation, Comcast is allowed to purchase a major supplier of content, NBC and all its divisions?? 16 hours a week of sitting and watching something just because it is available and the lesser, easier choice of having to actually do something else like, a hobby, or reading, or playing a game must mean that we are all too tired to use our brains and need some red bull to fill the gaps in our energy. TV is and always has been dope for the mind (for the most part) so how could you expect a doped up mind to figure out how to use something that is technically challenging as well as taking so much time to find what you want to watch and…the big question, what are we willing to pay for this opportunity. Remember, most television shows and for the most part movies have been designed to appeal to the widest audiences possible with the combined intelligence of a sixth grader for the least amount of money. That is why it is a “commercial medium” because somebody has to pay for all of it and what used to be totally free because of advertising, now costs us money plus we still have to sit thru the commercials unless you can afford a DVR in every room and can zip thru the ads. Television or better yet audio/visual presentation is meant to satisfy our needs for escapism for a little while from the drudgery of our mundane middle or lower income lives. And, with regard to current affairs in our world, the most popular parts of the news are the weather and the sports – that’s why they don’t show them til the end of the show…just like grocery stores put milk in the back of the store…it’s what most people want and we are made to wait til the end so they can get us to watch more ads or walk around the store in the hopes that we will buy more than the milk we came in for. Options are wonderful and we live in a land and time that is full of choices – the question is, when someone is presented with too many choices the mind tends to give-up and shy away and make no decision because it is easier. Just like dressing a child, you don’t offer them 6 different options to choose from, you offer them 2, and, just like on the car lot giving a person too many options makes it too difficult for people to decide and make a purchase so the smart salesman guides the customer to make a decision based on price and looks (and also why the car makers have thrown all the options into one to three packages basic– luxury or luxury unlimited). I always ask myself, what did people do before indoor plumbing?, what did people do before electricity? how did people find each other in a crowd before cell phones? what did people do before television? how many encyclopedias were sold before the internet? and, has all our modern technology really saved us all the time it was promised to do or has it just zapped our energy and time trying to use it and maintain it? If I spent all the time taking those quick “little customer service surveys’” everyone asks me to take or all the time it takes me to de-bug, re-start or try and figure out why my computer keeps doing things it’s not supposed to do, I probably wouldn’t have much time to even use that wonderful indoor plumbing. No matter, in another 50 years or so we will all have personal robots to do it all this for us or we will all be dead.
I cut the cord over 10 years ago. Soooooo glad I did. I can watch what I want, when I want, and have a more open schedule. I do think the average person will be scared to cut the cord.
I did this same exact thing this summer. I went about 4 months without cable when my wife was traveling. I used a laptop loaded with Boxee software and the Hulu desktop app connected to my TV via HDMI. A wireless keyboard and mouse was my input. Long story short it worked ok but it just wasn’t as good as cable. The biggest thing I missed was sports. ESPN 3 filled the hole when I was really jonesing for sports, but if I wanted to watch a Red Sox game I had to go to a bar because the subscription to the MLB service was as much as a month of cable. When I my wife returned I tried to show her the set up but within a day I got a request to turn the cable back on.
At the moment I have whetted my appetite for streaming media by ripping all my DVD’s to ISO and buying a Seagate GoFlexTV and making a small Media Server out of an old desktop that was sitting in the basement. With the Seagate I can watch all my DVD’s without having to find the DVD case and it also streams Netflix, something we use all the time.
We spent over a year without cable, and used a Roku box or streamed from a computer. True – if you watch the big cable tv shows (Showtime, HBO etc) or sports for most of your viewing content, you will be disappointed by the lack of instant gratification (If using Netflix you have to wait until the series is on DVD; Hulu and other on-line services play the show anywhere from an hour to a few days after original airing). However, if you are like us and watch mostly movies, like to see series from start to finish and watch only a few regular network tv/pbs show, then it’s a great deal. No matter where you live, you will still be getting your basic free network channels (PBS, NBC, ABC, CBS as well as UHF channels) anyway.
I agree that participants should have been given an antenna for over the air digital cable. I have been looking into getting ride of cable and the digital antenna will solve a lot of problems. Being a Mac person, I have been looking to get mac mini instead of the apple TV to supplement the antenna. The mac mini has an HDMI out, so it can be easily connected directly to your TV, it also has a front row which allows for easy access to use itunes account and any ripped movies or cd’s incorperated to your itunes account. Finally being a computer it will allow you to access the web and use services such as Hulu, netflix, amazon or Youtube as well as access websites like Fox, NBC, CBS, etc……. My plan is to drop my cable provider after the Superbowl.
Projected costs – $120/yr netflix – $50/one time fee for the antenna – $600/one time fee for mac mini
Projected savings – $85/month ($1,020/yr) – 1st yr $250 – 2nd – $900
We actually plan on not purchasing the mac mini until we save the 600 from the cable bill.
I just did not find this surprising or even that interesting. You take a bunch of people who are used to passive media consumption and you try to get them to adopt an activce consumption model and, surprise surprise, becuase it’s not what they’re used to, they don’t like it. I have owned a teleevsion for yours – all of my media consumption is “active” in the sense that I actively choose what and when I want to watch something. On those occasions in recent years when I have been able to watch broadcast television, I’ve not enjoyed it – it doesn’t match my expectation of what media consumption through should be (passive, you can only choose from what’s on at the time). Not really that groundbreaking…
clicker.com is the multi-site search function they needed to test. it and it’s queue (which is like a tivo season pass) solves the search/discovery/decision making problems…now if only more content was available…
Interesting study but you are comparing apples and oranges. The devices you supplied are really more of a comparison with in-home video rentals as they lacked “live TV”.
If you do the study again then I would suggest doing it and cutting off video rentals from kiosks, rental stores and the VOD portion of cable and satellite OR
supplying devices that do have a more complete offering. One service that has both is from skyangel. The other platform could be Microsoft Media Center running on a PC. Media Center also allows you to connect an antennae or cable. In the case of cable users can subscribe to basic for $16 – $20 per month. In many cases people will pay less than traditional cable or satellite and many will be satisfied.
BTW, ESPN3 is a very nice service, sometimes “better” than the broadcast ESPN channels. I am a tennis fan and found many tennis matches that were not broadcast but could be watched through a PC connected to a TV.
Hope this helps.
[...] Agency Hill Holliday did a presumably biased experiment (they have a vested interest in the continuity of the existing market structure) where they took [...]
[...] Agency Hill Holliday did a presumably biased experiment (they have a vested interest in the continuity of the existing market structure) where they took [...]
I’d be curious to get your thoughts on why these families have “well-formed expectations about how a TV should work.” is it because TV is a technology we’ve all grown up with and have developed a learned expectation? Or is it because TV designed in a way which forms to a hard wired manner in which we prefer to consume its content? I think the implications of this answer would have a big impact on the future of the smartified, connected TV.
[...] about it and watch the video here. emma wiselady Brand [...]