The Augmented Home

In Silicon Valley we like to think about “first principles.”

What does this really mean? Go back and really study how things work without bringing your biases to the table and then come up with new products and services that help you do that task better.

Steve Jobs went further.

When he started the iPhone team he told the first 12 people “do not hire anyone who has ever worked on a phone before.”

Why?

He wanted a fresh look at the industry and he wanted to see how new technology and a new approach could change the game. Elon Musk recently did the same when he and team designed the Cybertruck. They threw out all our beliefs about what a truck should even look like. So many people have been polarized by that design, which shows the power and danger of taking such an approach.

As we see companies like Apple and Meta, er, the company formerly known as Facebook, try to introduce new devices into the home, that will “augment” the home in various ways, it’s useful to study what people are doing in their homes, so you can judge how much usage they will get, and how accepted they might be, or, even, for competitive analysis to see who is strongest at a specific use case in the home.

I’ve been fortunate to have been invited into people’s homes all over the world. Rich, poor, and everything in between. That let me observe what people actually do, vs. what I think they do. And what they have and how they spend their time.

That turned into a framework, a simplified version shared here, to help everyone think through new companies in a new way. If only to help build competitive analysis in the future.

This led me to notice that very few people do “high engagement computing” while walking around. They almost always are sitting down when doing high-engagement computing (although if you have a standing desk that messes things up a bit but very very few people have one of those in their family rooms so we’ll leave that for a future discussion). This tells me the augmented reality industry has focused on the wrong type of experience.

Apple and Unity aren’t going to continue making this mistake, because they have done the kinds of human factor research I have. Apple even built a fake home to study these different contexts in the home. Apple’s strategists are focusing on this high-engagement computing, I hear from people working on its new products.

How do I define engagement? Are you touching or interacting with the computer at a high rate?

“Low engagement?” Things like listening to music (you do realize that is computing, right? Try playing music in your headphones, car, or home without a computer). But you rarely touch a screen or interact with the music. Other than to start up a playlist or play a song. Yeah, we could say if you are doing a ton of searching, liking, curating, etc, you are moving toward high engagement. I still include that in low engagement computing.

High engagement? If you are designing something in a 3D app, or working on a spreadsheet or doing a slide deck, or even answering email, I put that in high-engagement camp.

While following people around and understanding what they actually do, both are important. An architect can design a new building while also listening to music. One is a high-engagement task. The other low.

Generational shifts

Visit a home with two teenagers and you’ll see that while the parents do most of their entertainment-focused computing in the family room, that isn’t true of the kids. Some are very focused on being in their rooms, watching videos, or, more importantly, gaming. Some families even banish the gaming system to the garage, so that when adults are having a party the kids are out there playing to their heart’s delight.

Apple has new eyes on these uses of technology in the home. Can Apple win back the kid who has a gaming console or a PC in their bedroom? My research shows yes, but that will be very tough short term, particularly if Apple is planning on charging $3,000 for its visual headphone.

Kids also have their own media and music preferences that often clash with that of adults. Which is why they are relegated to the hinterlands of bedrooms or garages also. Can a new kind of headphone change that mix? Yes. But we need to see what actually happens vs. what our theories about what will happen. Lots of unknowns here, like the role of AAA games, and what the music and metaverse industries push to the kids.

What is the conclusion?

That there is a new unserved market for VR/AR companies: focused around people that are sitting down to do a variety of high-engagement tasks. One that is now working at home, along with doing other high-engagement tasks. How many couples are now both working at home, struggling to deal with Zoom calls and using new collaborative tools.

Even when the kids come out to watch a movie with their parents, you see another thing.

Everyone is still on their personal devices, like phones or tablets — which are completely crappy screens compared to an 83-inch TV for watching video content. Still the mobile phone persists, even in such a home. TikTok is the biggest beneficiary of this new behavior. Even while watching TV or a movie, things that 20 years ago would get everyone angry at you for being distracting. Now everyone is in this state of continuous partial attention. Former Apple/Microsoft exec Linda Stone told me that was coming back when I worked at Microsoft 17 years ago. Here we are. Watching a Netflix show on the big screen while watching TikTok videos on the small.

This tells me that if a new device gets launched that lets you “soak” in both worlds that it will be very popular. Of course I’m thinking of Apple, but others will try to compete for the home. I don’t think they will be very successful short term, but we’ll see why when Apple comes in June.

When Uber was invented it was after the iPhone shipped. I was there in the Paris snow storm when Garrett and Travis were complaining, saying “why can’t we see where the cars are on our phones?”

The introduction of a new computing device let us think differently about a task we had done thousands of times before (getting a ride in a taxi).

Same will happen in the home.

After we get augmented reality devices we will change what we want in the home.

But, like with Uber, the businesses that take on things we already do will have a major leg up on companies that will try to introduce a completely new behavior.

The big money spot

So, where is the sweet spot? The one that will end up with most of the competition and money.

Let’s focus on just the family/living room. You know, the one room that has the best TV in the house. When I visit homes I look for commonalities. Literally everyone has a TV in their family room. Even in very poor homes with dirt floors. Almost always had a TV and if they didn’t have one, a neighbor had one and invited everyone over to watch the local football team (soccer in most of these places).

While in the family room, I keep track of who has a state-of-the-art 2020 or newer TV or audio system?

Even in “rich” Silicon Valley literally no one does. This is key for Apple, because a very tiny percentage of people have a surround sound system or a big TV that is capable of 4K or better resolution.

Our family invested in a $20,000 media room, complete with everything state of the art, or at least state of the art as of last June (and state of the art for that price level — the billionaires sometimes have $250,000 rooms but I study mass market consumer behavior, not that of billionaires).

Doing this research also led me to study people’s reactions as they visit our home (we had a family here from Texas for 10 days at Christmas time, too, so got a ton of insights from that).

Even with such a mind-blowing system people still are poking at their phones, looking at TikTok or playing a variety of games or watching YouTube. I started keying in on this. Conclusion: custom media beats high quality media. People regularly tell me they won’t switch from Spotify, even though it has easy-to-demonstrate far shittier music quality than, say, Amazon or Apple Music on my new Sonos system. Why? Personalization beats quality. Over and over.

What if someone gives consumers both?

My research shows that such a device will sell very very well.

Other things to pay attention to if you are starting a company that is going after home users?

Home users multi task. How many of you watch a TV show, while dinner is cooking, or while you are actually working on a different screen? Many of you keep TV on in the background all day long. Others, music. While I’m typing this on my iMac, which is in our kitchen (Maryam is working in the office down the hall) I’m watching what’s going on over on Twitter (TweetDeck is on our big screen) and what is going on around me (Maryam emerged from her office to get some lunch). A bunch of high and low engagement behaviors all mixed together. Some of which will get us to put on headphones. Some of which get us to take them off.

A funny thing I discovered with homes that have a modern Sonos in them, by the way, is that very few percentage of people who have purchased a modern AV system have them properly tuned and setup (Sonos has a feature called “TruePlay” that makes sound dramatically better if you use it right. Many even hyper-rich people haven’t done it at all. Plus, it took me months to figure out that I wasn’t getting the best picture quality on my TV because of one setting I missed). Most people also have small TVs. Only a tiny percentage of homes, even in “rich” Silicon Valley have bigger than 65-inch TVs.

The resistance

This means there’s a lot of resistance to devices that cause task switching. You hear a ton of resistance when doing market research. Many many people tell me “I will never wear anything on my eyes.” Many tell me that and then pull headphones out of their pockets when I ask. Never believe what people tell you, always watch. I have thousands of examples where people told me “I’ll never do that” and then three years later they are doing that same thing all the time. Same here. It’s hard to see how a VR device would gain so much mainstream acceptance that everyone will wear it for hours every day. I’ve studied several paradigm shifts close up. Right before each hit, and even as each was hitting, there was extraordinary resistance. I got yelled at in a coffee shop when I was trying to complete the first credit card transaction in that store, for instance, and when I introduced email to the office I was working at for the first time people complained, saying they liked paper memos better.

Even people with VR headsets rarely brought them into the family room. Already they have been relegated to other spaces for a variety of reasons. Why? The task switch from being immersed in, say, Beat Saber, to making sure the kids are doing their homework, or making sure your food isn’t burning, is extremely high and keeps VR from being used nearly as often as it should be.

Such headsets also bring new problems into family rooms (my son hurt someone who walked into his playspace while he was playing a high-engagement game and swinging around wildly). I hear Apple has focused on this task switching a lot, along with the safety issues in the home (one reason we’ll see how it solved the problem in June at WWDC when it introduces its new visual headphone).

Also, the person who is watching a TV show, while playing Words with Friends on her iPhone, still wants to be social with others in the room. While you COULD use, say, Apple headphones while others are in the room (I tested this at Christmas dinner, everyone got used to me being a dork, and I could listen to music while still talking to them with the transparency mode), very few feel comfortable with wearing a device while trying to be social. Will that change? I predict it will based on my research, but Apple has to solve the task switching problem. It’s possible for it to be watching and listening to everything for you. What if you were playing Beat Saber and it told you “the kids are now playing on their computers.” Or, “your oven timer is going off.” Or, “a smoke alarm is going off, so we are pausing all high-engagement computing until you get it to go off.

I asked lots of questions about what would happen in the home if everyone had a device on, and new kinds of experiences were introduced (like new kinds of board games you could play with your kids). Here people are intrigued, but say it depends a lot on the execution. So far only weirdos like me try wearing devices. My research shows that could change in a huge way.

The catalyst? Football. What gets people to come over more than any other event? Football. Since I have the biggest, sharpest, TV in our family/friend group, I find that every Sunday I have people over here to watch the TV. That didn’t happen before.

The Context is key

If you are building a game that is really about walking around the world, er, your neighborhood outside, it will have a tough time serving the “sitting on your couch” context.

Let’s back up. I wrote a book about contextual computing. Back when Shel and I wrote that we didn’t have the kinds of AI we do today. Today we have new kinds of AI that hasn’t yet been introduced into the home.

If you are dancing, the potential things you will do are a lot different than if you are lying down watching TV.

If you are building augmented reality services for the home thinking through the context is key.

There will be different distribution channels to get to those contexts too. Would you launch a sports app the same places you launch a math app? No! So, people who are helping entrepreneurs, like me, have to keep track of each and build relationships with those who run such. I already have a list of 2,200 companies building the 3D Internet (AKA metaverse) and am rethinking them thanks to this context map I posted here.

Already well-funded companies are going after each of these contexts. So I’m tracking the money spent and also what new startups are up against as they try to get consumers to take time they used to use to watch TV shows or movies to do something new, like playing multi-player augmented reality games in our homes. That competition will lead to many millions of words spent, and lots of opportunities for partnerships and mergers.

What does this mean for entrepreneurs?

Each will need to know what they are up against. If you are competing for sitting on the couch time you are up against Ted Lasso, football, and everything else that comes. Unity, alone, is going to make a HUGE play for this space. Becoming “the virtual refrigerator” company will be a lot easier, based on my experience, for a low-funded startup than trying to be “the virtual coffee table company.” That said, if you have a real shot at the coffee table the minutes spent there in each home is much higher than the time spent at the refrigerator. Each choice you make will have consequences down the road (and different financing and staffing needs).

Look for what are the big companies not yet doing? Lots of analysts will track that, but I don’t know anyone who has approached each context in the home yet.

This is why I am talking with so many startups lately, all for free with no strings attached. This lets me figure out how the market will break down, and where to look for newer approaches for each. My phone is open at +1-425-205-1921, please text first.

I’ll keep this all up to date and keep adding more details based on what else I learn between now and when Apple comes later this year.


Dolby Atmos and Apple’s rewritten audio stack

This is a reprint of my email newsletter. Subscribe here.

Human storytelling is about to become MUCH richer.

I’ve been spending the summer understanding the technology changes coming to consumers soon. Lots of people at lots of companies, from car companies, to audio companies, have told me what is coming. Which made me realize that by Christmas of 2022 we will be seeing a HUGE upgrade in all media quality.

What am I expecting to upgrade? Well, audio is being upgraded right now, so that’s one, and on a high-end audio system you already can experience better music than I did by standing in the front row of more than 150 concerts. When I was last at Preservation Hall in New Orleans Reggie Watts gave a performance three feet from me. Yes, that’s amazing. But what is more amazing is that while only a few people can fit into Preservation Hall, now we have the same audio quality in our homes. Actually better, most of the time, truth be told. 

Soon photos will upgrade. So will video, which is already seeing dramatic changes on the high end. So will VR and AR, which will provide radical new experiences in your homes from multi-party video games to new kinds of virtualized TV screens.

Lately I’ve been telling everyone to pay attention to Dolby. Why? Dolby has been working for years on what Apple is about to bring to market.

In this newsletter I’ll focus on Dolby Atmos and the changes that happened the past few months on Apple Music. 

What impressed me is that Apple totally rewrote its audio stack on all of its devices, and put two AI chips in its headphones to enable Atmos in headphones. Apple’s audio renderer is the best in the business, Sean Olive, head of R&D at Harman, who makes JBL, told me. He should know, he’s one of the few humans who has built a double-blind audio testing lab, among other feats.

I start with music because music has ALWAYS been an important part of storytelling. My son just forced me to watch a “silent” film from more than 100 years ago. It had music! On a “silent” film.

This is why I am trying to get the VR/AR industry to care about the changes going on with Apple Music and its investments in 3D music. With Dolby Atmos you often feel like you are in the front row listening to your favorite music. It surrounds you, just like if you were very close. 

When I grew up I listened to tons of music coming over AM radio (KFRC in San Francisco, amongst others) on a small transistor radio. The quality was worse than today’s iPhone’s speakers (which also just got improved and on my iPhone 13 with iOS 15.1 even has a pretty interesting version of Dolby Atmos which blows away my 13-year-old self). But on Apple’s AirPod Pros or Max headphones you can hear the difference already. Dolby Atmos content is richer, nicer to listen to for long periods of time, and is in 3D around you (yes, we can argue that it’s not the same as on, say, my Sonos gear, but that is missing the point. Dolby Atmos is 3D music AND has way better fidelity to boot. It’s better even on cheap speakers and when you put it on a $4,000 system it’s explosive, and beats most Broadway Plays and concerts for quality).

So, to get you started, this summer I went through tens of thousands of songs on Apple Music, Tidal, and other places, to try to find all available. 

I still remember walking into Tower Records back in 1979 when CDs were just coming out and they only had a few dozen to choose from. Now we have 6,000 songs. Yes, still only a fraction of the millions and millions of songs available, but anyone who doesn’t use 3D music in the future will sound lame. Everything from movies to TV shows to new VR experiences will bet heavy on 3D audio, er, Dolby Atmos, and every week we are seeing major new releases in Atmos and the number is going up exponentially. 

So, here’s a playlist so you can get up to date. Apple has a new device to play all these coming by Christmas, so you’ll want to definitely come back and listen to all these again on that.

The complete list (way bigger than Apple’s own list).I then went through all 6,000 songs, looking for the best examples of Dolby Atmos usage. 

That led to this “innovators” playlist.

Yes, if you aren’t on Apple Music you are an idiot.

It’s way better sounding than Spotify. Better in Apple headphones than Tidal. So, I’m starting this newsletter to cover the changes for people who can afford $250 headphones and up on the Apple ecosystem. If you have an Android phone, good for you. It doesn’t sound as good as Apple’s ecosystem. I don’t write these to cover why to move ecosystems, but from now on this newsletter will cover everything content for Apple users. If you have another device, I really don’t care. You aren’t experiencing the best media out there if you do. Might be harsh to state that but it’s true and the gap will greatly grow next year.

Over on Apple Music you should follow me, so you can see what I’m listening to in real time. You also can see dozens of other Dolby Atmos playlists. Over on my Apple Music Profile you will see playlists of Classical, Jazz, Rock, Country, and many more. All Dolby Atmos.Soon everyone will understand why this is important as we move to a 3D metaverse. 

Either way, I don’t know how often I’ll write these newsletters, but will keep them coming when something big releases. For instance, last week the music from “Hamilton” was released in Dolby Atmos. This recording made me cry as I realized the music on my Sonos system was better than what I experienced in the real play when it visited in San Francisco. When you get a chance to hear this on an amazing Atmos system, it is revelatory. A friend who was in the studio told me that they arranged each track on speakers in a room to argue about where in 3D each performer should be placed in your home. Wonderful to listen to and much better than the same music released a few years ago in 2-channel sound.

Another one that just got released that’s notable is Miles Davis “Sketches of Spain.” On a high-end system you can hear the musicians at the right height and it just is exquisite. My wife and I have listened many times already.

Finally, I will also point to other people who review music and experiences and will bring you the best that I find, like this one in Sound and Vision about another excellent Miles Davis album in Dolby Atmos.

The Apple Table is Set

The big meal is about to come. We’ve been waiting for years for Apple to reveal its mixed reality products, including visors and glasses. We’ve been seeing the potential coming in other products like the Oculus Quest, the Magic Leap 1, or Microsoft’s Hololens. For years we’ve dreamed about an augmented world. Steve Jobs called them “bicycles for the mind.”

Yesterday it announced a bunch of things to developers and a few of these “human helpers.” Techmeme has all the reports here: https://www.techmeme.com/210607/p22#a210607p22

The most interesting was that, as of yesterday, Apple Music now is available in Spatial Audio. In a way, its headphones that enable that feature are the first to use its new philosophy: “how many ways can we improve lives by including more AI and 3D visualization in products?”

Apple knows that great audio is the foundation of great experiences in visual apps, like video games, entertainment, virtual shopping, education, and concerts. So it makes sense to upgrade all audio, which Apple is in the middle of doing. Another major upgrade comes next year when audio gets locked to the real world.

Apple is confusing people with all the new audio terms. Lossless and Spatial Audio are two I hear a lot, but Apple hasn’t been clear about why we want either, or both. By this time next year it’ll be clear, or, rather, it won’t matter. Most new media will be available in the better of the two formats, Spatial Audio. That format gives you infinite surround sound and, coming next year, it will be locked to the real world. Really these announcements are about moving us toward the experiential world.

When I say “experiential world” what do I mean? Well, going to a concert or a sporting event is experiential. You experience these things by being immersed in the event you are attending. What Apple is heading toward is shipping a “Holodeck” that will let you attend virtual concerts. Or go virtual shopping. Or attend a virtual school. Among other things. 

Apple will, before the end of the year, I hear, announce two products: a brand new iPod that we haven’t yet seen and a new “Holodeck” which is basically a high-end headphone and visor for experiencing mixed reality.

That said, I was expecting a little more about the 3D map. The fact that they showed us a new 3D map, but didn’t give developers a bunch of new capabilities, is interesting to note. That tells me that Apple will be more muted that I was expecting. That signals to me that it will do a “Viewmaster” product approach where the announcement of the product might not need, or have an affordance for, a lot of third-party apps. 

Look at Apple Music. To me that’s where Apple is setting the strongest tone about what is to come. It just upgraded a good chunk of its catalog to support Spatial Audio. No external developers needed. In fact, most people who work at music companies have no idea what Apple just did to the music industry. Let’s put it this way: I just cancelled my Tidal and Spotify accounts. 

Because of this trend toward experiential services and away from tons of new developer efforts, Apple has continued the course set by Steve Jobs. Apple knows the consumer electronics industry better than anyone. The iPhone was launched directly against the Consumer Electronics Show, held every January in Las Vegas. That wasn’t an accident.

Anyway, now I’ll turn my efforts to the business impact of the moves now that Apple is coming after owning your home. Both on its competitors and on developers that rely on Apple’s ecosystems for their business.

The “Holodeck” (there’s a rumor that Apple calls it “Apple View” but I’ll call it the Holodeck until Apple officially announces) will be aimed at our family/living rooms, entertainment context. I’m not expecting people to wear it a lot outside. I’m counseling entrepreneurs to focus on family time. Playing games, watching TV/movies, reading books, listening to music, etc. Entrepreneurs who have a strong showing for what people will do in their family rooms have a shot at building significant businesses.

Entrepreneurs who I’m working with are finding it challenging to raise money at the moment. That will change over the next few months, particularly after Apple announces. This isn’t the first time investors will change their attitude. Back when AltSpaceVR started up the founders told me that no one would invest in it. Then Mark Zuckerberg bought Oculus and several of those investors they visited called back.

Same will happen here. So, have an investment plan for before the Holodeck gets announced, and a different one for after. Same for your PR plan and your go-to-market plan. 

I’ve been talking to a bunch of companies who are aiming to where the puck will be, to use a hockey metaphor. One entrepreneur, Robert Adams, stands out as a good example of an entrepreneur that is building technology that Apple will need in the future. His business, Global edentity, builds a variety of sensors and AI that sees biomedical identity, among other things.

Now, that usually sounds dystopian, but Apple is already showing us how to thread that needle: stay focused on delivering products that help humans and minimize the consequences (and with this new genre of technology there are many). The things it showed us yesterday will improve our lives. Spatial Audio, for instance, makes music that sounds way better. That doesn’t sound dystopian now, does it? Nope.

Adams has shown me many new technologies that companies like Apple could add to future products and services. 

For instance, one of his inventions looks at the vascular and/or skeletal system of the user. This is something that a camera can do that your human eye can’t. Can you see the blood flowing through people’s faces? Nope but his system can. What does that lead to? All sorts of things, from earlier sensing of disease, to much better identity systems. If Apple wanted to make users much more secure and healthier it could, but it would need to invest in companies like the one Adams is building before a big company as Google or Apple buys them. This is what Apple was doing the past decade: buying a bunch of little companies that you don’t know much about that now are becoming important parts of Apple’s tech stack (like the 3D sensor in the latest iPhones, which came from a small company in Israel, Primesense).

Another of his has ways to make our health better and even further add identity capabilities by adding a smell sensor to a future phone or glasses. Such a sensor and associated AI could even smell that you are experiencing troubling emotions like anxiety, or, even, a health problem. I’ve seen dogs that can smell those, along with other things, on humans and sensors that can “smell” are coming along. They say even a blind dog knows its owner! So why not your phone? Imagine the possibilities.

Anyway, I’m seeing an explosion in entrepreneurial activity as Apple, Facebook, Snap, Google, all spend billions of dollars readying products that will ship over the next few years. So, I’m expecting to talk with a lot more people like Adams. Am available at +1-425-205-1921.