As the chair Professor of Pervasive Computing at City University London and founder and director of the Mixed Reality Lab in Singapore, Dr. Adrian Cheok research is focused on a multi-sensory platform that will enable new ways of communication. He envisions a near future where smell, touch and taste information enrich virtual environments. In the meantime, we’ll use what’s available to us and depict the technology through an interview with Adrian Cheok and the ancient tradition of storytelling.

He spoke to us about his vision for the future of the Internet and human learning, and how the classroom will be much like something out of Star Wars.

You’ve said that in the near future we’ll move past the age of information towards an age where we’ll communicate remotely with all our senses. How do you think this multisensory communication will be helpful in the education field?

The limbic system of the brain is responsible for emotions and memory: It regulates smell, taste, as well as touch. Senses cause subconscious changes in emotion and memory. You can get kids to read a book, or 10 books, or 100 books, about cycling, but unless they actually get on a bicycle, they won’t know how to do it.

Similarly, with smell, taste and touch, you can describe how they work, but unless you experience them, you won’t really understand what they are. Let’s say you’re a teacher in a class about ancient Rome, explaining the kind of food ancient Romans ate. Besides a 3-D visual tour of ancient Rome, what if you could also do things like tasting or smelling the content of the dishes? That would be a case of embodiment, of using your body to learn.

You mentioned a teacher in a classroom. Do you think brick-and-mortar classrooms will still be relevant in the near future? How about human interaction with a teacher?

I think schools will survive but the model will change. A lecture or a class can equally be taught on YouTube or even better with an interactive software that will guide the student to master a topic. It’s just information exchange. I think schools will have to flip the classroom: All the lectures would be online, and the school would be focused on the so-called hard work, the teacher’s helping you solve problems, using the classroom to work in the embodiment we’ve talked about, the experiential learning. The knowledge exchange where students are sitting on their chairs listening to the teacher can be done online; it doesn’t need the classroom.

I think universities will change earlier, though. Schools are mostly controlled by government bodies. It’s a huge system, it takes longer to change. Universities are more independent bodies. What you find already is that a lot of universities are putting their content online. Right now a lot of the content is free. Stanford University, MIT, Harvard–all the top universities of the world are putting content for free on the Internet. And you can do a course online on a lot of topics. Some universities are offering degrees online, fully or partly. People pay for a degree in an elite university even if it’s fully online because the strong branding helps their career dramatically. I think universities will offer all degrees online, the prices will be very low, and what’s gonna happen is that the whole middle part is gonna become extinct. You won’t need those middle-range universities any more. I predict a lot of universities will shut down in the future but others will thrive even more by focusing on the Internet.

I can imagine middle-range universities trying to survive by becoming creative studios for educational content…

The Internet makes things global, so with education, it may happen something similar to the music industry: You can offer a thousand songs, but teenagers will just wanna hear the hip ones. If you’re going to take an online course on psychology, why take it at ABC University when you can take it at Harvard? On the other hand, I think there will be some online universities, possibly Amazon University, that will make extremely good content and will charge much less than an actual university.

I suppose some of the content will be in the multisensory range, as the ancient Rome class you mentioned. What application of multisensory communication you would like to materialize first?

I think that currently [the] technology is not really there. Schools are very much traditional schools everywhere in the world. What I’d like to create for the future is multisensory telepresence; the Internet not being used only for audio-visual information but full five senses communication. Have you seen the Jedi council meetings in the Star Wars Episode I: The Phantom Menace? Some of the attendants are physically there, some are holograms, and [it does] not matter if they are in another galaxy, the interaction is almost the same as if they were present. In the future we’ll be able to interact with 3-D graphics, maybe holograms, touch, taste, and smell. So you’ll see a virtual flower but you could also touch it, pick it up, and smell it. This will allow us to experience a virtual world that’s very similar to the real world.

Will this technology be used mostly by kids?

Yeah, but you can also consider there’s a rising elderly population in all countries throughout the world. Many of them don’t spend much time physically with people. Their families may live in a big city. Multi-sensory communication could help these elderly population deal with their loneliness.

The following science fiction short story explores what the future of education might look like with these multi-sensory communication technologies…

The Punishment Room
By Hernán Ortiz

I’ll have to put up with an old man in the punishment room. They say it’s the new rule against bad behavior: listening to a geezer talking nonsense. It means I gotta stifle a yawn, fake interest, and nod like a moron.

The only thing I did is something I didn’t do: Wear the uniform.

The only thing I did was earn a punishment I cannot escape. They used to make me sweep the schoolyard or change the basketball court’s light bulbs, simple tasks that opened the possibility of me jumping the fence that leads to the street, getting in the multi-sensory rooms at the mall and spending all morning playing my favorite simulation: the one that 3-D-renders your social network friends and turns them into post-apocalyptic zombies. Even though it’s a harmless video game, the bruises on your arms are very real.      

I felt good about my ability to turn punishment into rewards. They elevated me from troublesome high school student to respected obstacle alchemist.

But teachers say it’s harder now to play hooky. They say there’s a new room that was designed by a prison architect. They say it’s locked and watched by cameras and there’s no one inside, not even the old fart I’m forced to speak with.

This is the extent of what we know. Hearsay about the walls compressing and heating up to extreme temperatures, the students compelled to make eye contact with a mechanical demon that projects, directly to their retinas, their worst fears, forced to stare endlessly at the beast’s eyes to lower the room temperature and stop the motion of the walls–was simply pure imagination. We speculated severe punishments, but none of us knew what was hidden behind the room’s iron door.

In today’s first class, Spanish class, I thought I was going to go unpunished for my infringement. Being the only class I work hard in to learn, the teacher wrongly thinks I’m gifted in languages, that I’m not a lost case, that I can be a good student despite my flaws.   

If she’d dig more, she’d discover the reason I’m interested in Spanish is the same reason I didn’t wear my uniform: to impress Natalia.

If she’d dig more, she’d find out Natalia is the Colombian girl who works at the multi-sensory room’s ticket counter. The girl who doesn’t give me the time of day. The girl who’s in college.  

If she’d dig more, she’d conclude that I was looking to be grounded so I could visit Natalia at work.

In today’s second class, the PE teacher saw my clothing–polo shirt, jeans, shoes–and scolded me in front of my classmates.

“Mr. Johnson,” he said, combing his mustache with his fingers. “Can you explain to the class why are you not wearing your uniform?”

It’s an old technique, shamelessly taken from The Teacher’s Manual. He’s not the only one in need of an explanation, but the whole class, which in reality doesn’t give a shit. Grouping himself with the students under the “class” umbrella, he’s looking to increase the number of offended parties, and thus my blame. But the technique is also vulnerable to made-up excuses. I could’ve said my uniform was stolen, or the wrong detergent discolored it, or rats chewed through it. Excuses that, depending on the delivery, would have exonerated me. But I was looking for punishment. So I said, “Because I felt like it.”

The PE teacher began breathing very quickly and told me with undisguised pleasure that I was going to be the first student to enter the new room. Although this was a deviation from the original sneaking-out-of-school plan, his words, instead of making me feel bad, elevated my status from troublesome high school student to atonement technologies pioneer. While he lectured me on the disrespectfulness of my transgression–words thrust out in spit particles–I realized PE is the only class that requires a special uniform. Maybe that’s why he’d been so sensitive about it.

My dad is the complete opposite. He’s not concerned about my misconduct. He offloads his responsibility to the school and doesn’t read the notes they send home–he just signs them. He says grandma keeps him busy with temporary jobs: framing artwork, feeding fish, patching walls. He says he makes a living out of it, but he’s always skipping rent payments.

At the entrance to the punishment room, the PE teacher performed the biometrics authentication routine (fingerprint, retina, voice). The doors opened vertically from the middle, and a gust of compressed air leaked out. My favorite feature of the room is that when you walk inside and the doors close behind you, the PE teacher vanishes. The world shuts out and the room’s lights click on, illuminating a white cubic space.

“I don’t think you should be here,” the Spanish teacher says through the wall’s PA system. “I mean, anyone can get confused. Alas, there’s nothing I can do. The rules are the rules. Being a moderator is the only help I can offer you.”   

The question is: Who asked for help? This teacher is insane. If I bombed the school, I swear she’d arise from the debris to defend me. She’d try to persuade the injured teachers not to punish me. She’d say it was a case of teenage catharsis, not nihilism.

“I didn’t get confused,” I say. “It was my decision not to wear it.”

“I have a feeling you hate the look of it,” she says, sympathetically. “And you know what? I’m not too fond of it, either. Plus, I have yet to get used to the bio-fabrics, and those blind stitches make the clothes so similar to human skin that I feel naked. But you gotta agree bio-activewear is perfect for PE. Clothing is not just about aesthetics. We use insulated garments to protect ourselves from cold weather, weight bells and diving cylinders to submerge and breathe in the ocean’s depth. And here in this room, we use dynamic multi-textured fabrics and mixed-reality lenses to communicate sensorily with remote users.”

The wavy motion of the walls temporarily pause the teacher’s speech. White light beams projected from all four walls connect on my body.

“Close your eyes,” the teacher says. “The room is performing a scan to provide you with a better uniform.” I ignore her suggestion so I can keep looking at the surrounding projection of white light, the organic look of the skin walls, the oval hole on one side of the room that’s growing and stretching like a yawn. My first instinct is to get away. A demon’s mouth, I think.

“The room just wants to help you,” the teacher voice says after checking on my nervous reaction, making me keenly aware of being watched.

This is very different from the multi-sensory rooms where Natalia will guess your size and hand you something that can only be described as a deflated space suit. In here the air flow switches trajectory, and my body is drawn towards the room’s mouth, absorbed by it. And when the mouth shuts, a series of foaming tongues wrap around me like boa constrictors, sewing a layer of transparent skin on my legs, arms, and head, around my waist. Tongues coat my face and fasten a device in my nose, eyes, and ears. My mouth is the only part that remains uncovered.           

The room’s mouth opens once again to spit me out, and just when I feel like I’m about to lose my balance, the air adjusts accordingly, pushing me up and pulling me back with enough intensity to keep me upright.

The uniform feels soft; a layer of skin that coats me without overheating me.

“Looks good on you,” the teacher says, her voice sounding like it’s coming from my own head. I see her image (she’s sitting at the teacher’s computer room) taking up a window of the virtual interface. “You just have to wear it during the communication. I hope you don’t mind.”      

Wondering how Natalia would react if she saw me walking around the mall wearing this uniform, I ask her if I can take it home with me.

“I’m afraid you can’t,” the teacher says. “The room sucks it up for recycling purposes. Next generation bio-activewear is all about automated sustainability. But feel free to take the bubblegum home.”

“What are you talking about?”

The room spits a white pill.

“Try it,” the teacher says.

A green circle instantly appears around the pill. How stupid is that. You don’t need visual aids to find things in empty rooms. I toss the pill in my mouth. Tasteless.

“Taste is a hassle to deploy,” the teacher says. “Early prototypes looked like braces; they fitted in the mouth of the users and provided a poor texture simulation. It was a long way before the programmable bubblegum was released. Please let me know when you’re ready.”     

I smile with the gum between my teeth.

3-D-animated fruits float in front of me: lime, mango, and watermelon. “Pick one,” the teacher says. I reach out my hand to grab a lime, and even though it is just pixels, I feel the roughness of the peel on my fingers. A message in the virtual interface invites me to trace a continuous path on the lime. In what spot do I want to cut it? it asks. I slice the lime in half with my knife-fingers. The smell of citrus overwhelms my nose. My mouth waters. I lift a piece of lime and squeeze it over my opened mouth: the digital juice melts the bubblegum into a liquid texture, pervading my senses with the unmistakable tart flavor.

The lime fades away. “The functionality of the uniform has been successfully verified,” the teacher says, the bubblegum in my mouth restoring to a solid texture. “Before starting the communication, I need to remind you this project is the result of a partnership between Friends of Seniors–the foundation that built this room–and the school. The goal is to hone the academic skills of troubled students.”

The teacher notices I’m shaking my head no. She asks me what’s going on. I’m calling bullshit. I sneer: “The goal is to help an old fellow feel less lonely. Don’t disguise it as something good for me; it only benefits the school and an old man.”   

“It’s a win-win situation, believe me,” the teacher says. “We’re looking to emphasize the skills of troubled students. Given your talent in Spanish, the room prompted us to find Mr. Pepe Lugo, a respected teacher enrolled in the Friends of Seniors program. With the technology resources given by the foundation, he’s not limited by the classroom.”

The image of a smartypants old man teaching me how to live triggers a coping mechanism. I picture him talking to himself while I slip away to imaginary conversations with Natalia. My happy place; the strategy I use when boredom strikes.

“Mr. Pepe successfully completed the uniform’s functionality test,” the teacher says. “Where would you prefer to meet him? A park, a library, a restaurant?”

“Are there more options?”

“Thousands. The thing is, Mr. Pepe has walking issues. Even with the possibilities of mixed reality–the illusion of movement specifically–he chooses seated simulations he can experience from his wheelchair.”

“The restaurant is all right, then.”

“Great. I’ll set myself in invisible mode. Feel free to ask any question. I hope you have a magnificent learning experience!” the teacher says, compressing into a red circle.

A countdown pops up: 5, 4, 3, 2, 1… and the punishment room is transformed into a restaurant. Soft chatter in Spanish mingles with vallenato songs and grotesque carnival masks. The place is decorated with ribbons of outlandish colors. The aroma of exotic cuisine and fruity drinks wafts in the air. I’m impressed by the quality of the graphics, they’re considerably better than the ones you get in the multisensory rooms. The simulated air is warmer and moister. The simulated roar of nearby surf makes me feel like I’m in a coastal area. This is a persistent dream that doesn’t disappear by closing your eyes.

A green circle encloses an old man who’s sitting alone. Even without visual cues, the old man stands out among the other customers dressed for summer: his blue hat with a black ribbon and blue suit with a black tie match his dark skin and white beard. He’s sitting in a wheelchair at a table for three. The Spanish teacher notices through the cameras that I’m walking with my arms extended in front of me. She says, “Don’t worry about stumbling into walls. You’re not really walking. The floor is a multi-directional treadmill.” While she talks, the red circle she’s been compressed into vibrates.

“Just a bad habit I got from zombie games,” I mutter, dropping my arms. The teacher laughs. She thinks I was joking. “Save your whispering for the classroom,” she says. “Our communication is private. You can hang-up by just pressing–” and that’s what I do, I press the red circle before she can finish because the old man has just seen me and he’s taking off his hat. “Hola, ¿cómo estás? Me llamo Pepe” he says, introducing himself. Another green circle encloses the chair in front of him, and I sit in it. The old man reaches out and I lean over the table to shake his hand, feeling his saggy and wrinkled skin. “Mucho gusto, yo soy Johnson,” I say. The surname guarantees some emotional distance.           

“I like your Spanish accent, Johnson,” the old man says, showing his white teeth. “It’s one of the most difficult things to master, you know?” I chuckle at his comment–my accent comes from watching telenovelas.

I open the menu of the so-called Restaurante Bohemio, surveying a diverse gastronomic offering: carimañola, patacón, arepa de huevo, posta negra… flashy names without pictures or videos to help me understand what they mean. “This restaurant is a simulation of one in Cartagena where you can find the best Colombian cuisine,” the old man says. “Are you ready to order?” The old man raises his left hand, beckoning the waiter. I notice his wedding ring.

“¿Está casado?” I say. I don’t really care if he’s married, I just want to practice so I can ask the same question to Natalia.

“Soy más o menos viudo,” he says. I don’t get it, you’re either a widow or you’re not.

“The digital version of my wife is here with us,” he explains, pointing towards the third chair. “The foundation built digital versions of our deceased loved ones to help us cope with loneliness. You can’t see her because she’s in private mode. I can give you access if you want.”    

“¿Qué es esto?” I say, pointing at the menu, looking to understand the food options but also trying to change the subject. The old man describes each item. He tells me what he’s going to order. “Is your wife ordering too?” I ask, feeling like an idiot. He shakes his head. “She only eats when we’re alone.”

The waiter is carrying a stainless steel plate cover on a tray. The old man helps me order in Spanish, and when the waiter lifts the cover, the order appears right there, on the tray. Simmering doesn’t exist in this digital world. Instead of fast food, they serve instant food.

The waiter put the plates on the table. The smell whets my appetite, and when I take a bite, I realize the extreme transformation of the gum that’s still in my mouth: the crispy outside/soft interior texture with the salty flavor of the carimañola con queso becomes the fibrous texture with sweet and sour flavor of the posta negra and becomes the sandy texture with bitter-tasting flavor of the patacón con suero. The gum melts down to a liquid when I drink the aguapanela con limón and hardens back to solid with the panelitas con coco desert.  

I finish the meal feeling ambiguous: Mentally satisfied, physically hungry.

“¿Cuál comida te gustó más?” says the old man, wondering which food I like the most. The waiter put the dirty dishes on his tray, covers them with his stainless steel plate, and lifts the cover to reveal a set of clean dishes he uses to serve at another table.

“Carimañolas con queso,” I say. “Or posta negra. I don’t know, maybe all of it.”

“You remembered the names,” he says, kneading his white beard. “That’s the power of embodiment. I believe in using the body to learn.”

“I’ll forget them in a few weeks for sure,” I say. “This requires practice, right? Dad is never gonna take me to such a fancy restaurant just to improve mi español.”

“Don’t you have anyone to practice with?”

“I have a Colombian girlfriend,” I say, wishing that just by saying it aloud it will become reality, in this virtual world of possibilities.

“Did you hear that?” the old man says to the empty chair. “The boy is also in love.”

Lies talking to lies. I had managed to ignore his ghost wife, but that simple dialog, so everyday and harmless, drastically changed my perception of the old man from that of a weird teacher to a senile crackpot. Should I help him accept his wife’s absence? Should I tell him the foundation cannot fix something as inevitable as death?

“The truth is she’s an acquaintance,” I say. “She’s just a girl who works at the mall. Can you imagine if we run a mall simulation in this room? I could learn so much Spanish!”

The idea deflates the old man’s cheerful attitude. “I’m sorry, but I’m stuck in a wheelchair,” he says. He stays silent for a while, remembering his condition. His eyes are cast down, his fingers scrubbing his forehead.

The awkward silence is interrupted by the Spanish teacher. “According to the foundation’s rules, it is possible for me to run the mall simulation,” she tells me, “but two conditions must be met before I can do that: Mr. Pepe has to agree to go with you, and you must help him get over his fear of walking. He knows he has to use his hands to roll the wheels without touching them; hand gestures are the only thing that’s needed. Alas, he suffers from an elderly anxiety disorder, and past experiences show that he panics as soon as his virtual representation stands up.”

“Is Friends of Seniors helping him?” I ask.

“Researchers at the foundation think young students like yourself are more able to solve this kind of personality puzzle,” the teacher says. “That’s why they’ve invested a huge budget in this room.”  

It’s clear to me now that I’m trapped in a lie. If grownups believe such a sham, we young people are hopeless. There’s no mechanical demon in this room, just the projection of my worst fears. I touch the red circle.

While I was talking with the teacher, the old man initiated a private conversation with his fictional wife. Seeing him interact lovingly with the lie he chose to believe in, I realize she’s the only one who can help him.    

I ask him if his wife likes to go to the mall.

“Just when the digital friends at the foundation go with her,” he says, making eyes at the empty chair. “She doesn’t like to go alone.”  

I ask him why he doesn’t go with her.

“During the 50 years we were married, my wife never saw me walking. I don’t want her digital version to have that privilege.”

I ask him if we can go together.

“My digital wife and I have always been here,” the old man says, looking around the restaurant. “She’s my stability in these strange virtual worlds.”

“But I’m here, too,” I say. “And I’m real. And I’m sick of going to the mall almost every day, unable to talk to Natalia about something different than video games. And my dad…I told him about this looking for advice, but he only complained he had a migraine.”

A waiter interrupts, asking us if we’d like something else.

“Thank you sir,” the old man says, “but we have things to do.”

The old man says something private to the empty chair. He hugs the air in front of him. His tears fall to the ground. I imagine things from his perspective: His wife’s scent while he embraces her, the tears on her wet shoulders. He pushes the empty chair in.

A countdown pops up: 5, 4, 3, 2, 1… and the restaurant vanishes. We’re at the main entrance of the mall. It’s an exact replica, so accurate that if we go across the hall, we can reach the multisensory rooms.

The old man closes his eyes and says something I don’t quite get. I suppose he’s saying goodbye to his other wife, the real one, the one he keeps in his memories.

The old man stands up. “There’s a compliment my wife loved,” he said. “Me gustan tus ojos. You can tell her that.”

I walk towards the multisensory rooms, excited by the possibility of running into a digital version of Natalia, a version I can practice Spanish with.

The old man follows me. He looks down to contemplate the movement of his arms and legs. Arms and legs that are gesturing over a wheelchair. A wheelchair that rolls forward until he catches up with me. Until he leaves me behind.

See also:

• Imagineering Institute — Inventing the Future of the Internet
• MIT Media Lab’s bioLogic
• Skin Deep: Making Building Skins Breathe With Smart Thermobimetals 
• Keiichi Matsuda’s Hyper-Reality