How to Speak Machine

Technology is a wonderful, truly, magical thing.

Because of its nature, it’s also quite mysterious. We know how to use it, but we don’t know very much about how it really works.

But computation is made by us, and we are collectively responsible for its outcomes; something to be aware of now that computing impacts virtually everyone, and new kinds of interactions with increasingly intelligent devices and surroundings are being more common in a growing digital world.

Therefore, we should have at least a basic understanding of how computing works, to maximize what we can make and be more mindful of how we shape it.

Based on the wonderful book “How to Speak Machine: Computational Thinking for the Rest of Us” by designer and technologist John Maeda, this article is a distillation of key ideas of the first three chapters of the book, that aims to explain (to nontechie people) the properties of computation, which he describes as being “an invisible, alien universe”.

The Three Alien Properties of Computation

1. Machines run loops:

There’s one thing that a computer can do better than any human, animal, or machine in the real world: repetition.

A computer running a program, if left powered up, can sit in a loop and run forever, never losing energy or enthusiasm. It’s a metamechanical machine that never experiences surface friction and is not subject to the forces of gravity like a real mechanical machine—so it runs in complete perfection.

This property is one of the reasons why it’s tricky when tech companies operate computational systems, that never take lunch breaks or vacations and get down to extreme levels of detail, to know everything about you based on your data (especially when you are unaware of how and what exactly is being collected).

Of course, it is also being used positively, to increase comfort and convenience by anticipating needs and automating processes, (which we all benefit from nowadays) making everyday operations seamless and easy.

You may also take advantage of this relentless property to use it in innovative ways, like the Seoul Digital Foundation, which is employing robots to teach older adults to use KakaoTalk, one of South Korea's most popular messaging apps (if you have explained technology applications to your parents or grandparents, you already know it requires a lot of patience and repetition😅).

A Möbius strip, or loop, is a simple object that can be made out of folded paper and reveals surprising properties as you play with it. How to Speak Machine (2019)

2. Machines get large

Exponential growth is native to how the computer works. This is how the amount of computing memory available has evolved. The same can be said about processing power.

“So when you hear people in Silicon Valley talk about the future, it’s important to remember that they’re not talking about a future that is incrementally different year after year. They’re constantly on the lookout for exponential leaps”

— How to Speak Machine (2019) p. 39

Loops inside loops open new dimensions. In short, it’s a means to open up spaces that are much larger than the ones that surround us as the physical scale of our neighborhoods or cities. There are no limits to how far each dimension can extend, and no limits to how many dimensions can be created with further nesting of loops.

Computation has a unique affinity for infinity; however, you are in complete control when you write the codes and construct the loops to your liking.

“There is a certain comfort as you come to realize that, with eventual ease, you can craft infinitely large systems with also infinitely fine details”. (p.52)

This is unnatural to those of us who live in the analog world, but it’s just another day inside the computational universe. 

A magical aspect of the Koch snowflake, a fractal curve, is that its perimeter is infinite but its area is finite. Image: Matthias, LaTeX Stack Exchange (2017)

3. Machines behave like the living (originally stated as machines are living but corrected here)

The traditional approach to creating AI was to teach a computer how to reason through if-then rules. Deep learning (a technique used in machine learning), on the other hand, uses a model of the brain—neural networks in particular—to teach a computer how to think by observing a desired behavior and learning the skill through analyzing repeated behavioral patterns. 

For it to work well, the computer needs to observe our behavior. Preferably constantly and interminably.

This is how we can obtain products like Open AI’s GPT-3, a system released last summer that was trained on a vast corpus of text and that can create text to order that is close to writing created by people, based purely on pattern matching and analysis at massive scale.

OpenAI recently launched DALL·E, a 12-billion parameter version of GPT-3 trained to generate images from text descriptions, using a dataset of text–image pairs. OpenAI website (2021)

Computation’s potential for connection, not only allowing them to process massive amounts of information but facilitating machines that connect and collaborate at speeds that fast surpass ours, is another of its outstanding qualities.

Inventor and scientist Ray Kurzweil predicted that by 2015 computing power would surpass the brainpower of a mouse, and by 2045 there would be more computing power than all of the human minds combined on earth (p. 96), which makes learning to speak machine an imperative need.

“The fact that computers can talk with each other means that they are collectively as smart as the most brilliant computer that they can access.”

— How to Speak Machine (2019) p. 93

Similarly, the social networks and tools we have created now connect us to an expanding network of knowledge and possibilities. 

Whether in the real world or the computational world, connecting work is the catalyst for making changes happen at a scale that’s larger than just the doing work that an individual can perform. (p.89)

How can we figure out ways to collaborate and leverage the full power of our collective intelligence, inspired by how computing works? 

I believe too, as Maeda affirmed in the book, that computers won’t replace us if we remain audacious.

We are creative, complex beings that have created (and are continuously creating) marvelous tools with technology and its special computational powers.

It really feels almost magical, although we are getting used to it by now, to be able to connect, learn, teach, communicate with, observe, and in a close future, possibly even feel, locations, people, and objects all around the world.

What a time to be alive.

---

References:

1. Maeda, J. (2019) How to Speak Machine. Penguin Publishing Group.

---

Fundamentals of User-Friendly Design

As a creative, it’s fascinating to dig deeper into the origin of the ideas you swim among, so that you might better examine—and even challenge—the values you put into the things you make.

The book User Friendly: How the Hidden Rules of Design Are Changing the Way We Live, Work, and Play, by Cliff Kuang and Robert Fabricant, goes through the journey of explaining how the “user-friendly” concept was invented, how it shapes our everyday rhythms, and where it may go next.

I do recommend going through the whole book (I especially enjoyed the historic perspective), and from it, these are key concepts to consider in the design of user-friendly products:

1. Feedback

Feedback is what defines how a product behaves in response to what you want, allowing designers to communicate with their users without words.

“There may be no greater design challenge for the twenty-first century than creating better, tighter feedback loops in places where they don’t exist, be they in the environment, health care, or government.”

— p.34

As the authors put it, “feedback is the fundamental language of user-friendly design”, which is a well-known idea in areas of user-experience. However, the big challenge with designing feedback is figuring out when and where to provide it.

“I am awed by the ever-expanding universe of ways designers provide feedback. And yet feedback is often nuisance—just think of the phone alerts that always seem to appear at the wrong time. It turns out that appropriate feedback is a harder design problem to solve than you think, and we are all intuitively aware when it misses the mark”

— p.311

Dots Hello by Equal Parts Studio

2. Mental models

Mental models are the intuitions we have about how something works—how its pieces and functions fit together. They’re based on the things we’ve used before; you might describe the entire task of user experience as the challenge of fitting a new product to our mental models of how things should work.

Almost all of design stems from making sure that a user can figure out what to do, and can tell what is going on.  The beauty and difficulty lie in what happens when the object at hand is new, but needs to feel familiar so that its newness isn’t baffling.

We need to understand how products fit into existing ideas and concepts of function, so new tools make sense in people’s lives. Otherwise, they may result too confusing or strange, like promising digital assistants that end up with only 3 percent of people using them regularly after two weeks of buying them.

“Humans might fail—but they are not wrong. And if you try to mirror their thinking a little, even the stupidest and strangest things that people do have their own indelible logic.”

— p.26

3. Behavior

Behavior is the spine of the user-friendly world, no matter whether you’re talking about smartphones or toothbrushes or driverless cars: a deference to the complexity of understanding people as they live.

“All the nuances of designing new products can be reduced to one of two basic strategies: either finding what causes us pain and trying to eliminate it, or reinforcing what we already do with a new object that makes it so easy it becomes second nature. The truest material for making new things isn’t aluminum or carbon fiber. It’s behavior.”

— p.96

Similarly, I like the idea of designer Naoto Fukasawa, that the best designs “dissolve into behavior” so that they become invisible rather than stand out for their artistry.

4. Metaphors

Metaphors are a powerful tool for designers and are also linked to our mental models and general understanding of the world. 

It is through metaphors that language and understanding grow from simple things to more complex things. We start with things we understand, like the body, and our own simple behaviors, and create new language:

“The human body is a particularly generative “metaphier”, creating previously unspeakable distinctions in a throng of areas. The head of an army, table, page, bed, ship, household, or nail, or of steam or water; the face of a clock, cliff, card, or crystal; the eyes of needles, winds, storms, targets, flowers, or potatoes… and so on and on…

All of these concrete metaphors increase enormously our powers of perception of the world about us and our understanding of it, and literally create new objects. Indeed, language is an organ of perception, not simply a means of communication.”

— Jaynes, J. The Origin of Consciousness in the Breakdown of the Bicameral Mind (2000)

This extends to everything, including products and technology.

An example is the idea of your email “inbox”, which borrows its logic from your mail, versus social network “feeds” based on streams of constantly flowing content, like a river. You probably at least glance at every email that you receive—simply because they were meant for you. 

Streams are entirely different metaphors. It’s there for the taking, if we wish to, not that we have to. It’s a reason why random, multiple messages from your friends on Instagram are okay but they would be rude via e-mail. Each one has its own set of rules but thanks to metaphors we don’t have to list them all out.

@zambonato

5. Trust

A technology’s success depends not only on the fact of whether it works or not, but whether we trust it, and “the secret is that we come to trust machines only if they mimic the way we come to trust other people.”

Therefore, transparency and clarity are key.

“There are three things an autonomous car has to get right. Above all, we need to know what mode a car is in, whether it’s driving itself or not. [Then] for us not to get surprised, then freaked out by a driverless car, we need to know what it is going to do before it’s actually done. Finally, we need perfectly clear transitions when a car takes control, or when we take control from a car.”

— p.106

Especially when something is new to use, what may seem like over-communication is crucial to make sure there is an understanding between machines and people, and a clear idea of what to expect, according to the features and limitations of the product.

Accidents with self-driving cars have happened, in some cases, due to a mismatch between what people think (or are promised) the “autopilot” function can achieve, and what it actually does, generating a lack of trust.

Along the same lines, the manners of a product (not living, nevertheless interactive elements) are also fundamental. Designs have to understand what’s appropriate or tactful, or simply nice, because that’s the way humans build trust.

“While politeness seems like a trivial detail, it is a design constraint as real as the heat tolerance of steel or the melting point of plastic.”

— p.108

Clippy was unconscionably rude, and “a rude machine is worse than one that simply doesn’t work.”

6. “Form follows emotion”

User-friendly design is about much more than usability—many designers are often surprised by how much user satisfaction is driven by the emotional rather than the functional benefits of an experience. 

The connection between emotional aesthetics and usability was first documented in 1995 by researchers from the Hitachi Design Center who tested variations of an ATM user interface, finding a stronger correlation between the participants’ ratings of aesthetic appeal and perceived ease of use, even over their actual ease of use.

Not that you have to choose one or the other, of course, but “the right emotional connection with a user can make up for some of the challenges, from poor feedback to a convoluted mental model”, so a beautiful, emotional savvy product will contribute to, or even define, how usable it is.

Facce by Stefano Marra

7. Hierarchy of needs

In hiding great complexity behind alluringly simple buttons, we also lose the ability to control how things work, to take them apart, and to question the assumptions that guided their creation.

Modern user experience is becoming a black box: “this is an iron law of user-friendliness: the more seamless an experience is, the more opaque it becomes.”

“A world on instantaneous, dead-simple interactions is also a world devoid of higher-order desires and intents that can’t readily be parsed in a button. While it may become easier and easier to consume things, it will become harder and harder to express what we truly need.”

— p.269

But human needs are not the same as convenient consumption, yet we have been living for decades assuming they are alike.

“User friendly is about deferring to the desires of the users. But there’s a hierarchy of desires. There’s a sense of wanting to eat that cheeseburger, but there’s also that higher-level desire [...] to be healthy and happy long term.”

— Justin Rosenstein, p.269

It is easy enough for us to tell our phones what we like in micro-detail: whether we like this or that story on our feed, or whether we like a song on Spotify and would like to have similar suggestions. These interactions have been optimized to a fine point. 

Yet what we cannot tell our phone is what kind of overarching experience we’d like in our digital lives.

It would be wonderful to have products that help us achieve our bigger goals instead of seamlessly pushing instant gratifications (frequently associated to draining our attention or budget in the package of giving you what you want).

The next phase in user experience should be to change our founding metaphors so that we can express our higher needs, not just our immediate preferences. 

“In helping people understand their world better, in creating the incentives and feedback loops for us to achieve better things, user-friendliness will be an assumed part of whatever comes next.”

@yukai_du

---

References:

Kuang, C., Fabricant, R. (2019) User Friendly: How the Hidden Rules of Design Are Changing the Way We Live, Work, and Play. MCD.

---

On photography

“I prowled the streets all day, feeling very strung-up and ready to pounce, determined to “trap” life in the act of living. ”

— Henri Cartier-Bresson

People are making more images than ever before. Virtually everyone with a mobile phone has a digital camera to take pictures. Facebook says more than 350 million pictures are posted daily, which is more than the total USA population. This is only on their platforms.

In a moment where photography is readily available everywhere you see, the seminal collection of essays by Susan Sontag (published in 1977) On Photography, are still relevant to think about our relationship with images.

The World is Beautiful (one of the first photographic bestsellers) by Albert Renger-Patzsch. (1928)

Photography as reality

With photography as a way of ‘collecting’ reality in a way we believe as accurate, Sontag warns about the traffic of images as proof, or even substitute, of the real world:

“Photographs furnish evidence. Something we hear about, but doubt, seems proven when we’re shown a photograph of it.””

“The problem is that not only can photographs lie – something we still struggle to believe – but they lie on every level. They lie because they are a selective choice of what reality we intend to show. They lie because most photographs are anything but what people think they are – an accurate representation of what is photographed.”

“It seems positively unnatural to travel for pleasure without taking a camera along. Photographs will offer indisputable evidence that the trip was made, that the program was carried out, that fun was had.”

With social media photostreams, the ultimate attempt to control, frame, and package our lives, this idealized photographic image is visible, with extreme examples such as influencer fake travels.

Sontag comments that “ultimately, having an experience becomes identical with taking a photograph of it”, or as she recalls nineteenth-century aesthete Mallarmé who said everything in the world exists in order to end in a book, she claims: “today everything exists to end in a photograph.”

Tourists heading to Bali to take pictures at the iconic lake in #gatesofheaven have been left stunned to discover it doesn't exist. Getty images.

Photography as a mental frame

It is not only those things we photograph what represents our way of seeing, but the photographies to which we have constant exposure can also frame our responses towards beauty and pain, being capable of diminishing our behavioral and mental responses.

“Knowing a great deal about what is in the world (art, catastrophe, the beauties of nature) through photographic images, people are frequently disappointed, surprised, unmoved, when they see the real thing.”

“To suffer is one thing, another thing is living with the photographed images of suffering, which does not necessarily strengthen conscience and the ability to be compassionate. [...] Images transfix. Images anesthetize.

[...] The vast photographic catalog of misery and injustice throughout the world has given everyone a certain familiarity with atrocity, making the horrible seem more ordinary—making it appear familiar, remote (“it’s only a photograph”), inevitable.”

There are too many images of catastrophes and atrocity but we can’t go numb. We don’t need to see too many of them either. Just evaluate what is real and act accordingly.

Child Swims In Polluted Reservoir, Pingba. Reuters/China Daily.

As Jessica Helfand writes in The Self-Reliance Project series, stories of suffering are now a fixture in our news feeds. But it is images like Fabio Bucciarelli's portrait of a woman in her home in Gazzaniga, Italy, that cut through all the reportage and the rhetoric because they offer us a mirror on something else, something hard to see, perhaps, because it is so familiar.

To stop and reflect for a moment on the universality of one person’s profoundly human expression is an aggregation of everything you know to be true.” 

Maddalena Peracchi photographed by Fabio Bucciarelli for The New York Times. March, 19, 2020.

Photography as participation and alienation

The act of photography has a duality of transforming the photographer into both an intimate observer of the frame, catching details that could skip the common eye, and an agent that is detached of its surroundings.

“Photography is also a powerful instrument for depersonalizing our relationship with the world.”

According to Sontag it would not be wrong to speak of people having a compulsion to photograph and turn the experience itself into a way of seeing, supporting the idea that photography interposes itself between us and the ‘real world’ in a way that merely looks like engagement, but is in fact satisfied with a symbolic, morally immobilizing gesture:

“Photographing is essentially an act of non-intervention.”

“A way of certifying experience, taking photographs is also a way of refusing it — by limiting experience to a search for the photogenic, by converting experience into an image, a souvenir.

Most tourists feel compelled to put the camera between themselves and whatever is remarkable that they encounter. Unsure of other responses, they take a picture. This gives shape to experience: stop, take a photograph, and move on. ”

Tourists posing around the Tower of Pisa. Photography credits: unknown.

Photography and time

The capacity of photographs to achieve high detailed images and to “trap life” as it occurs, as mentioned by photographer Cartier-Bresson, gives it the ability to frame a unique reflection of reality during a specific time.

It is a way of collecting unrepeatable moments for future memory when perhaps you can’t remember them as precisely anymore.

“For while paintings or poems do not get better, more attractive simply because they are older, all photographs are interesting as well as touching if they are old enough.”

It is also touching that what you capture will never be the same anymore. Your niece won’t be as little a baby, your face and body will change, the people in the background won’t be there the next time you go.

Photography is both an attempted antidote to our mortality paradox and a deepening awareness of it:

“All photographs are memento mori. To take a photograph is to participate in another person’s (or thing’s) mortality, vulnerability, mutability. Precisely by slicing out this moment and freezing it, all photographs testify to time’s relentless melt.”

First cellphone photo ever taken and shared with others online. Philippe Kahn. (1997)

As stated by Paul Strand, your photography is a record of your living, for anyone who really sees. You may see and be affected by other people’s ways, you may even use them to find your own, but you will eventually have to free yourself from them.

---