#sev2

i still can't get over this moment in V2 yo

That moment from V2 will always haunt me but hey atleast schwaiger got one last laugh over Karl.

OK, yesterday I explained what social science was in the Serendipity Engine algorithm. Today, I explain the way it functions, how does it do what it does. I unveil what else is inside black box. 

Remember, the Serendipity Engine was an exercise in trying to understand two things: 1) what is serendipity, and 2) how it might be “produced" by a digital technology. The first version of the engine (created with Kat Jungnickel) tackled the first question, and the second (created with Ben Hammersley) the second. 

So. How does it produce its results?

The Engine uses a combination of automated and human-powered techniques to generate a personalised Serendipity Recipe. 

Why?

Computers are incredible at holding lots of things in their processors at one time. This means they can cross-reference information in a way that our brains simply are unable to do. That makes them inherently better at making connections than we are, and making connections between things is a prerequisite for serendipity.

But at the same time, computers aren’t very good at assessing human qualities like physical attractiveness or creativity. To say whether someone has these traits require human judgement, which is why the Engine is also made up of human components.

How does it work?

The forms you complete feed the Engine with two things: 

A) keywords that identify what should be relevant and valuable to you 

B) your level of Serendipitousness, measured across seven scales

Step 1: The Paper Form

The information fed to the Engine on a paper form contributes to your personal Serindipitousness rating.

You are asked to do three things: draw a circle, create a drawing from a squiggle and photograph it with your face.

The photograph of your face, your circle and your drawing are sent to a system called Mechanical Turk. Mechanical Turk is an automated, human-powered “computer” run by Amazon that farms out the projects that can’t be automated by a machine to people who, for a small sum, take a look at what needs doing and complete the process. In this case, someone somewhere in the world was paid $1.50 to look at the photograph you submitted and is asked to assess it in four ways:

1)  The circle you draw is compared with a Japanese ensō, a symbol that represents universality, openness and creativity

2) The drawing you make based on the quiggle is rated for its creativity

3) Your face is assessed for its physical attractiveness

4) Your face is rated for your psychological well-being

Step 2: The Online Form

The information you provided the Engine on the online form is used to identify the keywords that are part of your personalised Serendipity Recipe. The form splits into three sections, with three different types of information:

1) The first four questions authenticate who you are by cross-referencing what you have told the Engine about your basic demographic information. This is done by using a Google search of your name and checking the search results with other details (house number, post code, age, etc). After identifying you in Google, the Engine extracts keywords from the top three search results that define who you are in public.

2) Your social media handles are used to identify what is most relevant to you right now, by extracting nouns from your 10 most recent status updates on Twitter, Facebook and/or Google+.

3) The remaining questions are used to extract keywords about you that online databases are very unlikely to have on record. Questions about your favourite subject at school or your Desert Island Disc, or even the food you’d eat tonight if today was your last day on Earth require a much more nuanced level of self-assessment and personal expertise than is catered for in online profiles. This is also the information that can’t be collected based on your browsing behaviour. 

In other words, what you listed as the most influential film in your life may (or may not) be different from the film you’ve watched the most, or rented from Netflix most often, or even what you listed on a social networking site as your favourite film. And your parents’ specialist subjects may be completely different from what they do or did for a living. The Engine extracts keywords from tags associated with each from open data sources for each of these categories.

Step 3: Your Serendipity Assessment

Finally, there are 23 questions in the Serendipity Assessment section of the Engine. In the physical version, they're inside a suitcase and made of knobs, switches and dials. The questions form seven scales, each of which has been identified in the research literature as important predictors of serendipity. I described the scales yesterday. The outcome of your answers to these questions, combined with the responses from the Mechanical Turk, is a measure of your personalised Serendipitousness.

There are also two questions in this section that extract even more keywords: What is your profession? What is the nature of your business?

How’s my personalised Serendipity Recipe determined?

By completing the Paper Form, the Online Form and the Serendipity Assessment, you provide all the information the Engine needs to deliver a personalised Serendipity Recipe.

First up, the words.

5-10 keywords based on your responses to the keyword questions in the Online Form and the Serendipity Assessment are randomly selected from the long list of keywords you gave the Engine access to, whether you told it directly, or it extrapolated based on its automated searching system. These keywords are then put through a process of filtration determined by your Serendipitousness score. 

What’s Serendipitousness?

Your level of Serendipitousness is based on your answers to the questions in the Serendipity Assessment and to the Mechanical Turk’s responses. These are aggregated and weighted to produce your Serendipitousness Score.

The scales are weighted differently according to how important each is in predicting whether someone will think something is serendipitous or not. This is a difficult combination of things, but briefly, it’s about how likely it is that you’ll be able to connect the random keywords in your personalised Serendipity Recipe, and how likely you are to think that those connections could be valuable to you. 

Creativity, Attention and HeadRAM are the two most important scales in the Assessment, and so they’re weighted the most heavily. The other scales - Social Support, Physical Well-Being, Psychological Well-Being and Grit - are also important, but for the purposes of this Engine, they are equally weighted.

See more about the scales and their weightings.

You said it’s filtered? What does that mean?

The randomly selected keywords are filtered through Google Translate. If you have a low level of Serendipitousness, the keywords will be translated only once from English to German and back to English. If you have a high score in the Serendipitousness Assessment, your keywords will be translated up to 5 times.

Why do that?

The Engine creates relevant randomness. It takes things that you are likely to pay attention to and puts a new spin on them. 

In order to the Engine to work for as many people as possible, it has to cater to the different abilities of the people who use it. Serendipity is a very relative thing, based on where you are in the world, what time it is, how you feel that day, what resources you have access to, what political regime you live in, and which culture you’re from. Some of these can vary day-by-day, and even hour-by hour. So to cope with the constantly moving target of your own Serendipitousness, the Engine works for everyone.

The more filtered the keywords are, the less like the keywords you provided. Which means the more connections you might make that are tangential to who you are and what you know. The less filtered the keywords are, the more like the ones you put in. In other words, the Engine makes you work a bit harder to find the connections, have the insight and see the value if you are more serendipitous, and less difficult if you aren’t. 

Is that it?

Almost. There is one more thing. Serendipity is often associated with a flash of insight out of the blue: when walking the dog, taking a shower, having a cup of tea. And so the Engine suggests three contexts to guide you to where you should consider the words, what you should be doing, and who you should be with.

Yup. Predicting serendipity. It's that easy.

So we created this machine - this Heath Robinson device (as Prof John Naughton described it in The Observer on Sunday) - but  how does it do what it does? What's inside the black box?

The Serendipity Engine was an exercise in trying to understand two things: 1) what is serendipity, and 2) how it can be "produced" by a digital technology. The first version of the engine (created with Kat Jungnickel) tackled the first question, and the second (created with Ben Hammersley) the second. 

Here's the answer to how the first fit into the second:

Scales, weightings and their sources

Your Serendipitousness is based on the answers to questions that fit into seven different scales. Some of the questions are considered more “important” for predicting attributions of serendipity (the likelihood you’ll have the insight to make the connections and that you’ll consider the connections valuable), and some of the scales are considered more “important”.

These scales are based on psychological tests. As such - and just like a computer - they are unable to capture everything about the thing they’re trying to measure. They can’t take into account all of the various things that should also be taken into consideration. 

Social Support

If you are high in social support, you’re probably in a tight-knit group of friends and family whom you rely on. So the decisions you make that seem out of character will likely to be supported by your friends and family. Because of your demographic make-up, doors may open for you more easily than for people with a lower score. But you may not have as much access to new information as someone with a lower social support scale. On the other hand, if you’re low in social support, you are probably on the periphery of lots of different networks, and so you’ll have lots more information at your fingertips and you’ll be more innovative because you’ll have less to lose if you decide to do something out of character.

Creativity (x 2 weighting)

How creative will you be in making connections between things that seem to have no apparent connections? If you’re more creative, you’re more likely to see affiliations between things, and it’s more likely that they’ll be new and unexpected. If you’re low in creativity, you will still see connections, but they’re probably more literal.

Physical Well-Being

If you’re body is too busy trying to keep itself in optimal condition, it won’t have the energy or resources to pay attention to things or to make connections.

HeadRAM (x 2 weighting)

This scale measures how much you can keep in your head at one time. If you can keep a lot in there, you will be able to access it more easily, and therefore will be able to make lots of connections. 

Attention (x 2 weighting)

How much attention can you give to your surroundings, and are you easily distracted? If you don’t see or can’t keep your eye on the little random confluences that could lead to a serendipitous discovery, you’ll miss ‘em.

Access to Knowledge

If you have a high score in this scale, you have a very wide range of knowledge at your fingertips. It may not be your own knowledge: are the areas of your parents’ expertise very different from your own? You’ll still have access to that, even if it’s just half-remembered titbits from your childhood. With high access to knowledge, you’ll be able to see more connections between things, and have a diversity of information to help realise their value.

Grit

Grit is a measure of your tenacity - to solve problems, to see something through to the end. This can be particularly useful in predicting whether you think something is serendipitous or not if you’re low in Social Support and the thing you want to do is outside your comfort zone: if you’re high in grit, you’ll do it anyways.

Tomorrow, I'll explain how this is used to deliver your personalised "serendipity recipe".

hmm.

Creativity tests are typically divided into four main components: Divergent thinking, Convergent thinking, Artistic assessments and Self assessments.

it feels rather uncreative to test creativity. but then again, we do like to standardise things, don't we. solves a lot of problems if we turn it into a number, no?