#measurability

Measurability

It is not uncommon for people to assume that everything can be measured. Certainly, that assumption fits well with our everyday experiences. If we look around us, we see many attributes like height or weight that are obviously measurable. It is also clear from our everyday experience that people vary in terms of their intelligence and personality in much the same way that they vary in terms of their height and weight. But does that mean we can measure intelligence and personality like we measure height and weight?

Not necessarily.

Consider a simple statement: John weighs twice as much as Jill. The meaning of this statement is clear. If John weighs 100 kilograms, then we know that Jill must weight 50 kilograms. Likewise, if we have John and Jill sit on opposite sides of a seesaw, we know that Jill will have to sit twice as far from the centre as John to balance the seesaw.

But what about intelligence? If John has an IQ of 100 does that mean he is twice as smart as someone with an IQ of 50? Well, not exactly. An IQ of 100 is considered average but someone with an IQ of 50 would be considered severely intellectual impaired. Certainly, many courts would be reluctant to hold a person with an IQ of 50 responsible for their own actions since such a person would be in the bottom 0.15% or so of the population.

Clearly then, two people with an IQ of 50 are not going to be able to do the same amount of intellectual work as one person with an IQ of 100. Contrast this with the weight example. Two Jills (each 50 kilograms) really do equal Jim’s weight (100 kg). Weight is additive but intelligence may not be, at least, not in the same linear fashion.

We can make a similar case for personality. If someone receives a score of 100 on a test of extroversion, are they really twice as extroverted as someone who receives a score of 50? Even a cursory examination of most modern tests of extroversion will show that such a belief is simply not tenable. It doesn’t make sense.

So what does this mean? Tests are instruments that assign numbers to people. However, that does not mean we can treat these numbers like we treat other numbers. Weight assigns a number. Weight is also quantitative (a very special property), which means we can make a range of very important statements, like :

5 kg + 10 kg = 15 kg

10 kg divided by 2 = 5 kg

10 kg multiplied by 2 = 20 kg

0 kg means that something has no (zero) weight

Those statements might seem obvious, but let’s try to apply them to IQ:

50 IQ + 100 IQ = 150 IQ

100 IQ divided by 2 = 50 IQ

100 IQ multiplied by 2 = 200 IQ

0 IQ means someone has no (zero) intelligence

Do these statements make sense?

If you take someone who is severely intellectually impaired and have them work with an average person, is their intellectual output similar to a genius? No.

If you take two severely intellectual impaired people and have them work together, is their intellectual output similar to a normal person? No.

If you take two average people and have them work together, is their intellectual output comparable to a genius amongst geniuses? No.

If you take someone with a score of zero, do they have absolutely no intelligence? No. Well, in fairness, they’re likely not very bright, but they would still possess some level of intelligence. Just being able to attempt the questions (even if they get them wrong) is a sign of intelligence.

Clearly, then, weight and intelligence are not the same, or at least, scores on them do not have the same meaning. Weight is clearly measurable. Intelligence may well be measurable too, but not to the same extent. This is where scales of measurement become important:

Nominal

Ordinal

Interval

Rational

When we measure things in a nominal way, all we can do is tell them apart. Think of names. We know that Tom, Sam, and Bill are different people. How are they different? We don’t know. We just know that they are different.

An ordinal scale can rank things. So, for example, if Tom came first in a race, Sam came second, and Bill came third, then we can have an ordinal measure of their speed. Not only do we know they have different speeds from each other, but we also know how they rank. What we do not know is how big the differences are (e.g., Tom might have only just beaten Sam while both of them were way ahead of Bill). If you are measuring things in an ordinal way, all you can do is rank them. You cannot comment on the size of the differences between them.

An interval scale is more advanced. It can accurately reflect the differences between things. The best example is the Celsius scale for temperature. The difference in temperature between 5 degrees celsius and 10 degrees celsius is the same as the difference between 10 degrees celsius and 15 degrees celsius. What is missing, however, is a meaningful zero. 0 degrees celsius does not reflect an absence of temperature. It is simply the freezing point of water. Without a meaningful zero, we cannot multiply and divide meaningfully. For instance, 10 degrees celsius is not twice as hot as 5 degrees celsius.

A ratio scale has all the properties of an interval scale but with a meaningful zero. The Kelvin temperature scale is a ratio scale since its zero point does represent an absence of heat. Weight and length are also ratio scales since 0 kg means an absence of mass and 0 metres means an absence of length.

It should be clear by now that when we say that we are measuring something, we like to imagine that we are measuring it on a ratio scale. That lets us do all of the fancy maths and statistics that we like, and even more importantly, it means that all of those maths and statistics actually mean something. But if we don’t have a ratio scale, what then? Well, we can still do the math and statistics… they just won’t mean anything.

And this is where things can get ugly. On what scale do you think we can measure intelligence, personality, and other psychological attributes? I think we can be confident in measuring them to at least an ordinal scale, but are we measuring them on an interval or ratio scale?

More worrying is another question: is it even possible to measure intelligence, personality, and other psychological attributes on an interval or ratio scale? If it isn’t, then tests of intelligence and personality are built upon foundations of sand.

Reliability

The question of measurability is highly theoretical and conceptual in nature. In contrast, the question of reliability is much more straightforward. For a test to be reliable, it must measure something consistently and accurately.

Consistently and accurately means that if the attribute being measured has not changed, then a test should give the same result when used on multiple occasions. Consider weight. If my scale is reliable, then if I weigh myself ten times in a row, I should get the same result each time. Likewise, if someone’s personality or intelligence has not changed, then a test of those attributes should give the same result each time.

Unfortunately, this is easier said than done since psychological tests are generally vulnerable to practice and demand effects. If I gave you the same intelligence test five times in a row, one after the other, you would undoubtedly get better at it because you have seen the questions before. Similarly, if you are applying for a job, and I give you the same personality test five times in a row, you might start to figure out what sort of personality is desirable for the job and change your answers accordingly.

This is why measures of test-retest reliability are seldom used in personality and intelligence. People learn when they are asked to do a test multiple times, and this in turn changes their scores. So how can this problem be solved? The simplest method is to design alternate versions of the test that are matched in difficulty and overall content (e.g., you might have two versions of a maths test that have the same problems, just with different numbers involved). You could then look at how closely scores on the two version corresponded. This is called alternate forms reliability.

Finally, there is something called internal consistency. If your test has multiple questions that are each attempting to measure the same thing (e.g., multiple questions about someone’s extroversion or multiple questions about problem solving), then someone’s performance across all of these questions should be fairly similar.

The reliability of a test can thus be measured in several ways:

Test-retest reliability

Alternate forms reliability

Internal consistency

A test that does well in these categories is consistently and accurately measuring something. Now, that doesn’t mean it’s measuring the right thing (that’s what we’re about to get to), but it is measuring something.

Validity

In simple terms, validity is all about whether a test measures what it is supposed to. More practically, a valid test is also a useful test since it can either predict some important outcome or corresponds to something important. In other words, if you want to know if a test is valid, the question you should be asking is: does this test measure what it is supposed to measure.

The simplest way to check validity is also the one that is most prized in human resources and organisational psychology: predictive validity. A test has predictive validity if scores on it can predict some important outcome (e.g., an intelligence test predicting job performance or a personality test predicting managerial ability).

If a test of something is good, then the scores it gives an individual should be similar to the scores given by other good tests of that same thing. For example, if you design a new intelligence test, then the scores it gives people should correspond to the scores given by pre-existing good tests of intelligence. Another way of thinking about it is to imagine that you’ve made a new ruler to measure length. If you already have a good ruler, then the two rulers should produce similar results when you measure the lengths of different objects. If they don’t, then one of them is wrong. This form of validity is called concurrent validity.

Content validity is about whether or not a test samples adequately and representatively from the domain of interest. For instance, if you are constructing a test of mathematical ability, including only questions about addition while neglecting subtraction and other operations would not be a good idea. Likewise, if you were designing a test to measure reasoning ability, having only problems of one type is unlikely to give you a valid measure of reasoning.

Perhaps the most complex form of validity is construct validity. In simple (but complex) terms, construct validity is whether or not your test measures what it claims to measure. The two mains ways of determining this are to see whether it relates to the things it is supposed to be related to (e.g., virtually all measures of intelligence are correlated with each other) and whether it does not relate to things it shouldn’t (e.g., measures of intelligence generally do not show strong correlations with measures of personality).

Conclusions

Before we use a psychological test (or any kind of test), we need to be certain that we are doing the right thing. This is particularly important when we move away from research into areas where the test serves as the basis for an important decision, such as in human resources or organisational psychology. But how do we know we’re doing the right thing?

A good test must be able to satisfy three criteria:

Measurability

Reliability

Validity

These criteria are not independent. A reliable test is one that measures something consistently and accurately. A valid test is one that measures what it is supposed to measure. Therefore a test must be reliable before it can be valid. Yet the issue of measurability is even more important. An attribute (e.g., intelligence) must be measurable before we can measure it with a test. Moreover, it must be measurable in certain ways, such as being quantifiable and measurable on a rational scale, if we are to apply many of the advanced statistical techniques that are so popular. It is not uncommon for people to simply assume that something is measurable, yet measurability, much like reliability and validity, is something that must be demonstrated, not simply assumed.

Tests and assessment instruments can have a powerful impact on people. It is the responsibility of the people using those tests and assessment instruments to make sure that they are using the right tests in the right ways.

If you want to read more about my thoughts on writing, education, and other subjects, you can find those here.

Nálunk jobbára másképp hangzik a jól ismert kérdés az újsütetű ügyféltől. “És milyen megjelenést tudnak garantálni, kedves Gábor? Részemről pedig az a válasz hangzik el, amit már sokszor mondtam és most sem tudok mást mondani. “Garantálni nem tudunk eredményt, de nagy valószínűséggel, a téma ismeretében, tudásunkra és kapcsolatrendszerünkre alapozva azt tudom mondani, hogy …”. Ebből aztán mindig érdekes beszélgetések alakulnak, legtöbbször szerencsére mindkét fél számára megnyugtató módon. 

A mérés és mérhetőség kérdése a mai napig a sajtókapcsolati munka egyik kritikus területe. Miközben a média maga gyökeresen átalakul és a közösségi média nem kis részben átvette a hagyományos tömeg média szerepét, addig a PR szakma sajtókapcsolati része még ma is küzd avval, hogy mivel is mérje eredményességét. Abban egyetértés látszik, hogy a cikkek, megjelenések és különösen az online említések száma önmagában nem mérvadó. Ennél egy lépéssel előrébb járunk, ha a megjelenés számot tartalmi elemzéssel kötjük össze, megnézzük az általunk fontosnak tartott üzenetek és kulcs kifejezések jelenlétének gyakoriságát. Még tovább haladtunk, ha mindezt az ügyfelek versenytársainak megjelenéseivel is egybe tudjuk vetni, fel tudjuk mérni, hogy a médiában, az újságírók fejében és tollából hogyan is szerepeltünk. Ha azt akarjuk, hogy a sajtómunka eredményességének megítélésében egyetértés legyen az ügyfél és az ügynökség között, 5 fontos dologra érdemes figyelni.

1. Az ügyfélt az eredmények érdeklik, nem a befektetett munka. Az outputon legyen a szemed, ne az inputon.

2. Mindig előzetesen és pontosan egyezzetek meg, mi az elvárás: megjelenések nagyságrendjében, jellegében, tartalmában.

3. Ne felejtsd, PR ügynökségként ma már nem kizárólag információforrás vagy, hanem csatorna is. Ügynökségként, szakmai szereplőként számos “saját” médiát tudsz működtetni, ezt pedig érdemes az előzetes egyeztetésnél is figyelembe venni.

4. Az említés nem elég. Koncentrálj arra, hogy a fontos üzenetek és kulcs idézetek minél nagyobb számban jelenjenek meg.

5. Azokra a tényezőkre figyelj, és azokat fogadtasd el az ügyféllel, amelyekre a PR munka minősége valóban tud befolyásolni. Ne hagyd, hogy a PR szolgáltatás eredményességét az eladások növekedésével vagy a korábbinál több munkakereső jelentkezésével mérjétek. 

In our case the usual question from a new born client is somewhat different: “And what coverage can you guarantee, dear Gabor?” I can answer nothing else but repeat the answer I usually give to these questions: “We can not guarantee any concrete result, but, with great probability, in view of the actual issue, based on our expertise and media relations knowledge, we can deliver the followings…” Usually this leads to some interesting discussions, ending in an agreement satisfying the needs of both parties.

Still today, measurability is one of the critical areas of media relations work. Media itself went through some fundamental changes where social media, to a considerable extent, took over the role of traditional mass communication. At the same time media relations work still faces the well known dilemma regarding how to measure ifs results. There seems to be agreement that the number of clippings, specifically in the case of online mentionings, is far from any exact measurement. We are one step ahead if we also consider message penetration and quote penetration. We are even further on the road if all this is matched against the performance of the competitors and we are able to express where our client and topic is positioned and stands regarding share of mind. If, at the end of the day, we want to have a clear agreement between client and agency regarding the measurement of media work effectiveness, we need to focus on 5 aspects:

1. The client is interested in output not input. Focus on the results, not on the work involved.

2. Come to agreement regarding expectations, character, size and content of coverage before the actual work starts.

3. Keep in mind that as PR agency your are not only the source of information, but the channel as well. As a professional player in the field, an agency can create and use its own media. Worthwhile to remember when discussing with client.

4. Mentionings are not enough. Focus on having key message and quote penetration as  deep as possible.