#thinkingthrough

What's the proper comparison "control group" for design-by-analogy (DBA)? Do we really mean to assert that it is "better than all other idea-generative mechanism alternatives"? And if we compare DBA (or some form of creative stimulation), can we really compare it to a "no treatment" control group? If we argue that analogy is a fundamental process used in idea generation, then it stands to reason that we may be introducing noise by comparing "trained" DBA with "naive" DBA. Now perhaps training may help, but it is also not clear how much training is sufficient to matter. Furthermore, it is implausible to suppose that individuals in "no treatment" conditions aren't using other idea generation strategies.

Perhaps a better analogy than "treatment vs placebo" is strategies in games. In games there are often mutiple routes to "victory", and which strategy ends up being more effective depends at least on the nature of the problem space (opponent you are facing, map characteristics, resource distributions, etc.); other considerations can come in, such as the resources at your disposal (if I am playing Terran, what are my specific strengths?).

By analogy, in conceptual design (and creative idea generation more generally), I suspect that particular important features of problem spaces (upon which design problems can vary in systematic ways) can have important bearings on what kinds of idea generation strategies are more or less optimal. For instance, if a design problem space has "good solutions" distributed in a very sparse fashion, and the end state is very poorly specified, with few related benchmarks, a broader search strategy involving more distant analogies and remote association chains may yield a higher payoff than a more incremental search strategy. On the other hand, if "good solutions" are distributed such that incremental search strategies are at least in principle capable of landing you at "good solutions", then the broader search strategy will likely lose out by virture of wasted extra effort for search and mapping, with nothing to balance off the lossiness of the approach (finding more duds, f.i.).

Essentially one important question that arises for the agent is "Why should I explore rather than exploit"? So perhaps one hypothesis is that certain design problems, by virtue of their structure, may favor exploration vs exploitation-focused strategies, and might therefore differ in the optimal mix of conceptual/functional distance of stimuli in terms of the net payoff for the design outcome.

Another aspect of the analogy would concern how the particular skills or strengths of the game player might influence what kinds of strategies are more or less useful, which maps onto how individual difference variables for the designer might differentiate better or worse idea generation strategies, over and above effects of the design problem space structure. Some potentially important variables might be individual conceptual history (what's the structure of my semantic network?), drive, intelligence, etc. I suspect the first one could be very important.

With this overall analogy in mind, perhaps a more informative contrast would be between a "naive" and "informed" approach to the use of design strategies. One implication of arguing that these "tuning" variables are crucial in determing what strategies are useful is that a designer who is attuned to these relationships between his own skill set, the structure of the problem space, and the optimality of available strategies, will be more successful in the long run than a designer who is naive and essentially uses strategies "at random".

This is a rather strong hypothesis/prediction. How can we test this theory about the relationship between design problem spaces, designer characteristics, and design strategies?

1) In-depth qualitative studies of expert designers

2) Computational simulations

3) Careful experimental manipulation of design problem space parameters and comparison of design strategy effectiveness across parameter settings