#compositionality

Yeah, I'm a big fan of wild speculation.  =)  I just need to be careful to label it as such, so as not to mislead people.

As for my research, I'm in an NLP department, studying some combination of NLP and computational cognitive science.

I'm interested in semantics, but I think most existing approaches to semantics are doing it wrong.  I won't go into detail here, but the two main formal approaches to semantics are formal semantics, which is logic-based, and vector space semantics, which (as you might imagine) is linear-algebra-based.  I disagree with these models on how we should represent the meaning of a word.

In formal semantics, the meaning of a word is represented by a logical predicate, so the word "snow" might correspond to a predicate "snow(X)", which is true iff X is an instance of snow.  This has a number of problems: it requires everything to be "definitely snow" or "definitely not snow", so it doesn't allow for concepts with fuzzy boundaries, where things can be "sort of snow, but not really".  Also, formal semantics is mostly focused on the rules of compositionality -- how do you put these logical predicates together to get the meaning of the whole sentence?  Like, if you have a sentence "all snow is white", and each word has some individual logical meaning, how do you put these meanings together to get the full logical sentence, "forall X: snow(X) => white(X)"?  Because formal semantics is mostly focused on compositionality, it never really goes into details about what the predicates "snow(X)" or "white(X)" might look like.  They're treated as black boxes.  And I think that ignores some of the most interesting problems in semantics.

So that's formal semantics.  Vector space semantics treats the meaning of a word as a vector in some space.  So maybe fruits could be represented in a space whose dimensions are "size", "sweetness", and "color" or something like that.  (In practice, the vector representations of words are extracted automatically, so the dimensions won't be this simple or interpretable.)  Anyway, this is an interesting idea, but I don't think a single vector can fully capture the meaning of a word.  In particular, it doesn't really capture the variance within a concept.  (Or rather, a vector could capture that, but it would be an inelegant representation.)  Also, while some serious theoretical work has been done on vector space compositionality, a lot of the research on that topic is pretty hacky.  ("Compositionality" always means "how do we put the individual units of meaning together to get the meanings of entire phrases or sentences?", so "vector compositionality" is just "how do we put the individual vectors together to get a vector or some-other-object representing the meaning of the entire sentence?".)

So!  With all of that background, I can finally explain what my research is.  You see, I don't like how formal semantics or vector semantics model word meaning.  But there is a model of word meaning I like, which can be found in the computational cognitive science literature under the heading of "concept learning".  In concept learning, the meaning of a word is a probability distribution over instances of that word.  So, the meaning of a word like "snow" is a distribution over instances of snow; the meaning of a word like "desk" is a distribution over possible desks; and so on.  I think this model is great.  But for the most part, no one has studied how compositionality could work for this model.  The meaning of "red apple" would be a distribution over red apples, and the meaning of "a man walked to the store" would be a distribution over situations where a man walked to the store, but how does one compute these distributions from the distributions for the individual words?

Since few (if any) people were studying this, I figured it would make a good grad school topic.  So that's what I'm researching: compositionality for probabilistic concepts!  My advisor and I have a pretty cool model of this, though so far it's proven kind of intractable to work with.

I should note: I'm not the only researcher working on probabilistic semantics.  There's a small group of people who study it, and I met a bunch of them last summer at a workshop that my advisor organized.  Not all of them treat the meaning of a sentence as a distribution over instances of sentence, but all of them use probabilities in their semantic representations.

Probabilistic semantics is important, because the whole point of semantic representations is to serve as input to some reasoning system.  Which is why formal semantics is such a popular approach: it was developed at a time when reasoning was modeled as logical inference.  A logical representation of sentence meaning was great, because it could be plugged in straightforwardly to a logical reasoning engine.

But now we usually model reasoning as probabilistic inference.  This means we need some new meaning representation that can plug into our probabilistic reasoning systems.  Hence probabilistic approaches to semantics!

Recall the famous embeddings from Goedel, McKinsey, and Tarski (generating intuitionistic logic) and Thomason (generating Nelson's Logic of Constructible Falsity) into \(\mathsf{S4}\). As an example, the former is:

-\(A^{G}=\Box A\) for atoms \(A\)

-\( (\neg A)^{G}=\Box\neg(A^{G})\)

-\( (A\vee B)^{G}= A^{G}\vee B^{G}\)

-\( (A\wedge B)^{G}= A^{G}\wedge B^{G}\)

-\( (A\rightarrow B)^{G}= \Box(A^{G}\rightarrow B^{G})\)

The Thomason embedding is similar. We may also recall the Fregean Principle of Compositionality that complex propositions are constructed from simpler propositions.

I think that these embeddings capture a very correct compositional principle and reveal a picture that ought to be considered by even a classical logician. We can ask what these embeddings in general mean. I think the answer to this is that they make clear that propositions necessarily bear "preambles." 

What is a "preamble"? Well, the underlying thought is that a proposition is empty without taking some stand on how it is to be understood. The classical logician prefaces her utterances with the understanding that the propositions are to be interpreted as true in the actual world. The constructivist prefaces his utterances with a different claim, namely, that the propositions are provable. 

Thinking about the embeddings purely syntactically, prior to considering their ranges as modal logics, a clear picture of compositionality upon this background emerges. The symbol "\(\Box \)" simply represents a preamble. If there indeed are such implicit preambles, then it is clear, e.g., that the meaning of a formula \( A\rightarrow B\) is not a function of \(A\) and \(B\), but rather \(\Box A\) and \(\Box B\). And this is precisely what is captured by such embeddings.