The Theory of Minimal Description: A Framework for Semantic Organization in Natural Languages

Abstract

The Theory of Minimal Description (TMD) studies how a language encodes a concept in the fewest words that still pick it out exactly. A minimal description must be understood both ways: the speaker's words identify one concept, and no other concept fits those words. It must collapse when any word is removed, it must stay inside the shared vocabulary of the language, and it must connect the concept to the rest of the language's web of meaning. From these four principles the theory derives a direction in the definitional network of a language, from specific words toward a small set of primitives that are learned by experience, and it proves that this direction is what keeps chains of definitions finite. It then measures descriptions in bits, using the surprisal of words, and from the measurements it states three testable proposals: that concepts in one semantic category share a characteristic information density, that categories can be recovered from that density, and that the total information carried by the working vocabulary of a speech community stays close to a constant while its words change. The same machinery treats metaphors as compression and treats expertise and culture as settings that change which description is minimal.

Minimal Description in Plain Terms

In the party game Taboo, you have to make your teammate say a target word without saying it yourself, and the fewer clue words you need, the better you are playing. "Domesticated animal that barks" gets "dog". Drop "barks" and the phrase covers cats and horses; drop "domesticated" and it admits jackals; drop "animal" and it stops describing anything. That is the whole idea of a minimal description. It is the shortest set of words that makes a listener land on exactly one concept, and it is minimal in a strict sense: take any word away and the description stops working.

Two facts about this game matter. First, the right clue depends on who is listening. Two builders can say "joist" to each other; one of them talking to a customer has to say "the beam that holds up the floor". The description is still minimal, and minimal for that listener. Second, the clues must themselves be understood. If you look up a word in a dictionary, its definition uses other words, and looking those up leads to still other words. The chain has to end somewhere, and it ends at words you never learned from a definition: words like "see", "big", "before", and "want", which you learned by pointing and by living. Every dictionary rests on such a base, and the theory shows that every chain of definitions must point toward that base or the dictionary would go in circles.

Once descriptions are counted in words, and words are weighed by how surprising they are, a language turns into something that can be measured. Concepts of the same kind, such as vehicles or colors, turn out to be described at a similar density of information. A good metaphor packs a long literal description into a short phrase. And a language as a whole behaves like a city with a fixed housing stock: new words move in only as old words move out or are compressed, so the total load on its speakers stays about the same. These are the claims the rest of the article makes exact.

1. Introduction

Language lets ideas be formed, exchanged, and passed on, and much of that power comes from economy. A speaker rarely says everything that is true of a thing; the speaker says what is enough. The Theory of Minimal Description asks what "enough" means when a concept has to be communicated without error inside one language, and it builds a framework around the answer: for every concept there is a set of words that is necessary and sufficient to identify it, and that set can be analyzed with the tools of information theory.

The question has a history. Aristotle defined by naming the genus and the difference, "a human is a rational animal" (Barnes 1984). Grice (1975) stated the pragmatic rule that a speaker should be as informative as required and no more. In computational linguistics, Dale and Reiter (1995) studied how to generate a referring expression that distinguishes one object from the others in view, and they showed that finding the shortest such expression is computationally hard. Wierzbicka (1996) proposed that all meanings can be paraphrased in a few dozen semantic primes. Studies of dictionary graphs found that every word in a dictionary can be defined from a small kernel of words, and that the kernel contains a still smaller core (Blondin Massé et al. 2008; Vincent-Lamarre et al. 2016).

TMD joins these lines and adds four things. It treats the minimal description of a concept, within a whole language, beyond objects in view, as an object with a formal definition and a test for minimality, the collapse property. It proves that the network of definitions has a direction, and that the direction is what makes definitions terminate. It measures descriptions in bits, so that concepts, categories, and languages can be compared. And it states three empirical proposals about semantic density, emergent categories, and the information equilibrium of a lexicon that follow from the measurements.

Universe 00110000

2. Core Principles

Four principles define a minimal description and constrain how it can be found: Bidirectional Uniqueness, the Collapse Property, Language Containment, and Network Connection.

2.1 Bidirectional Uniqueness

Fix a language L. Let ℂ be the set of concepts that speakers of L can communicate, let W_L be its vocabulary, and let D_L be the set of descriptions, the finite phrases that can be formed from W_L. An interpretation function ι_L sends each description to the concept a competent listener takes it to mean, or to the value ⊥ when the description is ambiguous or picks out nothing:

ι_L: D_L → ℂ ∪ {⊥}

A description d describes the concept c when ι_L(d) = c. Because ι_L is a function, this already says that no other concept fits d: a description that could mean two things is sent to ⊥. The two directions of communication are therefore built into one condition. From concept to description, a speaker who holds c chooses a d with ι_L(d) = c. From description to concept, a listener who receives d recovers c and nothing else.

Perfect bidirectional uniqueness is an ideal that real communication approaches. Speakers and listeners differ in knowledge, so the description that achieves uniqueness for one pair can fail for another. Section 4 makes the listener explicit. Zipf (1949) described the tension the ideal resolves: a speaker would prefer few words with many meanings, a listener would prefer many words with one meaning each, and a language settles between them. Bidirectional uniqueness is the listener's side of that bargain stated as a requirement.

2.2 Collapse Property

A description that identifies a concept can still carry dead weight. Minimality forbids it.

Removing any single word must break the description: either it becomes ambiguous or empty, so that ι_L(d − w) = ⊥, or it slides to a different concept c′. "Domesticated animal that barks" is a minimal description of dog. Remove "barks" and the phrase covers cats and horses; remove "domesticated" and it admits jackals; remove "animal" and it stops being a description at all. Every word does work.

The collapse property is a local test: it checks each word, one at a time. A description can pass it and still fail to be the shortest description of its concept, because a different phrase might reach uniqueness with fewer words. Section 5 adds the global condition. In the literature on referring expressions the local condition appears as the absence of redundant attributes, and Dale and Reiter (1995) showed that their incremental algorithm can leave redundancy in place while a search for the shortest description removes it at exponential cost. The collapse property is what a speaker can check without that search.

2.3 Language Containment

Each language is treated as a self-contained semantic system. A minimal description is built from words that the language community commonly understands. It does not lean on external codes, on translations, or on private notation. Two consequences follow.

• Word meanings are culturally embedded, so the minimal description of a concept is shaped by the culture that speaks the language. What a community already knows can be left unsaid (Section 4.2).
• Languages lexicalize the world differently, so the same concept can need different numbers of words, and different grammatical shapes, in different languages. A concept that one language names with a single word another describes with a phrase. Minimal descriptions are language-relative, and comparisons across languages must compare the descriptions themselves.

2.4 Network Connection Principle

A description is valid only when it ties the concept into the wider web of meaning of the language. A valid minimal description must satisfy three requirements.

• It connects the concept to at least one concept that is distinct from it and from its immediate synonyms. A description made only of synonyms stays inside a closed loop.
• It avoids circular definition. Defining "self" as "I" and "I" as "self" satisfies uniqueness in a narrow sense and tells a learner nothing, because neither term is anchored outside the pair.
• It keeps pathways open to other semantic domains, so that the concept can be related to neighboring concepts through the words of its description.

Three consequences follow. Network connection puts a lower bound on the length of minimal descriptions: a phrase that must reach outside the concept's synonym cluster usually needs more than one word. It explains how new concepts enter a language: a new minimal description must attach to the existing network before the new word is usable, which keeps the vocabulary coherent as it grows. And it explains part of the variation across cultures: communities organize their semantic networks differently, so the same concept is connected, and therefore described, in different ways. Section 3 shows that the ban on circularity is also what gives the network its direction.

3. Semantic Directionality

Dictionaries define words with other words. Following the definitions from any entry leads through more and more general words and stops at words that are learned by experience. TMD states this as a property of the definitional network, derives it from the principles of Section 2, and then reads its consequences for how a language organizes meaning.

3.1 The Definitional Network

Primitives are the words whose meaning a speaker acquires without a verbal definition: by pointing, by doing, by feeling. Wierzbicka (1996) argued that a set of about sixty semantic primes, words such as I, you, someone, something, want, know, good, big, before, and because, suffices to paraphrase every meaning in every language studied, and Goddard and Wierzbicka (2002) tested the claim across many languages. Dictionary graphs show the same structure from the other side. Blondin Massé et al. (2008) and Vincent-Lamarre et al. (2016) found that every word in a dictionary can be reached by definition from a kernel of about a tenth of its entries, and that the kernel itself contains a smaller core of words that define one another. The primitives of TMD are the words at which definition stops.

3.2 The Directionality Theorem

The proposition turns an observation about dictionaries into a consequence of the principles. Minimal descriptions always point downward in depth, toward the primitives. This is the semantic gravity of a language: a specific concept is pulled toward the general words that define it, and those toward the words learned by experience. The converse also holds. If some description pointed upward or sideways, following definitions could circle forever, and the collapse and uniqueness tests could never be completed, because a listener would never reach words whose meaning is already secured.

3.3 Levels of the Hierarchy

Depth sorts the vocabulary into levels, and three of them have distinct roles.

• Primitives, at depth 0, are the anchors. They are few, they are learned early, and they are reached by many chains, so they have the highest connectivity in the graph.
• Basic-level words, at small depth, are the workhorses of description. Rosch et al. (1976) showed that categories at one level, chair rather than furniture or kitchen chair, are the ones people name first, learn first, and name with the shortest words, and that members at that level share the most attributes. Basic-level words are where the genus of most definitions comes from.
• Specialized words, at large depth, carry the densest information. Their minimal descriptions are built from words lower in the hierarchy, and a single one of them can stand for a whole phrase of basic-level words.

Definition has the classical shape of genus and difference (Barnes 1984). The genus is one word from a lower level, and the differences are the words that separate the concept from its siblings under that genus. So the downward step of a definition is a single word, the genus, while uniqueness is supplied by the differences. A general word cannot serve as a minimal description of a specific one by itself: "animal" does not identify dog, because uniqueness fails. The asymmetry between levels is therefore an asymmetry of roles. Specific words are defined through general ones, and general ones are defined through still more general ones, until the primitives, which are not defined at all.

3.4 Stability of the Network

Directionality gives the network two stability properties, and both follow from Proposition 3.3.

• Anchoring. Every chain of definitions ends at a primitive, so there is no infinite regress of meaning. Many chains converge on the same few primitives, which makes those words robust: a change elsewhere in the vocabulary leaves them and the chains that reach them intact.
• Growth accommodation. A new specific word enters the language by receiving a minimal description in existing, lower-depth words. The new vertex has outgoing edges only, so it creates no cycle, and the depth of every existing word is unchanged. Vocabulary can grow without disturbing the foundation, which is how technical vocabularies are absorbed.

These properties match what has been measured. Steyvers and Tenenbaum (2005) found that semantic networks built from word associations, from WordNet, and from a thesaurus are small-world networks in the sense of Watts and Strogatz (1998), with short paths between any two words and dense local clustering, and that their connectivity is spread very unevenly, with a few hub words of very high degree. Hubs of high degree at low depth are what anchoring predicts, and a growth process that attaches new words to existing ones is the model they proposed for how such networks arise.

3.5 Efficiency and Cognition

Directionality lets a language satisfy competing demands at once. Reference is efficient, because a specialized word packs a phrase of basic-level words into one token; a specialist says "stenosis" where a phrase would otherwise be needed. Grounding is secure, because every word descends to the primitives. Learning is staged: a learner who holds the primitives and the basic level can acquire any specialized word by its description, which is how expertise is taught, and it is why Chi, Feltovich, and Glaser (1981) found experts sorting problems by deep principles while novices sort by surface features. The expert's vocabulary sits higher in the hierarchy and compresses more.

The same structure carries culture. A community's semantic network preserves the categories it draws, the differences it counts as decisive, and the primitives it grounds meaning in. New technical terms can be added without remaking the network, and specialized knowledge stays accessible in principle, because every specialized word has a description in words that non-specialists share. Directionality is a functional adaptation: it organizes meaning so that a language is both precise and learnable.

Universe 00110000

4. Context, Expertise, and Culture

The interpretation function of Section 2.1 belongs to a listener. Write ι_L,A for the interpretation shared by an audience A, and MD_L,A(c) for the minimal description of c relative to that audience. Minimal descriptions are not absolute. They are adjusted to what the audience already holds, and the adjustment follows two rules.

4.1 Expertise

Between experts, a single technical word is often a minimal description, because both parties hold its definition. Toward a lay audience the same concept must be unpacked into lower-depth words until uniqueness is restored. Consider the concept neuron.

• For neuroscientists: "electrically excitable cell that transmits nerve impulses". Each word is secured for that audience, and the phrase identifies the concept.
• For a general audience: "cell in the brain and nerves that sends electrical signals to other cells, forming networks that process information". The added words replace terms the audience does not hold with terms it does.

Both phrases are minimal for their audience. What changes is ι_L,A, and with it the set of words that count as secured. This is the form that the expert-novice difference takes in TMD, and it is why a speaker must model the listener before choosing a description, the common ground that Clark (1996) placed at the center of language use.

4.2 Culture and Implicit Information

Shared cultural knowledge is implicit information, and implicit information shortens descriptions. A concept that a culture holds in common needs fewer words inside that culture, because the audience supplies the rest. The Japanese concept Golden Week shows the effect.

• In Japan: "the spring holidays at the end of April". The audience knows that a cluster of national holidays falls there and that the cluster has a name.
• Elsewhere: "a series of Japanese national holidays in late April and early May that forms an extended vacation period". The words "Japanese", "national", and "extended vacation period" replace what the Japanese audience already knew.

Minimal descriptions are therefore optimized twice: for length within the language, and for the knowledge of the audience. Section 8 shows that metaphor uses the same lever, a shared source domain, to shorten descriptions further.

5. Formalization and Information Content

The principles of Section 2 fix what counts as a minimal description. This section defines the minimal description function and gives it a measure in bits, so that descriptive efficiency can be compared across concepts and across languages.

5.1 The Minimal Description Function

Write |d| for the number of words in a description d. The minimal description function selects, among the descriptions that identify c, one of shortest length:

MD_L(c) = arg min_{d ∈ DL} { |d| : ι_L(d) = c }

Three remarks make the definition precise. The condition ι_L(d) = c already contains uniqueness, because an ambiguous description is sent to ⊥ (Section 2.1). A shortest description satisfies the collapse property automatically, since removing a word from it would produce a shorter description of c, which cannot exist; the collapse property is the local test, and shortest length is the global condition it approximates. And the minimum need not be unique: several phrases of the same length can identify c, and TMD treats them as equivalent minimal descriptions, to be chosen among by the audience rules of Section 4. The Network Connection Principle is imposed as a constraint on the set over which the minimum is taken.

The definition transfers the principle of minimum description length (Rissanen 1978) from data to meaning: the complexity of a concept is the length of the shortest phrase that identifies it. Dale and Reiter (1995) proved that the corresponding search over attributes of objects in view is NP-hard, and the same holds here, so real speakers approximate MD_L with incremental strategies and check the result with the collapse test.

5.2 Bits per Word

Word count treats every word as equal. Words are not equal: "the" is expected almost everywhere, and "stenosis" almost nowhere. Shannon (1948) gave the measure that respects the difference. If a word w occurs with probability p(w) in the language, its information content, its surprisal, is

I(w) = −log₂ p(w)

and the information content of a description is the sum over its words:

I(d) = ∑_i=1ⁿ I(w_i) = −∑_i=1ⁿ log₂ p(w_i)

The probabilities come from a corpus of the language. The simplest choice is unigram frequency. A finer choice is the probability of each word given the words before it, which is the quantity that predicts human reading effort word by word (Hale 2001; Levy 2008). Under the finer choice the words of a description share information, and I(d) counts each word only for what it adds.

A useful special case sets a fixed defining vocabulary V and treats its words as equally likely. Then every word costs log₂|V| bits, and a prefix-free code assigns each word a binary string of length ⌈log₂|V|⌉, so that

I(d) = n × ⌈log₂|V|⌉

This is the coding-theoretic lower bound for n symbols drawn from a set of size |V|. Learner's dictionaries already work this way: the Longman Dictionary of Contemporary English writes every definition with a controlled defining vocabulary of about two thousand words, and Ogden (1930) built Basic English from 850. With |V| near two thousand, each defining word costs eleven bits, and a five-word definition carries 55 bits. The uniform bound is what the surprisal measure reduces to when nothing is known about word frequencies, and it is the right measure when the question is how many distinct defining words a language needs.

5.3 Comparing Concepts and Languages

I(MD_L(c)) is a lower bound on the bits a speaker of L must transmit to identify c from scratch, and it is comparable across concepts within a language and, with the same probability model, across languages. Two concepts with equal word counts can differ in bits when one is described in rarer words, and two languages can describe the same concept in different word counts but similar bits, which is the quantity the cross-linguistic hypotheses of Section 7 are about. Piantadosi, Tily, and Gibson (2011) found, across eleven languages, that the length of a word tracks its average information content in context better than its frequency. Languages already spend letters in proportion to bits; TMD applies the same accounting to whole descriptions.

Universe 00110000

6. Recursive Properties of Minimal Descriptions

Every word in a minimal description has a minimal description of its own. The meaning of a phrase therefore rests on nested layers of description, and TMD measures the whole stack.

6.1 Recursive Information Content

Each word contributes its own surprisal and the recursive content of its description. Primitives contribute only their own surprisal, because their meaning is not carried by words.

6.2 Recursive Minimality

A description d of c is recursively minimal when three conditions hold: d is a minimal description of c in the sense of Section 2; every word of d has a minimal description in L that satisfies the same principles; and I_total(d) is least among the descriptions of c that satisfy the first two conditions. The third condition prefers, among equally short descriptions, the one built from words that are themselves cheap to ground. A description in basic-level words is recursively cheaper than one of the same length in specialized words, which is the quantitative form of the advice to define in plain terms.

6.3 Termination

Recursion raises the question of regress: does the stack of descriptions bottom out? Proposition 3.3 answers it, and the answer is exact.

Circular definition is therefore worse than uninformative: it is the one thing that makes the information content of a language undefined. Primitives are the termination points, and the convergence condition can be stated directly: for every concept c there is a k such that the words reached after k applications of MD_L include a primitive, and every chain from c reaches one. Cognitive linguistics reaches the same base from the other direction: the small set of primes of Wierzbicka (1996) is a candidate for P, and the dictionary kernel of Vincent-Lamarre et al. (2016) is its empirical image.

6.4 Computation

A computational model can be built from lexical resources such as WordNet (Miller 1995), whose hypernym links supply the genus of each word, together with a corpus for the probabilities. Because a full recursion visits the whole dictionary, a depth-limited version is used in practice:

I_total,k(d) = ∑_i=1ⁿ [ I(w_i) + I_total,k−1(MD_L(w_i)) ],   I_total,0(d) = 0

where k is the maximum recursion depth. Take chair, with the minimal description "a seat for one person, with a back". Its content words have descriptions of their own: seat, "a thing made for sitting on"; person, "an individual human being"; back, "the part that supports a sitter's spine". With k = 2 the model counts the description of chair and the descriptions of its words, and stops. Raising k approaches I_total from below, and the value of k at which the estimate stabilizes measures how deep the meaning of a word reaches into the language. Comparing the model with human judgments of definitional relatedness is the natural test.

7. Information Density and the Semantic Density Regularity Principle

With descriptions measured in bits, TMD can ask how information is distributed across concepts, and it finds a regularity that ties information density to semantic categories.

7.1 Density Metrics

Let c be a concept with minimal description d and with a single word w that names it in L. When a concept is named by a multi-word term, a representative head word is used. Two metrics compare the description with the name.

• Bit Ratio: BR(c) = I(d) / I(w). The ratio of the information in the description to the surprisal of the name. A high Bit Ratio means the name compresses a great deal: a rare word standing for a long description has a low ratio, a common word standing for the same description a high one.
• Density Ratio: DR(c) = ρ(d) / I(w), where ρ(d) = I(d) / |d| is the average information per word of the description. The Density Ratio compares the information density of the description with the information of the name. A value near 1 means the name and the words that define it carry information at the same rate.

Both ratios are defined because I(w) > 0 for every word that occurs with probability below 1. Under the uniform bound of Section 5.2 the Bit Ratio reduces to |d| and the Density Ratio to 1 for every concept, so the metrics carry information only when word probabilities are used. That is the reason surprisal, and not word count alone, is the measure of Section 5.

7.2 Hypotheses

• Abstractness. Abstract concepts have higher Bit Ratios than concrete ones. Their descriptions must do more work to reach uniqueness, because they are less directly grounded in perception, while their names are often common words.
• Category clustering. Concepts in one semantic category have similar Density Ratios, and the variance of DR within a category is smaller than the variance across categories.
• Cross-linguistic invariance. Density Ratios agree across languages more closely than word counts do, because the ratio removes the language's lexicalization choices and keeps the information structure of the concept.

7.3 The Semantic Density Regularity Principle

The principle is an empirical hypothesis, and its rationale comes from the shape of definitions. Members of one category share a genus and differ by differentiae of similar specificity, so their descriptions have similar lengths and similar information. Their names sit at the same level of the hierarchy, so the names have similar surprisal, which is the basic-level effect of Rosch et al. (1976). The ratio of two similar quantities is similar. Within vehicles, car, truck, and bus are each "a road vehicle with an engine, for carrying" something, and each name is a common basic-level word, so their Density Ratios agree; a category with a different genus and a different level of naming will settle at a different ratio.

The threshold relation is not an equivalence: two concepts can each be within ε_s of a third while differing from each other by more. Category membership is therefore defined by clustering, and density is a necessary condition for it. Two concepts count as members of one category when they are semantically similar above a threshold θ and their densities agree:

C(c₁, c₂) = 1 if sim(c₁, c₂) ≥ θ and |DR(c₁) − DR(c₂)| ≤ ε_s; 0 otherwise

A semantic category is a maximal set of concepts in which every pair satisfies C = 1. Colors and emotions can have similar densities and still form two categories, because they fail the similarity test; two colors with different densities fail the density test.

Three consequences are testable. Category boundaries should appear as discontinuities in Density Ratio when concepts are ordered by similarity. If the regularity holds across languages, it points to a cognitive constraint on how meaning is packed, shared by speakers of unrelated languages. And the clusters recovered from density and similarity should agree with the categories speakers recognize without any taxonomy being imposed in advance, which is the emergence claim of Section 9.

8. Metaphorical Compression

Metaphor conveys complex ideas in few words. TMD treats it as a compression mechanism and measures the compression.

8.1 Metaphor as Compression

A conceptual metaphor is a mapping M: T → S from a target domain T to a source domain S, under which the target is understood and expressed in the vocabulary of the source (Lakoff and Johnson 1980). "Time is money" maps time onto a resource that can be spent, saved, and wasted. The mapping lets a short phrase in source vocabulary stand for a longer literal description of the target, and the gain is the Compression Ratio:

CR = |MD(T_literal)| / |MD(T_metaphorical)|

where the numerator is the word count of the minimal literal description of the target concept and the denominator the word count of its minimal metaphorical description. "You are wasting my time" has four content words. A literal description of the same content, "you are using the hours I have available without producing anything I value", has ten. The ratio is about 2.5, and the four words work only because the audience already holds the mapping. Metaphor is therefore the strongest case of Section 4: a shared source domain is implicit information, and it shortens descriptions by a measurable factor. The same ratios can be computed in bits with the measure of Section 5.2.

8.2 An Optimal Range

Compression has a cost. A metaphor that compresses too little offers no advantage over the literal phrase; one that compresses too much becomes obscure, because the listener has to reconstruct too much of the target from the source. TMD predicts that the metaphors a culture keeps lie in a band of Compression Ratios that balances brevity against the effort of decoding. The prediction can be tested by computing ratios for a large sample of conventional metaphors and relating them to prevalence, memorability, and cross-cultural portability. Metaphors near the band should be the most widespread and the most durable; metaphors far above it should be rare, literary, or short-lived.

8.3 Consequences

• Metaphors are efficient descriptions of abstract concepts, which is where the Bit Ratios of Section 7 are highest and where compression is most valuable.
• Their effectiveness depends on culture, because the source domain must be familiar. A mapping whose source is unknown to the audience compresses nothing, which explains why metaphors travel unevenly across cultures.
• Their optimal range is a cognitive balance between information density and processing load, the same balance that sets the length of minimal descriptions in general.

Universe 00110000

9. Information Theory and Emergent Categories

9.1 Descriptions as Programs

Algorithmic information theory measures the complexity of an object by the length of the shortest program that produces it (Kolmogorov 1965; Li and Vitányi 2008). TMD is the semantic counterpart: the complexity of a concept is the length, in words or in bits, of the shortest phrase that identifies it. Words are the units of the code, as bits are the units of a program, and the minimal description is the shortest program a listener can run to reach the concept. Three consequences follow from the analogy.

• Minimal word count, or better I(MD_L(c)), is a measure of semantic complexity: a concept that needs a long description is more complex, as an object that needs a long program is.
• Language is an evolved code. Its rules for building phrases balance precision against length, as a good code balances distinctness against length, and Zipf (1949) showed that the balance between speaker and listener effort leaves the statistical signature that Ferrer i Cancho and Solé (2003) later derived from a least-effort model.
• The analogy has a limit that the surprisal measure respects: program length is defined up to a constant that depends on the machine, and description length is defined up to the audience, so comparisons are meaningful within a fixed audience and probability model.

9.2 Categories as Emergent Clusters

Linguistic theory often starts from a classification and asks how words fit it. TMD reverses the order. Categories are recovered from measurable properties of descriptions, by the membership function of Section 7.3, and a category is a cluster of concepts whose descriptions have similar information density and whose meanings are similar. Nothing about the classification is fixed in advance. Applied to color words, the procedure should return the color terms as one cluster; applied to emotion words, another; and the boundaries between clusters should coincide with the categories that speakers recognize. Where they do not coincide, the measurement has found a category that the classification missed, or a classification that the language does not support, and both are results. The claim is quantitative and testable, which is what distinguishes it from a taxonomy.

10. Semantic Density Regularity and Linguistic Information Equilibrium

The density regularity of Section 7 governs categories. A second regularity governs the lexicon as a whole, and the two together describe a language as a self-regulating system.

10.1 Regularity at the Category Level

Categories emerge from information density without an imposed classification, and the thresholds ε_s that define them are themselves regular: categories at the same level of the hierarchy have similar thresholds, because their members are named and defined at similar levels. Regularity within categories reduces the cost of processing them, since one decoding strategy serves all members, and this economy is the reason the regularity persists.

10.2 The Linguistic Information Equilibrium Hypothesis

The hypothesis concerns the vocabulary a community actually uses, and the difference from the union of everything ever printed matters, because the printed lexicon of English grew through the twentieth century while many words fell out of use (Michel et al. 2011), and Petersen et al. (2012) found that the rate at which new words enter a language falls as the language expands. A growing archive and a bounded working vocabulary are compatible, and the hypothesis is about the second.

Change in the total is governed by three processes:

ΔI_compression(t) + ΔI_obsolescence(t) ≈ ΔI_expansion(t)

• Compression: metaphor, semantic bleaching, and the shortening of descriptions reduce the information carried by existing words.
• Obsolescence: words that leave use take their information with them.
• Expansion: new words for new concepts, techniques, and practices add information.

Equilibrium holds when the losses balance the gains, so that the total stays near K while the vocabulary turns over.

10.3 A Dunbar-like Limit

What fixes K is cognition. Dunbar (1992) argued that the size of stable social groups is bounded by neocortex size, and the bound is a limit on the relationships a person can maintain. The equilibrium hypothesis proposes a limit of the same kind on the information a community's vocabulary can carry: a bound on the total load of words and definitions that speakers can learn, hold, and use with reliable uniqueness. The bound applies to total load, and a word count is only a proxy for it. Exceeding it would cost learnability and communicative efficiency, so compression and obsolescence act as restoring forces against expansion. The equilibrium is the adaptation that keeps a language inside its speakers' capacity.

10.4 Equilibrium at Three Levels

• Category level: density regularity keeps members of a category at a common information density, which economizes processing and retrieval.
• Lexical level: information equilibrium keeps the total load of the working vocabulary near K through turnover.
• Cognitive level: both regularities express the same limit on memory and processing in the speakers of the language.

Together the two principles explain how a language absorbs new domains while staying coherent, why similar categories appear in unrelated languages, why categories have the structure they have, and why economy is a design force in language. The tests are concrete. Density regularity is tested on lexical resources with a corpus probability model. Equilibrium is tested on diachronic corpora by computing I_total for the working vocabulary of successive decades with one defining vocabulary held fixed, and the hypothesis predicts a bounded series with turnover under it.

11. Conclusion

The Theory of Minimal Description gives semantic organization a formal core. A minimal description is the shortest phrase that identifies a concept for a listener, tested by the collapse of the description when any word is removed, contained in the language, and connected to its network. The network of definitions has a direction, proved from the ban on circular definition: every description points toward a small set of primitives, every chain of definitions terminates, and the recursive information content of a language is finite exactly when this holds. Descriptions can be measured in bits by the surprisal of their words, and the measurement supports three proposals that can be tested against lexical resources and corpora: members of a semantic category share an information density, categories can be recovered from density and similarity without an imposed taxonomy, and the working vocabulary of a speech community carries a nearly constant total of information while its words turn over.

The framework is language-relative and audience-relative by construction, so it applies to expertise, to culture, and to metaphor with the same equations. Its applications reach natural language processing, where minimal descriptions and depth-limited recursion are computable from existing resources; education, where staged learning follows the depth of the hierarchy; cross-cultural communication, where implicit information can be counted; and cognitive modeling, where the equilibrium hypothesis states a limit that can be measured.

References

• Barnes, Jonathan, editor. The Complete Works of Aristotle: The Revised Oxford Translation. Princeton University Press, 1984.
• Blondin Massé, Alexandre, Guillaume Chicoisne, Yassine Gargouri, Stevan Harnad, Olivier Picard, and Odile Marcotte. "How Is Meaning Grounded in Dictionary Definitions?" Proceedings of the 3rd Textgraphs Workshop on Graph-Based Algorithms for Natural Language Processing, Coling 2008, 17–24.
• Chi, Michelene T. H., Paul J. Feltovich, and Robert Glaser. "Categorization and Representation of Physics Problems by Experts and Novices." Cognitive Science 5, no. 2, 1981, 121–152.
• Clark, Herbert H. Using Language. Cambridge University Press, 1996.
• Dale, Robert, and Ehud Reiter. "Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions." Cognitive Science 19, no. 2, 1995, 233–263.
• Dunbar, R. I. M. "Neocortex Size as a Constraint on Group Size in Primates." Journal of Human Evolution 22, no. 6, 1992, 469–493.
• Ferrer i Cancho, Ramon, and Ricard V. Solé. "Least Effort and the Origins of Scaling in Human Language." Proceedings of the National Academy of Sciences 100, no. 3, 2003, 788–791.
• Goddard, Cliff, and Anna Wierzbicka, editors. Meaning and Universal Grammar: Theory and Empirical Findings. 2 vols., John Benjamins, 2002.
• Grice, H. P. "Logic and Conversation." In Syntax and Semantics 3: Speech Acts, edited by Peter Cole and Jerry L. Morgan, Academic Press, 1975, 41–58.
• Hale, John. "A Probabilistic Earley Parser as a Psycholinguistic Model." Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics, 2001, 159–166.
• Kolmogorov, A. N. "Three Approaches to the Quantitative Definition of Information." Problems of Information Transmission 1, no. 1, 1965, 1–7.
• Lakoff, George, and Mark Johnson. Metaphors We Live By. University of Chicago Press, 1980.
• Levy, Roger. "Expectation-Based Syntactic Comprehension." Cognition 106, no. 3, 2008, 1126–1177.
• Li, Ming, and Paul Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. 3rd ed., Springer, 2008.
• Longman Dictionary of Contemporary English. 6th ed., Pearson Education, 2014.
• Michel, Jean-Baptiste, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, The Google Books Team, Joseph P. Pickett, et al. "Quantitative Analysis of Culture Using Millions of Digitized Books." Science 331, no. 6014, 2011, 176–182.
• Miller, George A. "WordNet: A Lexical Database for English." Communications of the ACM 38, no. 11, 1995, 39–41.
• Ogden, C. K. Basic English: A General Introduction with Rules and Grammar. Kegan Paul, Trench, Trubner, 1930.
• Petersen, Alexander M., Joel N. Tenenbaum, Shlomo Havlin, H. Eugene Stanley, and Matjaž Perc. "Languages Cool as They Expand: Allometric Scaling and the Decreasing Need for New Words." Scientific Reports 2, 2012, 943.
• Piantadosi, Steven T., Harry Tily, and Edward Gibson. "Word Lengths Are Optimized for Efficient Communication." Proceedings of the National Academy of Sciences 108, no. 9, 2011, 3526–3529.
• Rissanen, J. "Modeling by Shortest Data Description." Automatica 14, no. 5, 1978, 465–471.
• Rosch, Eleanor, Carolyn B. Mervis, Wayne D. Gray, David M. Johnson, and Penny Boyes-Braem. "Basic Objects in Natural Categories." Cognitive Psychology 8, no. 3, 1976, 382–439.
• Shannon, C. E. "A Mathematical Theory of Communication." Bell System Technical Journal 27, 1948, 379–423 and 623–656.
• Steyvers, Mark, and Joshua B. Tenenbaum. "The Large-Scale Structure of Semantic Networks: Statistical Analyses and a Model of Semantic Growth." Cognitive Science 29, no. 1, 2005, 41–78.
• Vincent-Lamarre, Philippe, Alexandre Blondin Massé, Marcos Lopes, Mélanie Lord, Odile Marcotte, and Stevan Harnad. "The Latent Structure of Dictionaries." Topics in Cognitive Science 8, no. 3, 2016, 625–659.
• Watts, Duncan J., and Steven H. Strogatz. "Collective Dynamics of 'Small-World' Networks." Nature 393, 1998, 440–442.
• Wierzbicka, Anna. Semantics: Primes and Universals. Oxford University Press, 1996.
• Zipf, George Kingsley. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949.