Language models and contextual meaning / Embeddings and semantic similarity
Use context to interpret meaning, then test if the passage answers.
Representations can help a system compare language, but a high similarity result cannot establish truth or complete the reader’s task. Separate the model, its training, the retrieval task and the evidence needed for an answer.
Distinguish the text from its numerical representation.
An embedding is a numerical representation produced by a model and its chosen processing method. It can stand for a token, passage, question or other unit. The numbers are useful only in relation to the model, task, input and comparison method that produced them. They are not the text’s factual evidence or a universal quality label.
Two systems can represent the same passage differently because they use different training, tokenization, context limits, pooling or task adaptation. Do not compare numbers across unrelated vector spaces or treat a similarity threshold from one system as a general SEO rule. A representation can preserve useful patterns while losing a decisive condition.
Worked context example: the word bank has different roles.
Compare these invented sentences: “The boat reached the bank beside the river” and “The customer called the bank about a loan.” The surrounding words indicate a river edge in the first and a financial institution in the second. A shared written word does not establish a shared referent or user task.
A contextual model can produce different token representations when the surrounding text changes. That is the relevant idea behind contextual language understanding; no vectors were computed for this example. A reader asking where to land a boat should not receive borrowing instructions merely because both passages contain “bank”. Supply the relationships and conditions that make the intended meaning clear, rather than repeat the word more often.
Separate BERT pre-training from a downstream task.
The 2019 BERT paper describes bidirectional contextual representations learned through masked-language and next-sentence objectives, followed by adaptation to tasks such as question answering. Pre-training learns patterns from its training material; downstream training and evaluation determine how a system uses them for a particular task.
The paper’s task evaluation results concern its specified models and datasets. They do not define an article-writing score, a keyword quota or a recipe guaranteeing search performance. A token representation is not automatically a suitable whole-passage retrieval representation. The way a model is adapted and how its output is used still affect the result.
Keep research evidence separate from deployed search claims.
Google’s ranking-systems documentation describes BERT as a system used to understand how word combinations express meaning and intent. That supports a bounded statement about Google’s public description. It does not disclose every model version, vector function, feature weight or current retrieval process.
The original BERT research and the later Dense Passage Retrieval paper are separate research records. Do not infer that a search product deploys a paper’s exact architecture, training set or reported task evaluation simply because the concepts are related. A third-party tool labelling a result “BERT score” cannot by that label establish Google’s assessment of the page.
Understand sparse and dense retrieval without declaring a universal winner.
Sparse retrieval commonly represents terms and their weights in a large vocabulary space; BM25 is a familiar lexical ranking method. It can be useful when exact names, identifiers or version strings are needed. Dense retrieval uses learned lower-dimensional representations to rank candidate text and can help connect different wording. Neither guarantees a correct answer.
Karpukhin and colleagues’ 2020 DPR work studies dual-encoder dense passage retrieval for open-domain question answering, with question/passage training and evaluation on stated tasks. Its results belong to that setup. An application can compare lexical, dense or hybrid candidates under its own evidence and constraints; “dense always wins” does not follow.
The similarity function and training objective also affect the result. A system may use inner product, cosine or another configured comparison; those are not freely interchangeable scores. Retrieval normally selects candidate material. A later reader, ranker or answer process still has to inspect the relevant evidence and conditions.
Worked answer test: two similar passages, one decisive condition.
This is a fictional product-policy example, not real export instructions or a measured embedding run. The question is: “Under policy version 2, may a guest export a private workspace without owner approval?” Passage A says: “In version 2, guests may export a private workspace only after its owner approves the export.” Passage B says: “In version 2, owners can export private workspaces and invite guests to read shared pages.”
Both passages concern guests, owners, private workspaces and exports. Only A states the approval condition for the guest’s export. In the fictional policy, the answer is no: owner approval is required. B supplies nearby background but leaves that eligibility question unanswered. Its similar topic does not justify filling the gap with a guessed yes or no.
| Passage/test | What the passage establishes | Decision |
|---|---|---|
| A: guests need owner approval in version 2 | Explicit condition for the subject and action asked about | Answers the fictional question; still verify the policy source in a real system |
| B: owners export and guests read shared pages | Related roles/actions, not the guest-export approval rule | Useful background at most; does not answer this question |
| Similarity result | A model may rank either candidate close to the query | No scores computed here; closeness cannot supply an absent condition |
| Truth and currency | A statement could be unsupported or from an obsolete policy | Check source, version, effective period and contradictions before answering |
Distinguish similarity, relevance and truth.
Similarity concerns a chosen representation and comparison. Relevance concerns if the material helps with this question, audience and conditions. Truth concerns support for its assertions. These judgments can disagree: a closely worded false passage can rank highly, while a differently phrased authoritative answer can be missed.
Negation, dates, quantities, entities and exceptions can determine the answer even when much of the vocabulary matches. Check the exact subject and relationship, not only topic labels. If neither candidate supports a required claim, report the evidence gap or continue retrieval; do not manufacture the missing premise.
Design an evaluation around the real retrieval task.
Fix a corpus snapshot, representative questions, acceptable answer sources and relevance judgments. Keep training and test questions appropriately separated so memorized pairs do not inflate the apparent result. Include ambiguous terms, exact identifiers, negation, changed versions and questions for which the corpus has no adequate answer.
Compare methods on the same task and resources. A top-k retrieval measure can show if judged answer-bearing material is found within a candidate set; it does not by itself establish the truth of a generated answer. Separately review source accuracy, complete answer conditions, unsupported claims, latency and failure behavior. Record judgment uncertainty and missing labels.
A small test set or one domain cannot establish a universal result. Investigate cases where lexical matching finds a precise ID but the dense method misses it, or where different wording defeats the lexical method. Keep the useful failure cases and retest after corpus, model or query-distribution changes.
Write for the question and keep the evidence inspectable.
Make entities, relationships, conditions and source scope explicit where they change the answer. Use a complete example and a clear limitation instead of trying to increase an unexplained embedding score. Do not insert unrelated synonyms, repeat a term or rewrite a sound answer solely because a similarity tool prefers it.
The practical record should name model or retrieval method when known, corpus/version, question set, judgments, measure, observed limitations and owner. Unknown deployed details remain unknown. Contextual representation research can explain functions and research methods; it is not a promise that a page will rank, be retrieved, be cited or produce a correct generated answer.
Sources and related learning
Devlin and colleagues, BERT research (2019). Karpukhin and colleagues, dense passage retrieval research (2020). Google ranking systems documentation.
The worked scenarios are fictional teaching records. They do not report live tests, client work or measured search outcomes.
Continue with semantic SEO learning and passage retrieval and answer relevance. Resolve ambiguous subjects. Test exceptions and missing conditions.