Advanced Scoring in Elasticsearch

Besides being a good matching mechanism, sorting is one of the key aspects of a good search engine. For ElasticSearch, sorting is based on generating a score for each document hit. Within this concept, the default ordering in ElasticSearch is relevance, which is indeed the best option for the majority of use cases. Before going into more depth on how to optimise a relevance score, I'll briefly explain how ElasticSearch defines a score.

Scoring by relevance

As previously noted, a search query clause will generate a score for each document, and how the score is calculated depends on the type of query clause. I won't elaborate on the differences between query types, but it's important to know these differences exist. I added all relevant links about this topic under references.

For ElasticSearch, relevance means the algorithm that is used to calculate how similar the contents of a full-text field are to a full-text query string.

The standard similarity algorithm used in Elasticsearch is known as term frequency/inverse document frequency, or TF/IDF, which takes the following factors into account:

  • Term frequency: How often does the term appear in the field? The more often, the more relevant.
  • Inverse document frequency: How often does each term appear in the index? The more often, the less relevant. Terms that appear in many documents have a lower weight than more-uncommon terms.
  • Field-length norm: How long is the field? The longer it is, the less likely it is that words in the field will be relevant. A term appearing in a short title field carries more weight than the same term appearing in a long content field.

When multiple query clauses are combined using a compound query like the bool query, the score from each of these query clauses is combined to calculate the overall score for the document.

Boosting

Af first step in score optimisation falls under the term boosting.

I'll explain boosting by a simple example. Lets say you want to find an item in a catalogue based on a search query against multiple fields. An easy way to do this is in ElasticSearch is by using a multi_match query. By default this will run a query against multiple fields. However it is possible to define a weight for the individual fields by adding a boost factor to them.

The following query will search for the text 'Lambda Expressions' in title, tags, speaker name and description. When it comes to scoring, a hit on tags will only count for 80% compared to a hit on title, a hit on speaker name will only count for 60% and a hit on description will only count for 50%.

Notice the boost parameter behind the caret symbol

{
    "multi_match": {
        "query":  "Lambda Expressions",
        "fields": [ "title", "tags^0.8", "speakers.name^0.6", "description^0.3" ] 
    }
}

Java API:

final MultiMatchQueryBuilder multiMatchQuery = QueryBuilders.multiMatchQuery(
      text, "title","tags^0.8", "speakers.name^0.6", "description^0.3")
          .type(MultiMatchQueryBuilder.Type.BEST_FIELDS);

There are different boosting capabilities available in ElasticSearch and when it comes to tweaking your relevance score, boosting is often the first step.

Boost by date: recency scoring using function score

Although sorting by other values besides score are available in ElasticSearch, these options are often not functionality you're looking for. When you would for example simply sort your results by date, you will probably completely loose you relevance score (you could easily imagine that your least relevant match was the most recent document by accident).

In many cases the desired outcome is to keep your relevance score and give the more recent matches an extra boost (a higher score) because the data is fresher. In order to achieve this, you can use ElasticSearch's function scoring.

Here's an example:

Say you want to boost all documents published the past half year. Documents older than six months will score gradually less until a threshold of two and a half years is reached. Documents older than two and a half years will not get any extra scoring based on recency.

In ElasticSearch you use function scoring to achieve this. Here's the actual code:

"query": {
    "function_score": {
          "query": {
            "multi_match": {
                 "query": "Lambda Expressions",
                  "fields": [ "title", "tags^0.8", "speakers.name^0.6", "description^0.3" ]
            }
          },
          "functions": [
              {
                "gauss": {
                    "publishedOn": {
                          "scale": "130w",
                          "offset": "26w",
                          "decay": 0.3
                    }
                }
            }
        ]
    }
  }

Java API:

final MultiMatchQueryBuilder multiMatchQuery = QueryBuilders.multiMatchQuery(
      text, "title","tags^0.8", "speakers.name^0.6", "description^0.3")
          .type(MultiMatchQueryBuilder.Type.BEST_FIELDS);

final FunctionScoreQueryBuilder functionScoreQuery = QueryBuilders.functionScoreQuery(multiMatchQuery);
functionScoreQuery.scoreMode("multiply");
functionScoreQuery.boostMode(CombineFunction.MULT);
   functionScoreQuery.add(
       ScoreFunctionBuilders.gaussDecayFunction("publishedOn", "130w").setOffset("26w").setDecay(0.3));

Note: it seems that not all time unit options are implemented in the Java API. The Y and M option will throw an error when used. That's why the values are translated into weeks in the example.

Explanation:

  • origin: The point of origin used for calculating distance. For date fields the default is now.
  • scale: Defines the distance from origin at which the computed score will equal decay parameter.
  • offset: If an offset is defined, the decay function will only compute the decay function for documents with a distance greater that the defined offset. The default is 0.
  • decay: The decay parameter defines how documents are scored at the distance given at scale. If no decay is defined, documents at the distance scale will be scored 0.5.

In the example the current date is used as origin because an actual value is omitted, The offset of 26 weeks gives all documents published in the past half year the highest boost. Both the decay parameter and the scale are responsible to calculate the decay for the documents older then six months and younger then two and a half years.

I used the gauss decay function but feel free to choose either 'exponential' or 'linear' instead. The difference between those function is nicely explained by Elasticsearch here

A graphical overview of the decay functions:

Decay graph

Boosting by popularity

Another way to enhance a document's score is boosting on popularity. This can be done if your data model contains data reflecting popularity in any kind. Common examples of popularity data are view counters, likes, dislikes, ratings, number of purchases, and conversion rates.

Let's imagine that we would like more-popular videos to appear higher in the result list, but still have the full-text score as the main relevance driver. We can do this easily by storing the number of view of a movie clip. In our query, we can use the function_score query with the field_value_factor function to combine the number of views with the full-text relevance score.

There's a catch though. If we would simply multiply our score with the plain popularity value (in our case the number of views) it would completely swamp the effect of the full-text score.
That's why we use a logarithm to temper the likes value. In other words, we want the first views to count a lot, but for each subsequent view to count less.

"query": {
    "function_score": {
          "query": {
            "multi_match": {
                 "query": "Lambda Expressions",
                  "fields": [ "title", "tags^0.8", "speakers.name^0.6", "description^0.3" ]
            }
          },
          "field_value_factor": {
            "field":    "counters.views",
            "modifier": "log1p",
            "factor":   2 
          }
    }
  }

Java Api:

  final MultiMatchQueryBuilder multiMatchQuery = QueryBuilders.multiMatchQuery(
      text, "title","tags^0.8", "speakers.name^0.6", "description^0.3")
          .type(MultiMatchQueryBuilder.Type.BEST_FIELDS);

  final FunctionScoreQueryBuilder functionScoreQuery = QueryBuilders.functionScoreQuery(multiMatchQuery);
  functionScoreQuery.scoreMode("multiply");
   functionScoreQuery.boostMode(CombineFunction.MULT);

  functionScoreQuery.add(
    ScoreFunctionBuilders.fieldValueFactorFunction("counters.views")
        .modifier(FieldValueFactorFunction.Modifier.LOG1P).factor(2.0F)
);

All the additional information on the above topics is available below.

References