Elasticsearch is a robust search engine widely used for its scalability and powerful search capabilities. Beyond simple indexing and querying, it offers sophisticated operations for handling document updates, deletes, and upserts. This article will explore these operations in detail, providing easy-to-understand examples to help you get started.

In Elasticsearch, data is stored in the form of documents. Each document is a JSON object and is stored in an index. Each document is associated with a unique identifier (ID). When working with documents, you may need to update, delete, or upsert (update or insert) them. Let’s explore how to perform these operations.

Updating a document in Elasticsearch can be done using the _update API. This allows you to modify the fields of an existing document without reindexing the entire document.

Example: Updating a Document

Assume we have an index called myindex with a document representing a user.

Step 1: Indexing a Document

First, let’s index a sample document:

curl -X POST "http://localhost:9200/myindex/_doc/1" -H 'Content-Type: application/json' -d'
{
  "name": "John Doe",
  "age": 30,
  "city": "New York"
}'

Step 2: Updating the Document

Now, let’s update the age of John Doe:

curl -X POST "http://localhost:9200/myindex/_update/1" -H 'Content-Type: application/json' -d'
{
  "doc": {
    "age": 31
  }
}'

Output:

The response will indicate the success of the update operation:

{
  "_index": "myindex",
  "_id": "1",
  "_version": 2,
  "result": "updated",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

The document’s version has increased, indicating it has been updated.

Deleting a document in Elasticsearch can be done using the _delete API. This operation removes the document from the index.

Example: Deleting a Document

Assume we want to delete the document we previously indexed.

curl -X DELETE "http://localhost:9200/myindex/_doc/1"

Output:

The response will indicate the success of the delete operation:

{
  "_index": "myindex",
  "_id": "1",
  "_version": 3,
  "result": "deleted",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

An upsert operation is a combination of an update and insert. If the document exists, it is updated; if it does not exist, a new document is created. This can be done using the _update API with an upsert clause.

Example: Upserting a Document

Assume we want to upsert a document in myindex.

Step 1: Preparing the Upsert Operation

We’ll attempt to update a document with ID 2. If it doesn’t exist, it will be created.

curl -X POST "http://localhost:9200/myindex/_update/2" -H 'Content-Type: application/json' -d'
{
  "doc": {
    "name": "Jane Doe",
    "age": 25,
    "city": "San Francisco"
  },
  "doc_as_upsert": true
}'

Output:

If the document doesn’t exist, it will be created:

{
  "_index": "myindex",
  "_id": "2",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

If the document exists, it will be updated:

{
  "_index": "myindex",
  "_id": "2",
  "_version": 2,
  "result": "updated",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

Scripted Updates

Elasticsearch supports scripted updates, which allow you to perform more complex updates using scripts.

Example: Incrementing a Field

Let’s increment the age field of a document by 1 using a script:

curl -X POST "http://localhost:9200/myindex/_update/2" -H 'Content-Type: application/json' -d'
{
  "script": {
    "source": "ctx._source.age += 1"
  }
}'

Sometimes, you only need to update a part of a document. This can be achieved using partial updates.

Example: Partial Update

Let’s update the city field of Jane Doe:

curl -X POST "http://localhost:9200/myindex/_update/2" -H 'Content-Type: application/json' -d'
{
  "doc": {
    "city": "Los Angeles"
  }
}'

For large-scale data modifications, bulk operations are more efficient. The _bulk API allows you to perform multiple update, delete, and upsert operations in a single request.

Example: Bulk Upserts

Prepare a bulk request to upsert multiple documents:

{ "update": { "_id": "3", "_index": "myindex" } }
{ "doc": { "name": "Alice", "age": 28, "city": "Seattle" }, "doc_as_upsert": true }
{ "update": { "_id": "4", "_index": "myindex" } }
{ "doc": { "name": "Bob", "age": 32, "city": "Denver" }, "doc_as_upsert": true }

Save this data to a file named bulk_data.json and execute the bulk request:

curl -X POST "http://localhost:9200/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary "@bulk_data.json"

Output:

The response will indicate the success or failure of each operation:

{
  "took": 30,
  "errors": false,
  "items": [
    {
      "update": {
        "_index": "myindex",
        "_id": "3",
        "result": "created",
        "status": 201
      }
    },
    {
      "update": {
        "_index": "myindex",
        "_id": "4",
        "result": "created",
        "status": 201
      }
    }
  ]
}

Handling errors during document updates, deletes, and upserts is crucial for maintaining data integrity.

Example: Handling Update Errors

from elasticsearch import Elasticsearch, helpers

es = Elasticsearch(["http://localhost:9200"])

actions = [
    { "_op_type": "update", "_id": "1", "_index": "myindex", "doc": { "age": 31 }, "doc_as_upsert": True },
    { "_op_type": "update", "_id": "2", "_index": "myindex", "doc": { "age": 26 }, "doc_as_upsert": True }
]

try:
    helpers.bulk(es, actions)
    print("Bulk update completed successfully.")
except Exception as e:
    print(f"Error during bulk update: {e}")

Elasticsearch provides several APIs to monitor the status of your operations:

• Cluster Health API: Check the health of your cluster.
• Index Stats API: Retrieve statistics for specific indices.
• Task Management API: Monitor long-running tasks.

Example: Using the Index Stats API

curl -X GET "http://localhost:9200/myindex/_stats?pretty"

This command returns detailed statistics for the myindex index, helping you monitor the impact of your update, delete, and upsert operations.

Handling Document Updates, Deletes, and Upserts in Elasticsearch: Best Practices

• Use Bulk Operations: Utilize the _bulk API for batch processing multiple document operations, reducing overhead and improving performance.
• Optimize Refresh Policies: Control when changes are visible to searches by setting appropriate refresh parameters, enhancing indexing performance.
• Minimize Script Usgae: If you want to avoid resource-intensive operations, use scripts carefully and choose bulk operations or partial updates.
• Monitor and Tune Performance: Regularly monitor cluster performance using APIs like Cluster Health and Index Stats, identifying and addressing bottlenecks for optimal performance.

Handling document updates, deletes, and upserts in Elasticsearch is essential for maintaining and modifying your data efficiently. This article provided a comprehensive guide to these operations, complete with examples and outputs to help you get started. By leveraging these capabilities, you can ensure that your Elasticsearch indices remain up-to-date and consistent with your application’s needs.

Experiment with different configurations and techniques to fully leverage the power of Elasticsearch in your data processing workflows. With a solid understanding of these operations, you’ll be well-equipped to manage your data effectively in Elasticsearch.