第 24 章 - Elasticsearch 索引生命周期管理

0. 前言

本章讲解 Elasticsearch 的索引生命周期管理。
通过 ES 的索引生命周期管理，我们可以非常方便的实现，索引自动删除、索引备份功能。
下面我将介绍生命周期的相关概念，以及基本使用。

1. 生命周期基本概念

1.1 生命周期的5个阶段

elasticsearch 将索引的生命周期分为以下5个阶段

• Hot: 索引频繁被修改、查询
• Warm: 索引已经不更新，但仍在查询
• Cold: 索引已不更新，也很少被查询。仍然可以被搜索，查询速度较慢，也没关系
• Frozen: 与 Cold 阶段类似，但查询的速度会更慢
• Delete: 数据将被永远删除

1.2 不同阶段所能处理的行为

在生命周期的不同阶段，会按照以下所列顺序执行操作。

• Hot: Set Priority,Rollover,Read-Only,Shrink,Force Merge,Searchable Snapshot
• Warm: Set Priority,Read-Only,Allocate,Migrate,Shrink,Force Merge
• Cold: Set Priority,Read-Only,Searchable Snapshot,Allocate,Migrate
• Frozen: Searchable Snapshot
• Delete: Delete

1.3 行为描述

• Set Priority
  当节点恢复时，优先级高的索引会被优先恢复

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"warm": {"actions": {"set_priority" : {"priority": 50}}}}}
}

• Rollover
  在满足条件时（如索引大小或年龄），创建一个新的索引并将写入操作指向新索引。

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"hot": {"actions": {"rollover" : {"max_primary_shard_size": "50GB"}}}}}
}

• Read-Only
  将索引设置为只读状态

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"warm": {"actions": {"readonly" : { }}}}}
}

• Allocate
  将索引移动到指定的节点上, 或设置索引的副本分片数

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"warm": {"actions": {"allocate" : {"include" : {"box_type": "warm"},"number_of_replicas": 1}}}}}
}

• Migrate
  将索引移动至当前阶段所对应的节点。该阶段仅在 Warm, Cold 生效

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"warm": {"actions": {"migrate" : {}}}}}
}

• Shrink
  减少索引的主分片数

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"warm": {"actions": {"shrink" : {"number_of_shards": 1}}}}}
}

• Force Merge
  强制将索引合并到指定的段(segment)数，此行为索引会变为只读

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"warm": {"actions": {"forcemerge" : {"max_num_segments": 1}}}}}
}

• Searchable Snapshot
  在当前阶段创建一个可搜索的快照。
  首先你需要创建一个快照仓库，以阿里云为例

PUT /_snapshot/my_backup
{"type": "oss","settings": {"oss.client.endpoint": "oss-cn-shanghai.aliyuncs.com","oss.client.access_key_id": "xxx","oss.client.secret_access_key": "xxx","oss.client.bucket": "xxxxxx","oss.client.base_path":"snapshot/","oss.client.compress": true}
}

接着，指定快照仓库

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"cold": {"actions": {"searchable_snapshot" : {"snapshot_repository" : "my_backup"}}}}}
}

• Delete
  删除数据

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"delete": {"actions": {"delete" : { }}}}}
}

基本概念介绍完毕，接下来带大家实操。

2. 生命周期管理实操

2.1 创建生命周期

步骤1 创建生命周期策略

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"hot": {                                "actions": {"rollover": {"max_primary_shard_size": "1GB", "max_size" : "2GB","max_age": "1d","max_docs": 1000}}},"warm": {"min_age": "30m","actions": {"allocate": {"number_of_replicas": 1},"shrink": {"number_of_shards": 1},"forcemerge": {"max_num_segments": 1}}}, "cold": {"min_age": "1h","actions": {"readonly": {},"migrate" : {}}}, "delete": {"min_age": "2h",                     "actions": {"delete": {}                        }}}}
}

参数	说明
hot	该策略设置索引只要满足其中任一条件：最大的主分片写入达到50GB、所有主分片写入达到150GB、索引自创建至今超过1天、文档数超过1000，就会触发索引滚动更新。此时系统会新建一个索引，该索引将重新启动策略，新数据会写入新索引，旧数据会移动到旧索引。旧索引将滚动更新后等待30分钟进入warn阶段
warm	索引进入warn阶段后，将副本分片设置1，主分片设置为1，强制合并为1个段。完成该操作后，索引将在1个小时后，进入cold阶段
cold	索引进入cold阶段后，ILM将索引设置为只读，并将索引从hot节点移动到cold节点。完成操作后，索引将在2小时后进入delete阶段
delete	索引进入delete阶段后被删除

步骤2 创建索引模板

PUT _template/test_24_template
{"index_patterns" : ["ilmtest-*"],"settings": {"index.number_of_shards": 1,"index.number_of_replicas": 1,"index.routing.allocation.require.box_type":"hot","index.lifecycle.name": "test_24_policy", "index.lifecycle.rollover_alias": "ilmtest"},"mappings": {"properties": {"name": {"type": "keyword"},"age": {"type": "keyword"}}}
}

参数	说明
index.routing.allocation.require.box_type	指定索引新建时所分配的节点
index.lifecycle.name	指定生命周期策略名称
index.lifecycle.rollover_alias	当索引触发滚动操作时，新数据往哪个索引别名写入

步骤3 基于时间+序号初始化索引

# PUT <ilmtest-{now/d}-000001>
PUT %3Cilmtest-%7Bnow%2Fd%7D-000001%3E
{"aliases": {"ilmtest": {"is_write_index": true}}
}

注意：<>{}/ 都是特殊字符，所以需要对 URL 进行编码

步骤4 通过别名写入数据

POST ilmtest/_doc
{"name": "elasticsearch","age": 18
}

查看 ILM 进度

GET ilmtest-*/_ilm/explain

2.2 更新 ILM 策略

PUT _ilm/policy/test_24_policy
{"policy": {"phases": {"hot": {                                "actions": {"rollover": {"max_primary_shard_size": "3GB", "max_size" : "5GB","max_age": "1d","max_docs": 1000}}},"warm": {"min_age": "30m","actions": {"allocate": {"number_of_replicas": 1},"shrink": {"number_of_shards": 1},"forcemerge": {"max_num_segments": 1}}}, "cold": {"min_age": "1h","actions": {"readonly": {},"allocate": {"require": {"box_type": "warm"}}}}, "delete": {"min_age": "2h",                     "actions": {"delete": {}                        }}}}
}

查看 ILM 进度

GET ilmtest-*/_ilm/explain

如果在滚动期间，我们更新了策略，那么滚动期间使用的还是旧的策略。新策略会在下一次滚动生效。

2.3 切换 ILM 策略

创建新策略

PUT _ilm/policy/test_24_policy_new
{"policy": {"phases": {"hot": {                                "actions": {"rollover": {"max_primary_shard_size": "3GB", "max_size" : "5GB","max_age": "1d","max_docs": 1000}}},"delete": {"min_age": "2h",                     "actions": {"delete": {}                        }}}}
}

应用新策略

PUT _template/test_24_template
{"index_patterns" : ["ilmtest-*"],"settings": {"index.number_of_shards": 1,"index.number_of_replicas": 1,"index.lifecycle.name": "test_24_policy_new", "index.lifecycle.rollover_alias": "ilmtest"},"mappings": {"properties": {"name": {"type": "keyword"},"age": {"type": "keyword"}}}
}

切换策略后，新策略不会立即生效。当前正在滚动写入的索引依旧使用旧策略，直到当前索引 rollover 生成新索引，新策略才会生效。
应用旧策略创建的索引，依旧绑定旧策略。如果需要为这些索引绑定新策略，可执行

PUT ilmtest-*/_settings
{"lifecycle.name": "test_24_policy_new"
}