Applying an updated ISM policy to existing OpenSearch indices
Why the team kept deleting old indices by hand, and the fix that made retention run on its own.
In plain terms
Nclose monitors its clients' security, and a big part of that is collecting their logs: the records of what happened on their computers and networks. Those logs live in OpenSearch, a database built for searching huge amounts of text quickly.
Some clients need their logs kept for a set time so that auditors can check them later. Keeping everything forever isn't an option, because the storage fills up. So OpenSearch has a feature called Index State Management (ISM): a set of rules along the lines of "keep this data for so long, then delete it". OpenSearch stores data in chunks called indices, and each index follows the version of the rules it was given.
The catch is that updating the rules doesn't change the indices that already exist. They carry on with the version they started with.
The problem
The compliance clusters hold client log data for third-party audits, and ISM policies decide when that data rolls over and gets deleted. After a policy update, the team kept having to step in and manually delete the oldest indices to keep the clusters healthy.
What I did
I found that the updated policy version was never applied to the indices that already existed, so they kept running on the old rules.
I used the change policy API to align the existing indices with the updated policy, on every compliance cluster.
The outcome
Retention runs on its own again on all 12 compliance clusters.