Bastion outage

[kevinlul]

There is an outage in card search currently being investigated. More details as they come

[kevinlul]

Outage is estimated to have started around 2:27 PM ET. Recovered now at around 3:49 PM ET.

Tentative root cause is the backend (OpenSearch) falling over due to having more and more card data while having limited compute/memory in conjunction a recent change in ingestion processes.

[kevinlul]

Apologies for the trouble everyone. There was another outage today from around 2:26 AM ET to 9:23 AM ET, for the same reason. Will be keeping an eye on it throughout the day and conducting badly needed upgrades in the coming days so this stops happening.

[kevinlul]

This is fully resolved, but long-term upgrades still need to be scheduled.