# Maintenance Notifications

# 06 March, 2024

## Issue Summary

  
A network outage occurring on 6MAR2024 impacted front-end access to the Cyber Crucible  
dashboard and telemetry access via API. This issue has since been resolved as of 7MAR2024.

##   
Root Cause

  
At 0400 UTC on 6MAR2024, telemetry traffic began escalating to an eventual 1600% of normal  
agent traffic. Additionally, the network bandwidth between AWS instances began to throttle to  
limited speeds between the AWS-hosted load balancers, servers, and databases. Across  
multiple zones, bandwidth constraints were variable and ranged from complete transmission  
loss (0% of capacity) to 50% of capacity. AWS DNS availability was also compromised.  
Network traffic inbound to Cyber Crucible was confirmed to be valid agent traffic from existing  
customers. Bot traffic to load balancers did not increase in volume.  
The additional bandwidth was not the primary driver for server unavailability, but exacerbated  
the middleware and database bandwidth issues.  
The spike in agent traffic during that time has partially subsided.  
At this time, Cyber Crucible communicates no correlation between published cloud outages on  
6MAR2024 to the AWS issues or the spike in traffic.

##   
Impact

  
Access to the Cyber Crucible dashboard and APIs would have been delayed or unavailable.  
Telemetry from installed agents was fully preserved, and may have been delayed in submission  
to the database, but no telemetry was lost. Agent endpoint protections were completely  
unaffected, functioning as normal, as no analysis or protections rely on network connection to  
any upstream APIs. At this time all telemetry has “caught up” in the dashboard.

##   
Resolution

  
Cyber Crucible created multiple servers inside and outside of AWS. Network stability inside of  
AWS seems to have improved overnight (7MAR2024), with improvements observed as early as  
0200 UTC. Cyber Crucible is currently operating at 500% server capacity, calculated based on  
the peak traffic bandwidth which has subsided. The team has been replacing AWS hosted  
servers as they lose network connectivity.

  
To be clear, additional server scaling was not an effective resolution in this matter.  
Cyber Crucible has begun to transition from using AWS as a sole cloud hosting provider. Work  
to support being cloud-platform agnostic has already been in progress for months, with initial  
plans to move from AWS as a sole provider in Q2 of 2024. End to end testing, including all

necessary double-encryption and x509 authentication schemes for our various servers and  
services, was completed at the end of February 2024. The network issues that we experienced  
with AWS have accelerated the move, originally planned for Q2, to begin a month ahead of  
schedule.

<div id="bkmrk-created-by-kyle-nehm"><div class="aui-page-panel" id="bkmrk-created-by-kyle-nehm-1"><div class="view" id="bkmrk-created-by-kyle-nehm-2"><div class="wiki-content group" id="bkmrk-issue-summarya-netwo"></div></div></div><div id="bkmrk--4"></div></div>