City of Mont Belvieu, Texas

From a major outage to rapid response and full system restoration.

mont-belvieu

Overview

The City of Mont Belvieu, Texas, experienced a significant service disruption due to a resource quota limitation within their Zadara environment coinciding with the global CrowdStrike outage patch. This resulted in substantial downtime and recovery efforts.

Industry

Government

Solutions

On Premise aaS ITaaS with full stack, Storage, Compute

Location

Texas

Zadara's support team gives me complete peace of mind. As a service provider, it's critical to have reliable backup, and Zadara delivered when it mattered most. While I was traveling during the CrowdStrike event, I was able to confidently direct my customer to Zadara support, knowing they would be in excellent hands. Their responsiveness and expertise make them an invaluable partner."

The Challenge

  • Unforeseen resource quota limitations within Zadara.
  • Inability to create additional servers due to resource constraints during a critical outage.
  • Initial lack of awareness of instance and core resource limits.
  • Coincidence of resource issue with a global CrowdStrike outage patch.

The Solution

  • Temporary expansion of resource quota with Zadara.
  • Restoring servers via snapshots.
  • Identifying and implementing the CrowdStrike folder renaming fix.
  • Reducing CPU resources on servers to free up core resources.

Impact

  • Approximately 85 man-hours lost.
  • Estimated $10,000 cost to restore services.

Takeaways

  • Importance of understanding and monitoring resource quotas and limitations.
  • Necessity of having readily available contact information for MSP Admin.
  • Value of quick response and support from vendors like Zadara.
  • Importance of internal communication and collaboration during crisis management.

Acknowledgement

  • Special thanks to Zadara for their swift support in enabling system restoration.

Incident Timeline and Key Events

Date/Time

Event

Several days before CrowdStrike outage patch

Resource error encountered while attempting to build a new server through Zadara. Discovered quota limit set by MSP Admin.

Friday, July 19, 5:30 AM

City computers stuck in a reboot cycle. CrowdStrike identified as the issue via csagent.sys message on VM server console. Priority given to restoring domain controllers. Resource limitations with Zadara hindered creation of additional servers.

Friday, July 19, Morning

Contacted MSP Admin and Zadara to temporarily expand resource quota. Quota expanded from 50 Instances using 153 cores to 100 Instances using 224 cores. Servers restored via snapshots. Corrupted servers deleted. Team worked on restoring PCs to latest restore points. Fix involving renaming CrowdStrike folder discovered and shared.

Friday, July 19, 12:12 PM

Ticket opened for documentation.

Friday, July 19, 12:20 PM

CrowdStrike Support responded with a link to a fix.

Post-Resolution

CPU resources reduced on several servers to free up core resources. Adjusted to 43 Instances using 161 cores. Awareness of instance and core resource limits established. Plan to budget for more Instances and Cores next year.

Scalability & Flexibility Advantage

At the time of implementation, the City of Mont Belvieu had invested in the exact amount of compute and storage capacity needed for their operations. Blue Iron Technologies, in partnership with Zadara, delivered a consumption-based model that enabled the city to pay only for the resources they were actively using—eliminating the need for upfront overprovisioning.

During the CrowdStrike outage, this model proved invaluable. As the city required additional compute and storage capacity to recover and stabilize critical infrastructure, the existing solution allowed for rapid scaling without the delays or costs associated with procuring and deploying physical hardware.

Key Benefits Realized:

  • On-demand expansion: Resource limits were increased within hours, not days or weeks.
  • No shipping delays: Additional capacity was provisioned virtually—no new gear needed.
  • Cost efficiency: The city continued to pay based on actual utilization, avoiding overspend on idle infrastructure.
  • Modern architecture advantage: This kind of elastic scalability is not typically achievable with traditional hardware-centric solutions.

This event underscored the strategic value of leveraging a flexible, cloud-based infrastructure approach, ensuring municipalities like Mont Belvieu can adapt rapidly in the face of unforeseen challenges.

 

Cost Control & Support Coverage

One of the most valuable aspects of the City of Mont Belvieu’s experience was the ability to manage the unexpected crisis without incurring long-term financial strain.

Support during the incident—including vendor escalations, recovery assistance, and resource scaling—was fully covered under the city’s existing agreement with Blue Iron Technologies. There were no added support fees, and the collaboration between Blue Iron and Zadara ensured continuity without delays or excessive cost.

Additional resource usage was billed only by the hour and strictly based on actual utilization. Once the emergency subsided and the temporary file systems were removed, resource consumption quickly returned to baseline levels. As a result:

  • The total cost impact was minimal, with only a temporary uptick in that month’s usage.
  • There were no surprise charges or support overages.
  • The city maintained budget predictability, even during a high-pressure recovery operation.

This flexible, pay-as-you-use model proved to be not only scalable but also financially sustainable, offering municipalities like Mont Belvieu a practical way to handle spikes in demand without overcommitting on infrastructure or incurring hidden costs.

Looking For More Customer Stories?