Sometimes success can overwhelm a system.
Viridity Energy is a technology company serving wholesale electric power markets. As it scaled up to support additional clients, its Core Module faltered. As loads increased external services that the application required was unable to scale leading to failed requests for meter data. In addition to this, network and database outages resulted in data loss that showed up on energy and forecasting graphs displayed by the system. In each instance, manual intervention was typically required in order to correct the data.
Software Fix Not an Option
Chariot consultants teamed up with Viridity Energy to tackle a complete rewrite of the business critical software component. The Core Module gathers meter data from an outside service, provides real-time updates to other modules, and archives the data for later recall.
The Chariot team began by analyzing the features provided by the outside service that fed meter data to Viridity. The contractor was responsible for aggregating meter data, and averaging functions for various time ranges. When Viridity required a new grouping of meters, it relied on the outside service to do the work, and that cost a lot of money.
Data on Demand
The solution was to task the outside service with providing raw last minute data using a Redis store. Redis eliminated reliance on a REST web service that could not effectively scale while meeting the new demands of the system. We decided to store the data in its raw form only, and perform the aggregations on the fly.
To make operations more efficient, we moved some of the averaging functionality from the outside service into the application. Running averages for each meter were cached in Redis for quick access and updating. We tackled the issue of outages and lost data by keeping a known state for the most recent updates. By referencing that state, we could determine if we had missed data for any given minute.
Data Gap Closed
Upon noticing a gap, the application would immediately reach out to another service to retrieve historical data for the missing time and populate the raw datastore. We also added the ability for the application to detect any new meters being published to Redis, then immediately reach out for historical data to backfill entries for that meter. The state of each meter was monitored in its own actor, allowing an individual meter to update and correct itself without affecting the others.
Built for Speed and Accuracy
The result of the Core Module rewrite is a more resilient and scalable system that delivers data to users rapidly, and cuts down on manual intervention.
With aggregations of data on the fly, groupings of meters could be changed at will without performing a rewrite of historical data. The implementation of this highly requested feature in time for the summer market greatly improved the value of the product for Viridity’s customers because critical grid events are most likely to occur in the summer. A more robust module removed the need for manual intervention in failure cases, and significantly reduced the chance that customers would see missing data. Chariot was instrumental in implementing Core Module software improvements that led to fewer issue reports and increased customer satisfaction.