As stewards of TMC’s infrastructure, our Data Engineering team maintains a set of guiding principles. One of them is “Keep the Lights On”:
We strive to maintain focus on the day-to-day value we bring to the movement. While every team member will have the opportunity to work on new and exciting projects, fundamental responsibilities such as security and sync reliability are our top priority.
This principle sounds simple, but it reflects complex work happening behind the scenes. At TMC, our technology has to remain flexible and reliable during election season, when demand is at its highest. In the current political climate, we also want to prepare for unexpected moments requiring rapid response, should they arise. Our systems cannot be overwhelmed at a critical moment for our country.
Quick: Can you find Election Day 2024 on this graph? We process much more data through TMC Haven, our data warehouse, now than we did then, so we expect even higher peaks this election season.
Consider the analogy of the power grid, which supplies energy to light bulbs around the world. The electrical grid is immensely complicated; in 2022, the U.S. Department of Energy reported, “The U.S. electric grid consists of more than 22,000 generators; 55,000 substations; 642,000 miles of high-voltage lines; and 6.3 million miles of distribution lines that serve 153 million customers.” Those statistics represent decades of science, engineering, supply chain logistics, and maintenance.
Despite the complexity of the system, using electricity at home is remarkably simple. I plug in my lamp, flip the light switch, and it just works.
This metaphor raises an important question: what are the elements of infrastructural preparedness?
It is there when you want it, in the amount you need. Just enough power for a desk lamp.
It makes what is complicated to build simple to use. Turning on the light takes just one flip of a switch.
In an emergency, you know who to call and that a recovery plan is in place. My energy company tells me when the power line will be fixed.
There When You Want It, in the Amount That You Need It
TMC has put in a lot of work over the years to ensure the overall reliability of TMC Haven, our data warehouse, built on Google BigQuery. Pipelines should be updated regularly and on time so members preparing GOTV lists receive the data they need without interruption.
This wasn’t the case in 2022, when our Redshift warehousing setup led to capacity issues. Old documents from fall 2022 reference “heavy load on the Redshift cluster,” “delayed member queries,” and a complete 47-minute outage during the final GOTV weekend before the November election.
I pulled some alarming graphs out of the depths of our Google Drive. The graph spikes at the end because things got slow.
The outage! The graph drops to zero because the system stopped working completely.
After that experience, we realized it was time to switch architectures. Thankfully, we didn’t have to start from scratch. Google BigQuery is a serverless data warehousing platform that provides tremendous durability during usage spikes. When an election is nigh or a national rapid response event is coming together in a hurry, BigQuery scales up to meet that need. When everyone is on vacation over the winter holidays, BigQuery scales down.
The upshot is that, since we migrated to BigQuery in spring 2024, we have had much more robust protections against TMC- or member-induced outages. In October 2024, we saw 25 million queries, but the average response time remained low, at around 1.5 seconds. Our engineers can now focus on increasing the uptime and reliability of our data pipelines rather than managing computer clusters.
Simple to Use, Even if Complicated to Build
We don’t work solely with members who understand SQL or Python; some of our organizations don't have the budget for analysts or engineers. When an organizing director or communications director asks for automations, they’re thinking less about data structures than about the repetitive tasks they’d like to simplify:
When I receive a donation, I want the donor’s information to be added automatically to our email CRM. Then everyone will get my newsletter and hear about our latest trip to the state capital, or
I need to quickly see, on a map of polling locations, where the highest concentration of Hispanic voters is, so I can send election protection volunteers there on Election Day.
When TMC’s technology teams work, they seek to enable the simplest possible user experience.
However, behind these no-fuss time savers lies a mass of pipelines and considerable engineering. From 2023 to 2025, we refactored our pipelines for both inbound and outbound flows to our warehouse, using dbt to shape the data and standardized frameworks to extract and load it. We built a unified foundation of shared code and data processing, from which we can customize to meet members’ unique needs.
In this way, we can sustainably wrangle our 175+ data ingestion and reverse ETL (moving transformed data from our warehouse back into our members’ tools) pipelines and 6,600+ dbt data models, ensuring our members receive consistent, predictable data that doesn’t require a second thought. Bonus: our Identity Resolution (IDR) system runs on the same unified system, ensuring that organizations can seamlessly link new data sources to human identities, whether they’re gathering detailed qualitative data for large language model (LLM) processing or piloting innovative ways to reach community members.
A Recovery Plan for Emergencies
As a former engineer, I never promise 100% uptime. Sometimes things go wrong, especially because TMC systems intertwine with so many other systems, from Google to movement tech vendors. When BigQuery and Cloudflare had a major outage late last year, our members noticed and asked us for updates before those services updated their status pages.
If we don’t have robust alerting, communications protocols, and a history of trust with our members, we’ll lose precious time during emergencies. That’s why we’ve put systems and processes in place to prepare for those moments:
Our infrastructure teams maintain a regular rotation throughout the year (one team just calls it maintenance, but another calls it “The Triager” for extra verve) for frontline handling of bugs and unexpected tickets. They monitor our alerting, which can cover everything from broken pipelines to member dashboard requests and security flags. We try to fix most problems well before members even notice them.
In the two months leading up to a general election, we add an on-call rotation for nights and weekends. Often, urgent requests peak a few weeks before Election Day, as our members assemble the final pieces of their technical plan before organizers head into the field for the biggest push of the year.
We maintain channels for regular communication with our members, both for community alerts and single-member contact. If we have members facing greater impacts, our Member Experience team proactively reaches out to them to identify the need for mitigation strategies.
These systems aren’t necessarily novel, but they build on tried-and-true best practices from the rest of the engineering world, and represent a key value of TMC membership that is out of reach for most nonprofit organizations. When a fire drill is underway, our top priority is to build trust through tight, personalized communication loops with our members and users.
Infrastructure Is a Movement Responsibility
When TMC engineers talk about “keeping the lights on,” they don’t mean scrambling to keep things working. As with public utilities and hurricane season, when an election storm comes, no amount of last-minute reaction can substitute for resilient systems built months or years in advance.
Movement organizations join TMC because they want to prepare themselves before the pressure is on, not after demands for data and technology have already spiked. They want to know that the systems are stable, simple—or even invisible—and responsive to their needs. When that stable infrastructure exists, organizers can instead focus their time and energy on what matters: getting out the vote and winning.
About the author
Bella Wang
Chief Technology Officer, The Movement Cooperative
Bella Wang (she/her) has served as the Chief Technology Officer at TMC since 2024. Outside of work, she has two kids and organizes for voting rights. In a past life, she was a game theorist; she still enjoys puzzles and overusing the phrase “grand strategy.”
