ShapeShift has always prioritized customer support; we take it very seriously. Our industry is growing fast, and this means tens of thousands of new people start using this technology every week. ShapeShift, as a leading firm in this ecosystem, needs to help onboard these people into crypto, and help them feel comfortable. That is one of our jobs.
Unfortunately, over the past three weeks, we have been unable to keep pace with our support load, and we felt it warrants some explanation.
Traditionally the time it takes for us to respond to a ticket is an average of ~8 hours, and the time to get an issue resolved has been under ~20 hours, and this average includes nights and weekends. Zendesk tells us this is better than 90% of the tech industry, and we imagine it’s better than 98% of the crypto industry.
For the past three weeks, our resolution time skyrocketed to 5-10 days. This is not acceptable and is not the standard to which we hold ourselves.
There were two converging issues which resulted in this mess. One was solely our own problem, and the other was a network-wide, global problem. Both started happening concurrently.
- Our processing loop became overloaded. Without getting into technical minutia, the processing loop that makes transactions happen in ShapeShift was overwhelmed. This loop does many things: it is watching prices on multiple exchanges in real time, it is checking confirmations and our own enforcement rules, it is checking inventory balances, limits, and transaction details of the customer’s deposit. Running through all this takes 10-20 seconds per cycle. ShapeShift’s volumes have grown to the point where we are often receiving more than 1 customer order every 10 seconds. Thus, a backlog forms.
- Ethereum as a network has been under immense scaling pressure as the phenomenon of token sales continues to grow. Over the past three weeks, the Bancor and Status token sales caused spikes on the Ethereum network that destroyed the capability of Ethereum’s blockchain to process them. It took roughly a week to get back to normal after the worst spike. Tens of thousands of transactions across the network were stuck/delayed.
And then, because 2) happened, our processing loop 1) got even slower.
Orders that would normally process through ShapeShift in 2 minutes turned into days. We process about 10-15k customer orders per day now, and this means thousands of customer support tickets were opened up by customers who were understandably anxious about their order delay.
With six full-time customer support agents, we could get through about 800 tickets on our best days… but when 2000 tickets open each day, and this goes on for 1-2 weeks, you can do the math and discover the outcome.
Unfortunately, there was no easy solution of just “upping the server,” and new customer support specialists take 2-4 weeks to train up to skill, and that’s after the time spent to find quality, reliable individuals.
None of this is an excuse for the problem, nor for the anxiety our customers felt during these delays. We are truly sorry to all of you for the wait times you endured.
Volumes at ShapeShift have grown 30x since December. This kind of growth is just wild and difficult, no matter what, and is indicative of the exciting future that is fast arriving: one in which truly digital finance replaces the legacy analog financial systems.
While we’ve gotten ticket counts and wait times back to normal ranges, ShapeShift is retooling what needs to be retooled and is hiring more support agents, as quickly as can be done responsibly. Quick and helpful service to our beloved customers will always remain a top priority. It is a burden we welcome and a burden we will meet.
PS - I want to personally and publicly share my gratitude to our support team, who has put in ungodly hours over the past few weeks during this growth. You all have been rockstars. ShapeShift owes much of its growth and reputation to your steadfast work. Thank you.