SofaScore’s CTO: From student to expert who scaled a solution for 20 million users

Although they did not initially plan it, after receiving an invitation, CTO Josip Stuhli and co-founder Ivan Bešlić of the world-wide successful Croatian company SofaScore focused on real-time sport scores and results monitoring, decided to go to Career Speed ​​Dating, student & employer meetup at Zagreb’s Faculty of Electrical Engineering and Computing (FER).

You have to keep in mind that most of these events are attended primarily by tech companies’ HR representatives, who deal with recruitment, but the duo from SofaScore wanted to know what the event looks like firsthand. They did not expect or prepare anything special, but simply wanted to talk to students, potential future members of their team.

FER students didn’t expect the co-owner and CTO of a company whose solution benefits 20 million users either.

Their first conversation was casual, but since SofaScore hadn’t been particularly promoted in the labor market until then, they decided that it might be better to introduce themselves after all – who they are and what they do. The most common question was: “Are you from Croatia too?”

Josip states that for many years many people thought they were from the UK because the company really looks like a foreign technology platform with excellent English. Even when people realized “there was something in Croatia”, they would think it’s a branch, not the headquarters. When many FER students who love sports realized that they were talking to the owner and technical director of the company whose app they’ve been using for the last five years, they were met with – enthusiasm and curiosity!

“How do you even get a score in the app? How do you handle 20 million users after that point?” were just some of the questions they were given, and the answer is as obvious as it is demanding:

Practice.

Just as Josip and Ivan had to learn hands on how to present themselves at Career Speed Dating, they had to learn how to develop and scale a solution for millions of users – step by step, result by result, peak by peak!

After all, Josip came to SofaScore right through an ad on fer2net (Faculty’s forum), because he was interested in scaling…

“Half a terabyte of data to be saved – and later analyzed – every day”

Josip points out that, for maintaining a system for 20 million users, cooperation between teams is vital – both those working on solutions, for iOS and Android, for example, and the backend team. When quoting figures, it’s obvious that he is proud of what they’ve managed to achieve.

About half a terabyte of raw anonymized data on how the application is used is imported from Firebase Analytics daily – which is actually about 200 terabytes of anonymized data per year that we store for at least three years so we can learn and develop new machine learning models.

Every time we get a new user, during the first two weeks we try to understand whether they are a quality user for us and then optimize the acquisition of new users. By learning user quality, we have managed to double user retention, but in practice this means half a terabyte of data per day to be downloaded, processed, saved – and analyzed later!

Just as the founders of SofaScore, Zlatko Hrkać and Ivan Bešlić, developed blogs in high school, Josip and a high school friend of his started making software and various “projects”, as he describes it. He admits that, like all kids, they were young, ambitious and a little arrogant, so when they heard how much companies charge for software solutions for the state, they thought – we could do it for 100 times less! They thought they were the best, and later… as he honestly admits: 

In practice, you realize that you haven’t got a clue about anything. I ended up in SofaScore because I responded to an ad on fer2net. From this perspective, both I and the founders were lucky to have found each other. They were very flexible, which suited me, and in addition to programming, I was interested in working with servers.

It was 2012 and as soon as the opportunity arose for Josip to take over the servers, which would later be key to scaling the solution, he did so. They initially switched to Amazon Web Services which was in its infancy because Josip – just like today – really believed in cloud solutions!

As soon as a match would start, the traffic on SofaScore would increase not by 20%, but… by 20x as much. Josip had to make do, because in Croatia he could not learn the necessary knowledge either in college or from other companies, which mostly developed solutions for others and did not have as many users:

It was an ideal case for using cloud solutions and that’s exactly how I learned. I knew the concept, but I didn’t have practice – so I had to try it out, read the forums, and go to conferences to figure out what other people were using. For example, a friend of mine worked at a startup in the US, so I asked him how they do it. Through sharing knowledge and experiences that we came up with a version that worked.

Although cloud hosting, as Josip explains, is relatively expensive, it gave them the flexibility they needed – if something went wrong they would have backup, a snapshot, and they would be able to try new things. If something didn’t work, they would be able to shut down the server and turn on a new one:

The founders and I understood each other and investing a little more money to see what we could get was no problem. Once we set everything up, it was a “no brainer”. As more users came, we would pay more for servers and earn more than we would spend on them.

Will the app still work if the whole of Zagreb opens it?

During one of the first big surges, Josip found himself – like many Croats these days – on the coast:

The first time we had a surge and a bunch of users came, we first picked up two, then four and then additional “machines” – and everything worked. I remember because it was El Clásico and I was at the coast. I waited for the biggest game so far in my apartment with a hotspot connection and wondered – is something about to die on me?!

It was stressful for Josip, of course, but he was also happy when they found a solution that works. He sees this as the reason why, despite the stress, he did not burn out along the way:

It’s definitely not for everyone. For me, a challenge was something that made me happy! We still have large influxes, but since the number of users has increased and become more consistent, we no longer get 50-fold increases. We know how to deal with such situations and we have experimented enough to have the necessary experience.

Step by step, Josip learned about small, bigger, and sometimes banal solutions with the rest of SofaScore. For example, at one point while scaling, they learned the hard way that the cloud is not unlimited – they reached Amazon’s security limit in the number of servers that can be launched at a time. In another situation, they wanted a specific server model with new processors, at the time, it was not possible to activate 20 of them because it was an experimental program.

You cannot simulate such situations, because there is no ordinary way to simulate what will happen to you in production with such a large number of users, nor what the correlations are between the different services you use. Seemingly banal situations happen, for example, you have a service that works to connect to an external server, but if you do not set it up well, the whole internal system can lock up.

Some unimportant part that sends you emails which you didn’t set up well can ruin everything, which is not something that a college professor would be able to explain to you.

This is knowledge that you can learn exclusively through practical work. Such a success was celebrated in SofaScore, for example, when according to Google Analytics they had 800 thousand concurrent users. As they later explained at Career Speed Dating, it was a situation in which “the whole Zagreb” opened the app!

If you want to learn from practice, you have to love what you do

The situation in SofaScore is a bit different now, however, and Josip understands very well the importance of a mentor that he did not have at the time, because no one was dealing with such scaling:

If you have a mentor who will show you everything, then you will not experience on your skin when you screw something up. You will not need to try ten other things only to discover one that works – which is both good and bad. I had to give it a try and simply learn how to debug some things because I had no other choice! If something goes wrong, either I will fix it or no one will. When will it be fixed? I don’t know, because you’re working until you fix it. This is why I’d like to stress to anyone who wants to engage in such learning that it has to be something they really love.

Josip emphasizes that such situations would have been much more stressful for him if he did not like what he was doing, because as the first DevOps in SofaScore, everything depended on him:

In such a scenario, money is absolutely irrelevant.

Amazon is brutal, but your own solution, when optimized, is even more brutal and cost-effective

Amazon Web Services proved to be a great solution for Josip, but over time due to its advertising-based business model, SofaScore needed a better value for money:

To me, Amazon Web Services is brutal and I would play with it like a small child. They build the base they take care of and, if one data center doesn’t work, another one does. It’s great, and it might work for Amazon or Netflix, but for us specifically, given that we were able to optimize the system, our own solution is much more cost-effective.

The current SofaScore server infrastructure, which they set up and manage themselves, costs 10 times less than the solution on Amazon Web Services and similar services:

It’s a bigger expense and, while it would be justified for someone, we wanted to optimize those costs so we could invest more in marketing and people. We’ve always been on the ‘cloud’, but in 2014 we realized we were spending the same amount on 20 times fewer users. We optimized everything over time, so we didn’t need Amazon’s automatic scaling, but it paid off more for us to simply buy more servers. When our system reaches 60 percent, we simply invest in even more servers, and today we use a hybrid cloud where we have a company from which we can buy servers, but also virtual servers if we need them.

Josip discovers that thanks to their previous practice they can now do more things on their own. Something they might have activated on existing cloud solutions with one click, now they had to do it themselves by writing a few plugins and internal documentation:

This solution is certainly more cost-effective, and requires a little more effort around internal documentation, unlike some solutions which you activate and that’s it. It definitely pays off, because it’s not a difference of a hundred or a thousand dollars, but multiple times bigger on a monthly basis!

Since Josip loves cloud solutions, it would be too easy to opt for them, he admits. But system optimization for their business model is the constraint that helped him learn more and develop his own practice with the team that made SofaScore a successful company. 

If you are dealing with technology, it’s too easy to say that you are not interested in the financial part of the story. But if you are not interested in such things, you will not do your part of the job well. Too many startups or agencies, when doing a project for a client, activate Amazon Web Services because it’s cheap and will never have a lot of users. The client will pay more, the revenue will be higher, but what will you learn?

The biggest problem Josip has these days is with whom to grab a beer to talk about the challenges facing SofaScore and some future user peaks. These are mostly freelancers who work for foreign companies with millions of users, and which, with the exception of SofaScore and a few other companies, are usually not available in Croatia:

There are people who know how to write brutal code, quick code, but do not have the opportunity to apply that knowledge to a project that really has such requirements and millions of users. What they know is great in theory, but the question is whether they really had a chance to test their knowledge.

Josip’s conclusion illustrates the next challenge the Croatian tech scene is facing. There are few companies where you can work with solutions for millions of users in Croatia, and such practices and knowledge are scattered among hundreds of individuals who may work for foreign companies. The exchange of such knowledge and practice would significantly help create a foundation not only for the further development of companies like SofaScore, but also for the development of the whole scene beyond “only” quality code – and quality scaling!