Anatoly Yakovenko of Solana: How It Started

In FTX Podcast #64, Anatoly Yakovenko discusses building Solana from scratch, blockchain hardware, exciting projects and more!

*Compared to major competitors. Terms and conditions apply. This is not investment advice or an investment recommendation. NFTs are shown for illustrative purposes and the types of NFTs available may vary. Cryptocurrencies, including NFTs are highly volatile, subject to significant price risk, and may not be suitable for you.

Looking to learn more about blockchain technology? Join us as we talk with Anatoly Yakovenko, the mind behind the Solana blockchain. This episode is full of deep insight into the creation of Solana, what makes Solana’s technology unique, and how it’s changing the trajectory of the crypto world. 

Anatoly Yakovenko Interview 

Tristan Yver: 

Hello, everyone. Welcome to The FTX Podcast. Today, I have with me a guest that I've had before and I'm very happy to have here with me again, Anatoly Yakovenko, the cofounder of Solana Protocol. Welcome, man. 

Anatoly Yakovenko: 

Thank you, Tristan. Awesome to be here. 

Tristan Yver: 

It's good to have you back. I know it's been a crazy year or so since we last spoke. I do think a lot of listeners at this point definitely know who you are. I don't think everyone knows the backstory, how all of this started and I would love just to get filled in on that front and maybe talk about origination, where Solana began. 

Anatoly Yakovenko: 

For sure. Yeah, it's a good story, so I love to tell it. And I can start from the top. So my background, I spent most of my career at Qualcomm and I joined in 2003 straight out of college at a voice over IP startup in college. I was building these Linux systems with some friends, trying to build something that I don't know if you guys remember Google Grand Central Dispatch or Google Voice. We had the software that could manage phone lines, boring but cool because this was 2003. But we failed horribly because it was after the dot-com crash and it was Central Illinois. So no business plan, but I ended up learning a ton and Qualcomm hired me on the spot and I moved out to San Diego and started really working on performance optimizations almost from the start. 

Anatoly Yakovenko: 

So that was a lot of profiling, looking at code, trying to find places where things were slow for various reasons, whether it's because the math was bad or it was hitting memory too much or just poorly written code or poorly threaded code, things like that. And I got deeper and deeper into this part partly, because when there was a really gnarly problem, I could work on it and my managers would leave me alone. And I built up enough trust where they would give me these hardest problems and I could work on them without getting interrupted by small bullshit.

Anatoly Yakovenko: 

So that was the motivation and it was just fun to do that. And I ended up a lead on the Advanced Technologies Group and I built a stack that just moved memory and code between all these processors. A modern-day chip has 26 different, you can call them processors, we call them subsystems, but separate CPUs effectively with different architectures. There's over 30 of them on a modern phone. So it's a crazy complicated system and I wrote a bunch of code to make it really easy for devs to take advantage of all this hardware and to do cool stuff with it. One of those was Project Tango which was an augmented reality device in a phone. This was the first time anyone got that stuff to work from a computer lab, big system on a phone form factor.

Anatoly Yakovenko: 

2017 rolled around and a buddy of mine and I have this startup called, not a startup, side project called Tensor Cloud. We wanted to take deep learning hardware and go deploy it, but before we had customers, we would mine crypto with it to offset the cost of the hardware and buy more hardware. I'm sure there's still people trying to do this. We were more into deep learning. That was the cool thing. Crypto was like the hook and I started really thinking about proof of work, why it's slow, why it's necessary and that was the red pill for me or the start of it. 

Anatoly Yakovenko: 

We started talking about like, "Well, how would you design this, so it was more electricity efficient more dynamically efficient?" We're talking about single-threaded mining. And I had two coffees and a beer and I was up until four in the morning at this eureka moment that, instead of measuring electricity, you can actually measure another physical constant which is time. So at that time, there was a ton of proof of stake research and all of it seemed weak because of weak subjectivity, right? You have these coins that don't have any weight to them that are manufactured out of thin air and you're using that for security based on these assumptions about liquidity and all this other stuff, but none of it seemed as rock-solid as proof of work, right? Where real electricity was spent to secure the network. 

Anatoly Yakovenko: 

So to me, it seemed like some other physical constant would be critical to securing the system. And this was this eureka moment that you have a source of time and you can use that as the physical constant, but the cool thing about it was that I realized instantly that it wasn't about security, it was more about the actual physics of measuring time outside of consensus. So this is a problem in radio towers that was really observed from day one too when people put up radio towers for the first time, when two towers transmit at the same time or the same frequency gets noise because those waves interfere. So they gave each tower a clock and they will alternate by time. 

Anatoly Yakovenko: 

And this is just something that was in my head for working at Qualcomm for 12 years because wireless protocols fundamentally do this optimization and it's called time division multiple access, TDMA. And this was the foundation of these 2G cellular networks. It's the first time people actually had phones in their hand that they could carry around. They were called TDMA phones. And even now like LTE, 5G, CDMA, all that stuff is based on first dividing by time, then by frequency and they do this complicated frequency hopping thing and error correction, but there's always this time slice component. 

Anatoly Yakovenko: 

So once I had that eureka moment that there is another physical constant you can measure and that constant is time. I had this realization that the same thing that applied for optimizations at radio applies to blockchain because when you have proof of work, you have to block producers. If they produce a block at the same time, you get a fork, so the network is in a noisy state. Just like when radio towers transmit at the same time, you get noise, you don't know what information was passed. In that fork state, you don't know what the state of the network actually is, so you have to resolve it. And that same error correction kind of stuff, retransmit, retry kind of thing happens in radio, the same stuff happens in proof of work systems. 

Anatoly Yakovenko: 

And this is why you can't reduce it before block times to be really, really small like 100 milliseconds or 400 milliseconds because if it's a global network, you'll just constantly start producing forks and the network will never converge. So once I had this time component, I was like, "Oh, wow, we can actually build TDMA. We can start building a network that has the same properties as these radio networks but for blockchain." It was actually like a wild moment for me. I felt like I was manic for about a week. I couldn't speak because the idea was so big. I was so excited. I called everybody I knew. 

Anatoly Yakovenko: 

And some of the folks that I play the sport of underwater hockey simply because I'm in Silicon Valley, they're connected to VCs and the advice that I got was that, "Crypto is incredibly hot right now. You should go raise capital to start a company." So that was really what kicked me out of my, I don't know, lazy career path or fun career path to optimizations like avoiding bullshit work to actually go build a company. 

Tristan Yver: 

Well, now you have insurmountable amounts of bullshit work combined with other work. 

Anatoly Yakovenko: 

It's fun, man. I will crimp ethernet cables and sweep the floor if it's your baby. The thing is that, in my career, I've always had a side project that I was trying to, "Maybe this can go back, right? Maybe that deep learning thing could have gone big and we would be the Amazon for deep learning," but there's always this inception, I think, if you're a software engineer that at least in my time when I became a software engineer, I was there in college during the dot-com boom. And you saw the same craziness and hype that you see in crypto right now. I was an impressionable teenager and it was cool, right? Hackers was my favorite movie. I listened to industrial music. I built Linux from scratch. 

Anatoly Yakovenko: 

So this idea that you can write a bunch of code and change the world with something that I believed in, saw that happened with Google, with Facebook, with Amazon. So it's always there. I'm glad that I have the opportunity to take a stab at it. 

Tristan Yver: 

Well, now, it's actually really manifesting, but I think before we go to that, I'd like to hear a little bit more, so I don't think we ever talked about this, but those early days when you got this advice to go raise, how did you structure all of this? You're a full-time engineer at Qualcomm, how did you hash it all out? 

Anatoly Yakovenko: 

So advice to all the founders and this was immediately that my close friends told me it was that, "You have to quit your job because no one that's out there professionally funding companies will trust a founder that's not full time because they don't see a level of commitment, they can't tell if this is a full-time team or not, how serious they are." That's almost like the first filter. So it was a big change for me, but being in California as an engineer, you have this amazing safety net just simply because of the top tech companies in the world are here. Honestly, my manager at Dropbox told me this happens all the time, "If you come back and this doesn't work out in six months, we'll give your job back. No questions asked." So you have ... 

Tristan Yver: 

That's nice. 

Anatoly Yakovenko: 

... the best place in the world to do this. It's part of this huge advantage of being out in California, in Silicon Valley. It would have been harder to do it in San Diego simply because you have one or two companies there that give you that safety net. In Silicon Valley, you have hundreds from small to medium to very large big tech firms. So that was the kick in the pants that I needed, and once I started going, that was both fun because it was new and a grind because it was a wall of meetings. If you ever raise money, you quickly realize that it's hard and you're bad at it because you've never done it before. You're bad at sales, you're bad at talking to people, you're bad at getting intros, you're bad at approaching people and you fail a thousand times before you're going to figure it out. 

Anatoly Yakovenko: 

So that was a learning experience. It took about three months to get my first serious like ... I was able to get the ball rolling by going through all my immediate [inaudible 00:12:19] friends but immediate connections and getting commitments, soft commitments, that if we get a lead and raise a significant seed that they will also put in. So whatever that lead was at least knew that there's some other capital coming in and they didn't have to cover the whole round. That was the strategy that I took to get some form of like KPI that, "Okay, if I get enough soft commitments, then that number becomes more real and somebody will throw in that 250k check and set the terms." 

Anatoly Yakovenko: 

That was all that I was looking for and that works and it took about three months just talking to everybody before I got connected to actually like Edith Yeung and Chris McCann who are, I think, one of the early investors in FTX. They're really awesome seed investors that have seen everyone that's done this for the first time to have like pattern recognition for folks that, "This is the first time they're raising funding and they actually have something worth investing in." I think that's a unique skillset that a lot of VCs don't have. And they were really the first folks to recognize that there's something there, first money in, so that was it. We were able to get the company together and make payroll by April. 

Tristan Yver: 

So you had some money and now all of a sudden you had to build a blockchain from scratch basically, right? Because there's nothing you could fork here. Is that sort of what happened or- 

Anatoly Yakovenko: 

Yeah, at that time, Cosmos didn't work. This was before Cosmos, ETH1 was barely usable. 

Tristan Yver: 

You could make tokens on it. 

Anatoly Yakovenko: 

Yeah, a lot of stuff was really busted and a lot of people had crazy ideas about what tech to build but not a lot of experience building scalable technologies. Because myself and most of the cofounders are former Qualcomm engineers, we hired from that set. We got really lucky with the folks, so we were able to pull in from Qualcomm like senior director of technology that was still coding more than eight hours a day because that's what he loves to do. People that were promoted because they're awesome and just drive tech with just the sheer power, we were able to get those people in. 

Anatoly Yakovenko: 

Because we all worked together for over a decade, the building was easier in a lot of ways. We just naturally knew that there's a way to do this that's incremental. We didn't do a huge design upfront. We knew which pieces we needed to build that were obvious. We built those first and then started building on top of them and glued them together. If you think of it as the Three Little Pigs analogy, it wasn't like build a straw house then the stick house. You build all three houses at the same time on top of each other and you start swapping out ... 

Tristan Yver: 

Which one holds or? 

Anatoly Yakovenko: 

Yeah, exactly. You start swapping out the straw for stick when it was important or stick for brick and you just do this piecemeal. And we were able to bootstrap like ... We had smart contracts that you could run and see basically in about four months from starting point which was not something that we knew we were going to sit with because we had this vision for using Rust, a modern language that has an awesome packaging system, but you needed to get something, right? So that was part of our experience working together was that you can get that barebones MVP that just connects two dots and that was good enough and then you make that work and then you make it better. So that iteration was pretty critical to our success. 

Tristan Yver: 

That's what I tell people a lot when they ask about Solana. They go back to the fact that you worked in telecom, and honestly, in telecom, shit has to work. You don't get that leeway of other industries there. You have to make it work because so much is dependent on it. So that's part of my pitch for you, always like, "These guys all came from telecom. They know how to make stuff work." Something I have been curious about recently what's ... Obviously, this happens in most blockchains but in Solana especially, so the relationship between code and hardware and you were saying earlier this is something that you spent a lot of time optimizing on these cards. Is that exact same relationship that’s happening here with the hardware that supports the network or is it different? 

Anatoly Yakovenko: 

So there's a bunch of like Moore's law, as it's called, actually came from an early [inaudible 00:17:29]. I don't remember who the name of this person was, but they were talking about airplanes and industrialization of building airplanes. And that capacity and efficiency of airplanes was doubling at a rate. So they looked at the entire industry and they saw that there's a bunch of different parts that go into an airplane and everyone out there, some cog in a giant machine like me at Qualcomm was optimizing these little pieces, the aggregate of all this work is that there's an exponential curve of efficiency in terms of efficiency per dollar that's happening. 

Anatoly Yakovenko: 

And that was the observation that John Moore had on the semiconductor industry. If you look at it as a whole, that per dollar, the amount of computing that any system can do just keeps doubling every two years. And where that has impact is not just ... People first thought of it just frequency, right? And that became obviously not true. And then you can think of it as the number of cores in the system, right? We went from two-core [inaudible 00:18:38], AMD shipped 128-core system. 

Tristan Yver: 

That's all that. 

Anatoly Yakovenko: 

So those things are doubling, right? But also the memory bandwidth, the PCI buses, the network cards, all that stuff doubles. The amount of network cards, you can plug in that have high capacity and so same system doubles. The amount of SSDs you can plug into that and the SSD throughput doubled, right? So there's a bunch of little things that keep improving. So when you look at this overall, it's a singularity function. It's just this never-ending hockey stick that just keeps getting faster and faster. So same thing happened at Qualcomm. We started with these 200 megahertz, 16-bit ARM devices that ran flip phones, right? And you can do an address book and autocomplete on words and then I was building augmented reality 10 years later. It was- 

Tristan Yver: 

"Woo." 

Anatoly Yakovenko: 

Yeah, exactly, right? And so I could not have predicted that in 10 years nor can I predict what's going to happen now in 10 years like, "What applications? Are we going to run like 3D modeling block done on Solana, right?" In the smart contract, maybe, I don't know. If there's a demand for it, the hardware will make it possible, but that's what we need to realize. So the way we've thought of it is that the key part that you need to make sure when you're scaling is that if hardware doubles, that your software doesn't really need to change by a lot to take advantage of it and you just get this natural improvement in whatever the key part is like the fees to the user or like the number of systems that you're using for security, things like that. 

Anatoly Yakovenko: 

And in blockchain, I thought that the most important piece that's different about these systems than a database was the censorship-resistant component like, "How do we make sure that this thing is actually censorship resistant?" Well, you need to maximize the number of nodes that add up to that smallest set that can sensor or help the network. "What does that minimum set? If you look at the network as a whole and you drew a circle around that set of nodes, that if they all turned off the network would stop, how do we make that circle as big as possible?" 

Anatoly Yakovenko: 

The only way to do that is to get them to process more messages per second, sign more signatures, handle more bandwidth. It's got to come from hardware. You have to actually write the software to where it can take advantage of hardware. And as hardware doubles, censorship-resistance doubles, that means we're doing the right thing, right? We're optimizing for this key differential feature. And it was also kind of obvious to me that that piece is the same thing that is what is important to marketplaces. If you think about it, you look at NASDAQ, you look at NYSE, you jump through all these regulatory hoops, you get to put a box in a room, they will give you a cable that's exactly the same length as everyone else's cable and you get to plug into it. Now you can trade. 

Anatoly Yakovenko: 

And you have fair access to these markets amongst all the other people that jumped through these hoops which is effectively censorship resistance, right? They built this like the Fort Knox of regulation and oversight to make sure that everyone that gets inside is now at this censorship-resistant sandbox. So that's the purpose of all that stuff, right? To make sure that me as some market maker that's big, 10 million bucks for access, I guarantee that no one else will get ahead of me because we're all getting the same fair service, right? The same access to information, the same market, but if we're doing this with hardware and software in Moore's law and it's commodity hardware that just gets cheaper and cheaper, then we've built this for free for the world. And now you have free and fair access for marketplaces around the world globally. 

Anatoly Yakovenko: 

So I thought that that piece was really cool and price discovery seems like a natural fit for that like, "Let's build markets. Let's build whatever crazy things that people want to do." If you look at my early interviews, the only things that were there was like EtherDelta and 0x. Those were like the two things that existed at that time. They were both pretty horrible and what was really motivating to me too was that centralized exchanges were getting hacked all the time and run by some pretty shady people like Mt. Gox, right? They were either incompetent at security or shady where they would run away with the funds. 

Anatoly Yakovenko: 

That was the story of crypto and it seemed obvious that if you build an awesome decentralized exchange that it would just be better in every way. So that was part of the motivation there. So my early seed like level slide sets Solana's blockchain at NASDAQ speed. So I was scanning for NASDAQ, which I thought was the best and the coolest exchange, and what was really surprising in the two years that it took us to build Solana, you saw Binance and FTX really grow with crypto and become these massive amazing decentralized exchange platforms that I would say are already a decade ahead of NASDAQ in terms of innovation, products, fast to ship stuff. 

Tristan Yver: 

For sure. 

Anatoly Yakovenko: 

It's just been wild to see that. That to me is also a proof point that crypto, the coolest thing about it I think is marketplaces right now. Maybe we're in the phase of its adoption where marketplaces are equivalent to search engines in the '90s like indexing and storing and processing all the world's information. It's a big dream, right? 

Tristan Yver: 

Mm-hmm (affirmative). 

Anatoly Yakovenko: 

In the '90s, you saw the early hints of that. I think this idea that you have these open platforms that are accessible to everybody and anybody in the world can synthesize the world's information and then put a price on anything is pretty crazy, right? Making that open is awesome. Uniswap is 32,000 pairs, right? NASDAQ has, I think, a few hundred. 

Tristan Yver: 

And they can only support numbers up to a certain size as you see what happened with Berkshire Hathaway recently. 

Anatoly Yakovenko: 

Exactly. 

Tristan Yver: 

Well, I'm really happy Serum came into the mix then because it seems like in a lot of sense, it was that proof of concept of this marketplace on this decentralized network. That's awesome. 

Anatoly Yakovenko: 

This stuff is global and you build a globally scaled blockchain, it will be important, right? If you can get a billion people on a layer one, if you can do price discovery in a global scale, it will be important and cool and we should build it. 

Tristan Yver: 

And you're changing the world by doing it without a doubt, without a doubt. And then maybe later on, you can go to the part of having all the information on the blockchain as well. 

Anatoly Yakovenko: 

Exactly. That was something that we expected would take five years. You have this big dream like, "Can we build like a central limit order book on-chain, right? Can we even do this thing?" And then like, "Okay, we can, but how do you get to a point where that's the obvious thing that you can do, the adoption of it is clear and you have like the right partners to do it?" Because there's infinite number of pieces necessary to build it, right? You need liquidity providers, market makers. You need traders. You need people that care about using this product and iterating on it. It's a bunch of work. 

Tristan Yver: 

And I'd actually like to fast forward a little bit from there, because not now, maybe about three, four months ago and to this radical shift we've seen in the Solana ecosystem in the sense that that slow trickle of developers that we saw at the beginning is really turning into a torrent at this point and the excitement and the dev tooling. How is this feeling from your end? How has sort of your role as the leader of this shifted over the last six months? 

Anatoly Yakovenko: 

So what's been, I guess, different is that the work that the Serum folks put in in those early days in terms of building the APIs, and I don't know if you've been following Armani's building Anchor which is an awesome framework that just simplified development time for new people coming in from weeks to days, right? You can get something that would have taken you a week to build early on that takes like a day. So things like that are just ... It's part of the natural software cycle like you have a complex system and people look at it and they do the hard work if they have like this prize at the end of the road. They believe that it's worth it, right? 

Anatoly Yakovenko: 

They'll do the hard work, but then the learning that they built up and the pieces that they built to build that first product, they're all reusable in the next one. And you see this exponential growth from just the time that people put into developing the systems, developing these patterns and the knowledge that they're sharing amongst each other just because it's software and it's easy to share it, right? It's a piece of code you posted on GitHub, posted in Cargo and the Rust Toolchain will automatically download it. 

Anatoly Yakovenko: 

So you start seeing that cycle accelerate and accelerate and it's awesome. We had a thousand registrations in the first hackathon. This one is 7,000 and the registration period is, I think, not over yet. So it's crazy, right? So I'm excited that the level of innovation and the pace has really accelerated and the products that people are building are much more sophisticated. The first hackathon had a bunch of people that tried a bunch of random things. The second one had 11 teams that got funding at the end of it. 

Tristan Yver: 

Without us having to do anything because people just came in like, "Take my money," basically. And several of them are live now. They have live products that are on the market. 

Anatoly Yakovenko: 

So that's awesome, right? This is the most important part in anything, is like, "How quickly are you iterating? How quickly are you getting to market and failing and learning?" And our strategy building Solana was build those two pieces first as soon as they work. Swap the straw for sticks. That's good enough. How fast you iterate is really, really important to any kind of success. And we're seeing that in the ecosystem itself. So my job is I guess easier now. A lot of the early days, we would spend a lot more time with those folks getting them over those humps, helping them build those more complicated pieces and see where they can be very usable. 

Anatoly Yakovenko: 

But now, that stuff is there and they're able to learn it and pick it up on their own and launch even without us. There's projects that have built like working products that we've never talked to. They didn't come in through the hackathon. Just cool. 

Tristan Yver: 

I see that firsthand. A lot of what I'm doing on the Serum side is just incubation. Basically, I call myself an usher. I help these guys that come with ideas to get to commercialization basically and all the steps along the way. And it's really interesting because you're right. Now initially when I started in that role, people did have to talk to your team quite a bit more. Now almost never honestly. It's more likely they'll talk to Bartosz just to get some advice on structure than actual code on the Solana of blockchain and things like that. So totally agreed on that front. Do you still get to code sometimes? Are you now just at high-level executive decision-making meetings and stuff like that? 

Anatoly Yakovenko: 

I like to code, so every once in a while that I see something cool, I will jump in and write a shitty implementation, but I know somebody has to go and write all the tests and rework it. I don't know if it helps. I don't know if it helps when I do this and I hand it off to somebody else or would it better if I handed them off like a design doc, but I feel like it's productive. That initial swap implementation for Uniswap, that was massively surpassed by Radium. I wrote that first implementation and handed it off to Bartosz and John. 

Tristan Yver: 

Really? I didn't know that. 

Anatoly Yakovenko: 

It took about two weeks of restructuring my code and actually making it shippable, but I'm glad we had it because that came out pretty quickly when it was needed. So that kind of stuff, being a little bit forward-thinking, I get a chance to do that sometimes. I still sometimes jump into deep optimization work and design there. That's still probably ... It's weird we haven't done a ton of changes to the protocol really in two years. The core consensus protocol itself has been fairly stable. What we've been working on is just finding ways where we can optimize things, making a bit more horizontally scalable, making easier for validators that are running the software to throw hardware at it to fix problems. 

Anatoly Yakovenko: 

I would say we're still probably 5x, 10x inefficient compared to where we could be and just random optimizations all over the place. So maybe by the end of the year, we could get to 2, 3, 4x inefficiency and then it's going to be harder, right? Those last few years are going to take the next year just- 

Tristan Yver: 

That's right. I've heard from HFT devs, it's like you can do these big initial leaves, but then to get those nanoseconds at the end is the hardest part. I'm curious, when you say 3, 4x, does that scale linearly with transactions per second as well or it's a bit different? 

Anatoly Yakovenko: 

The TPS for applications is easier. The harder problem is the consensus and making sure that the network stays connected and doesn't partition during normal operation. And that's because if an application fails, they just retransmit, they just retry and they'll complain to us if it fails a lot, but it's not that like the network will stop and there'll be problems. So being able to make sure that ConsenSys can run in a very stable way at high performance with a very large number of nodes, we're always dealing with four opposing different variables. And when we start tuning those things, it just takes longer to verify that we didn't introduce some weird condition where things can go wrong because it's just complicated. 

Anatoly Yakovenko: 

We do like constant testing like CI nightly. There's big networks that we spin up and we try to destroy them like split them and stuff like that and make sure that everything's still stable. And like tweaking those parameters just takes a lot of verification time. 

Tristan Yver: 

I'm definitely a Solana evangelist. I think it's amazing that you can have a chain where you can have real big market makers, doing real market making on order books and the chain is maintaining fidelity. It's incredible. It's honestly incredible. The amount of moving pieces is out of this world. 

Anatoly Yakovenko: 

The harder problem too that people don't realize, in a lot of ways, it's easier ... There's a ton of hard work, so you make the chain fast, so you write to optimize, right? We dump as much data as you can into it and it tries to just suck it all in and order it and that's all it does, but then you have a bunch of users that actually want to understand what this data is and read it. They actually have to examine it. And that RPC nodes and scaling that out, that's the hard part. So on the surface, it looks like, "Sure, we can support Twitter amount of data messages coming in." But solely, Twitter generates about 5,000 messages per second. So people actually writing stuff into Twitter, it's about 5,000 per second. 

Anatoly Yakovenko: 

The hard part is how do you examine all this data, build all the feeds, construct all the threads and do it in real-time. You need probably 100x more servers to manage that while maintaining 10, 20, 30 million connections. So that's the hard part of actually scaling the stuff. The chain itself is a small component. If the stuff takes off, like we actually end up with systems where we have a billion people using Solana, Solana, itself, will be 100th of the actual rest of the ecosystem because you have to have all of these other components that are processing that data and structuring it so those billion people can actually use it and read it in some useful way. And that's going to be the hard part, but that's probably going to be a never-ending job. 

Tristan Yver: 

I imagine and I know that. So the RPC nodes have, in my mind, been the first real growing pain that Solana has faced, just at least from the user experience. I deal with a lot of the customer support user, frontend stuff. I'm curious to hear, I know there's a lot of people working on it now and people are coming to the forefront for IT solutions. Would you mind touching on a little bit on what is happening here with RPC nodes casinos and what is being done to resolve it? 

Anatoly Yakovenko: 

Yeah, so this is a classic database problem. You have something that's optimized for writing. It's very hard to read that state because the writing state is changing. So you have a system that's writing as fast as it can. How do you read information from it? The typical ways, you stop the writing, you read the current state and then you let the writing continue. But that causes that writing system to slow down, and if it slows down too much, it actually falls behind and then stops, loses consistency. There's a bunch of ways to work around these, different tradeoffs. You can build an index that's written to and then duplicate the index for writing and so you can split those logs and stuff, but all that stuff is complicated and nontrivial. 

Anatoly Yakovenko: 

And the one easy way to do it is you can basically just spin up more of these servers and you round-robin between them, so some of the writes go to server one, some of the writes go to server two, and when one gets too loaded, you stop and let it catch up and you move to another one. And that works fine. And that's how we were able to kind of get here. The challenge comes from hackathon projects spent two weeks, no sleep, building this project out as fast as they could, didn't optimize anything and then went straight to idea with 50,000 users a day, right? Really thrashing a lot of connections and data and demand on these hard-to-optimize systems and that's the challenge. 

Anatoly Yakovenko: 

Reality is that you need to do a bit of work on the application level, understand what is the data that it's fetching, cache it intelligently and correctly and then figure out like, "How do we split these applications, both from an application perspective and geographically to where if you have a big pile of users that are like, "I'll look up in one part of the world," but they don't necessarily cause the entire network for that application to go down, but there's a spike in demand that it's isolated. So a bunch of that stuff is all happening at the same time. 

Anatoly Yakovenko: 

At the lowest level, we have like a really cool group of validators that you know. Part of what's different about Solana was that it was never designed to run from home. Like Ethereum has this vision of like ... But Tarun says, "It's a lot easier to pull a cart with six horses and a thousand chickens." 

Tristan Yver: 

The chicken [inaudible 00:41:45]. 

Anatoly Yakovenko: 

So Ethereum is trying to pull a cart with a thousand chickens or trying to pull up horses. What we designed for was those smaller dependent validators, people that understand what managing Linux system is that can Google for local colocation and call that number and talk to the people there because their websites don't work surprisingly and order system and have it installed. So it's like a pain in the ass, right? You're not going to go click next on AWS and then launch something. You have to put in a bit of work, but the benefit there is you're at a high connection system with very cheap bandwidth that's really low latency and you have no virtualization, you've built the box yourself and you've optimized it. 

Anatoly Yakovenko: 

So that's our core validator group, people that can do that and they've been teaching other people coming into the space that, "It's not that hard. You can actually do it and it's fun because you're building your own systems and you're part of the future finance." So it's like you learned something, right? Not a lot of people think that they can deploy something at a data center over the weekend, right? They think it's a big deal. So we've actually had a really good group that's been building out an RPC network of servers that have optimized just for this all over the world. So that's been awesome to see. And that's just us getting more hardware, more really well -optimized systems around the world to handle all this influx of projects. 

Anatoly Yakovenko: 

I'm hoping that we will see something like, I don't know. I feel like there should be some kind of token for this, not because we need more tokens, but there's a cool component to tokens where it glues together like a social network piece and the use case and aligns everybody that's working on an open-source collaborative piece around the world, that they have nothing else in common besides like, "This is cool. We should build it," and the token makes it easier for everybody to come together and work harder and scale it faster, right? To be excited about it. 

Anatoly Yakovenko: 

And if it's also useful to humans and this is clearly useful, it's awesome to have really well-tuned optimized RPC systems. Feels like a token is the right economic incentive too. And I'm surprised like, I don't know. Infura to me is like it's great that ConsenSys built it, but it's stuck. I feel like the pieces that ConsenSys builds that they haven't been able to set free and let them go and actually let them run on their own have slowed down Ethereum development and innovation. Quite a bit. ConsenSys is like Qualcomm. 

Tristan Yver: 

They're big at that point. You're so right though totally. We had that clubhouse back in Jan where you we're talking about ... We're talking about different project ideas and things you'd like to see in the network which is actually probably my next question for you, but you're saying where you like to see something that brought 100 million people together, something that didn't matter what it actually was used for, but 100 million different people held this and had some identity with that coin. Little did we know it's going to be Doge. We should have just bought a ton of Doge back then. You called it, you were right, it was a phenomenon waiting to happen. 

Tristan Yver: 

We're just like, "Did it play it right?" But you really nailed that with that, man. I think like we've been seeing that now, some of these meme coins recently. And my next question for you is, what are some projects now that there are so many projects being built or already launched that you would be excited to see being built on this line of blockchain? 

Anatoly Yakovenko: 

So I think some of the tooling around what ... You've launched a token like COPE, right? There's a cool community in Discord and awesome developer that put it together and he launched it. And now you have this token and a community that loves it. How do you give him more features and tools? I think that's something that's underserved. We have a governance implementation, but you could literally almost think of like, "There's a community that could have an app store to pick and choose every permutation of Serum, DeFi and governance and other pieces and have COPE be the main token that glues it together and let it drive it, right? 

Anatoly Yakovenko: 

If that's what they want, it seems like communities are almost more important than the underlying protocols that actually do all the stuff that run ... The borrow-lending is less important than that Aave community, that Aave built. If you had that community already, you could add borrow-lending as a feature, right? It would get traction. I think there's something more interesting there in terms of product development, reshaping how products were built in Ethereum which is by protocol innovators and actually making it more like a community first by the folks that are connected more to the consumer, being more product facing. 

Anatoly Yakovenko: 

I don't know if there's a trend that people think that DOWS are going to be next. I think if they're going to be successful, it'll take some shape of this where you have decentralized communities that are legitimately drawn together by some vision, and then they pick and choose what features they want. Serum liquidity and the market makers that are trading on it are in itself a DeFi primitive that is a superweapon that doesn't exist anywhere else. How would you leverage that and COPE, right? That's an interesting problem. If you can find when those two things meet, it can be magic, right? 

Tristan Yver: 

I have one more question for you, Anatoly. Actually sorry, two more questions, one more technical and then one more personal. So the more technical side is, this I hear a lot, where can a dev start to learn things that will allow them to build on Solana? Where should they start? 

Anatoly Yakovenko: 

Do you have like the best post on this on Twitter? It's like Paul's guide on what are programs and then the link to Anchor like ... 

Tristan Yver: 

Perfect. 

Anatoly Yakovenko: 

Those are really good and then jump into Discord. There's a hackathon going on, so it's like a lot of team formation happening and folks helping each other. Bartosz, I'm sure you've heard this name a thousand times. 

Tristan Yver: 

A legend. 

Anatoly Yakovenko: 

The legend. The guy with infinite hours, but he's really helpful. A lot of our frontend devs are really helpful. And just one thing to keep in mind is a lot of the folks that got ... I think having a product post-hackathon that gets seed funding is a massive success. Those teams, a lot of them actually started in the first hackathon and didn't even get a prize. They failed, right? We didn't reward them and they continued and they won. They just built something, it didn't work or barely worked or the idea was done and they kept at it and then their second iteration is what worked. And a lot of the folks that we did give a prize in that first hackathon didn't continue. Sometimes failing is a good motivator. So keep that in mind and just push through it. 

Tristan Yver: 

And then to the listeners, just feel free to reach out to me anytime as well. Happy to be a resource there. And then my last question to you is, what's a lesson in your life totally that that has stood out to you and has helped you live a better life? 

Anatoly Yakovenko: 

Man, that's a good question. I was really into the Eightfold Path when I was in an impressionable age. It's this idea that you should try to ... You don't have to strive to be the best. You just try to avoid being bad. Try to not ruin the world, right? Try to avoid doing, spending ... Avoid carbon emissions because it's bad, right? It's a small incremental thing that doesn't really immediately have a negative impact, but if you can avoid it, then avoid doing those things and keeping that in mind has been something that's always been in my mind. I slowly stopped eating meat. That took a while. Not that I never do it. If I go home, my mom cooks meat, I'm just going to eat it because I don't want to argue with her. 

Anatoly Yakovenko: 

But those things are, I don't know, important rituals. If you have those, keep them going. They'll keep you sane through the ups and downs. It's not always about product market fit and growth. Those are things that you do day to day as a job. Having something to keep you going in your place in the world as a human being, I think, is pretty important. 

Tristan Yver: 

Absolutely agreed. Thanks so much for coming on the podcast. 

Anatoly Yakovenko: 

For sure. 

Wrapping up with Anatoly on Solana

Another interview in the books! Anatoly does a great job of explaining complex concepts like the blockchain and proof of work. Both of which are key to understanding the value and impact of cryptocurrency. For more technical information like this, tune into future episodes or browse the resources on the FTX blog. 

FTX

Buy your first crypto in less than 3 minutes

Get started