Play Games
2 months ago
75
"Beep."
On 10th January, 2018, Marcin Momot, CD Projekt Red's global community lead, fired up the social media engines and hit "Tweet". This was the first tease of CDPR's marketing push for Cyberpunk 2077, ahead of appearances at E3 that year and again in 2019. A group of CDPR developers, huddled around Momot's computer in the bustling Warsaw office of pre-Covid times, stood and watched as the Cyberpunk 2077 account's following skyrocketed.
It's just a tweet, but Momot, who joined the company as a humble community manager in 2011 and has since risen to the role of global community director today, now describes this moment to me as a "very, very big piece of history" for the studio. Exactly five years after the initial "Bullets" teaser trailer of 2013 (tagline "Coming: when it's ready"), this was the starter cannon for Cyberpunk 2077's proper reintroduction to the world. Preparations ramped up for the pending "re-reveal" that summer. Excitement was "through the roof" at the studio, while here in the outside world hype for the game - already sky-high following CDPR's critically-acclaimed The Witcher 3 and its expansions - began to explode. As Momot puts it to me, "I think people were thinking that this game would be like the second coming of Christ."
The promise was clear: a rich adaptation of a tabletop cult-hit. A true role-playing game, with a mature and branching story and range of character classes, abilities, and approaches to in-game problems. A world full of life and detail, in the kind of grim and gritty mid-future setting that, surprisingly for triple-A video games, at that time still felt somewhat underexplored. In fact that was quite literally the promise CD Projekt Red made. And as The Witcher 3 arrived, with unanimous acclaim, that promise began to look quite special.
What followed was one of the most disastrous launches in video games. Three rapid-fire delays. An extended period of crunch. A release so buggy it came with a warning label on Xbox, and was pulled from sale entirely on PlayStation. An indefinite delay of next-gen versions, a 30 percent stock price plunge, a period which CD Projekt Red developers described to me as "devastating", "hopeless", and "heartbreaking", which joint-CEO Michał Nowakowski calls "one of the worst moments of my life", and which threatened the very future of the company.
But fast forward four more years to today, and that moment almost feels like ancient history. Cyberpunk 2077, and its critically acclaimed expansion Phantom Liberty, have sold tens of millions of copies. There are few complaints about performance or bugs. There's a popular Netflix anime, Edgerunners, and another show, recently teased, on the way. Momot described public sentiment as "night and day", the game itself tipping over into "Overwhelmingly Positive" user reviews on Steam earlier this year. Cyberpunk 2077 has risen from the dead. This is the story of what happened, and how CD Projekt Red brought it back to life.
Image credit: Eurogamer
To understand Cyberpunk 2077's cycle of rise, fall, and rise again, it's best to start at the beginning. When Cyberpunk was first announced, via an on-stage presentation with tabletop game creator Mike Pondsmith in 2012, it was barely a prototype. (It was also still just 'Cyberpunk' - it wasn't until its first 2013 teaser trailer that it picked up the 2077 in its title.) It took until June 2016 for development to really begin in earnest.
"From that point on, we started working with the [engine] technology, we started designing everything and so on," explains Paweł Sasko, at the time a lead quest designer on Cyberpunk 2077, and now the associate game director of its unnamed sequel, codenamed Orion. The bulk of the team at CDPR, including Sasko, didn't move over to Cyberpunk 2077 until work on the final expansion for The Witcher 3 was out the door, and so he puts full-scale production time at around "four-and-a-half years". Contrary to popular belief, what with the eight-year gap between first announcement and Cyberpunk 2077's launch, by modern triple-A standards, "the production was, I would say, really fast."
Sasko and I are talking here at the Game Developer's Conference in San Francisco during the spring of 2023, the first of two conversations we have at GDCs either side of Phantom Liberty's launch. Diminutive in stature but bubbling over with giant ideas, Sasko is affable, chipper, and earnest, immediately ready both to divert onto grand, philosophical tangents or turn inwards into deep reflection. Each conversation with him feels like hitting the same setting-the-world-to-rights moments you might have deep in discussion some late evening at the pub, only here with genuine insight to show for it.
On both occasions he's also clad, head-to-toe, in CD Projekt Red gear, complete with bomber jacket, backpack and T-shirt. A true believer - although on the second occasion, after the success of Phantom Liberty, it's notable he's joined in uniform by a few dozen more of his colleagues. The studio wanted to make a point of being visible, I'm told - not for a victory lap, but to signal to other developers that they're there, and ready to offer some candid advice from their experience. Less a show of pride, it seems to me, and more a point of resilience.
In those four-and-a-half years - originally scheduled to be closer to three-and-a-half, before the many delays - the studio had to build the game's engine effectively "from scratch". RED Engine 4 "was written almost from zero," Sasko says. "There's very few things that were moved on from The Witcher, from previous iterations of that engine."
It's very much the norm for different elements of a triple-A game to only fall into place as development progresses. With Cyberpunk 2077, however, the impression is of a game where multiple challenges were always in play at the same time, often bleeding into one another as a result. While foundational work was still underway on the game's engine and general game design, for instance, CD Projket's PR, marketing and community teams had spun up the hype machine in earnest, and in doing so found themselves in the middle of the first of this game's many crises.
"It was really easy to get caught up in the hype. But also, the stuff that we had back then - it was really, really good."At E3 2019, less than a year out from Cyberpunk 2077's original release date, a splash of Keanu Reeves-powered "you're breathtaking" excitement threatened to be overshadowed by growing controversy over an in-game poster. The artwork, an in-game advertisement for a fictional drink called Chromanticore, depicts a transgender model with an erect penis showing through a tight-fitting leotard alongside the slogan, "Mix it up". As Kasia Redesiuk, CDPR's art director and designer of the poster explained to Eurogamer at the time, the poster was an intentional commentary on the "sexploitation" of Cyberpunk's corporate capitalist future. "So yes, we have a person with both breasts and a penis on an advertisement, done on purpose, because it's terrible to exploit people's bodies like this," she said. "It's a terrible thing to say 'mix it up'. We're emulating what a company would say in Cyberpunk 2077."
Nevertheless, the controversy was ill-timed, and heightened by a series of issues with the company's tone on social media. A group of poor-taste tweets from the account of digital storefront GOG, managed separately but owned by parent company CD Projekt, included a mocking use of a trans rights hashtag and ultimately led to the firing of an employee the year before. Earlier that year, the Cyberpunk 2077 account - this one managed by CDPR - made a "did you just assume their gender?" joke. The result was an erosion of trust that the developer could handle these kinds of topics with the right level of maturity - hence the concern at that poster.
"I remember when those things happened," Momot says, "I actually talked to the person who sent those out, almost immediately as they were happening." The team "internally discussed this, I feel like we've learned our lesson, and I really hope this sort of situation won't come up in the future because I think we are much wiser, a lot more experienced." The team, Momot explains, had discussed the kind of public tone they would take on social media as far back as 2011, right when plans for the game were in their infancy, deciding they wanted to "experiment" and be "a bit more rebellious, a bit more kind of in-your-face". Momot ultimately still feels it was "the right fit for the game, and represented it well, what the game world is about. But then unfortunately we had some mishaps and those, yes, I do have a regret for those."
Radek Grabowski, who led the developer's PR department since around 2018 and is now CD Projekt Red's PR director, puts it in black-and-white terms. "With full confidence, I can tell you there was no intention for CDPR to raise controversy," he says. "This is the part [where], if I could change the past, I would love to change it. But it's not something we had planned."
Sorry to all those offended by one of the responses sent out from our account earlier. Harming anyone was never our intention.
— Cyberpunk 2077 (@CyberpunkGame) August 21, 2018The whiff of culture war issues still lingering in the air, and crucial elements of design still very much in progress, Cyberpunk 2077's hype nevertheless began to snowball. A combination of Hollywood glamour, studio reputation, highly-impressive trailers and glowing reactions to hotly sought-after behind-closed-doors appointments at two E3s (including one from this author) sent expectations soaring.
It combined with the controversies to create an unprecedented amount of pre-release noise. More pertinently to the fate of the game itself, that noise fed back into the team.
"I've never, ever experienced this level of hype around a game," Grabowski says. Around six months after the game's blockbuster showing at E3 2019 came the announcement of Cyberpunk 2077's first delay, from a 16th April 2020 release date to 17th September 2020. It did nothing to slow down the momentum. The hype was enough to stretch the studio on the communications side. "For me and my team, to deal with the amount of enquiries we'd been getting - interviews, cover stories - that was something unprecedented. And that meant additional workload," Grabowski continues.
Delays added to that again. The studio had originally mapped out roughly a year of full-scale promotion for the game, and eventually had to almost double it. "You have a certain line-up of communication beats you want to convey, during a set amount of months, then the game gets delayed and you have to kind of stretch the communication to fit within that spectrum," Grabowski explains. Night City Wire, CDPR's series of deep-dive videos, was an example of the studio essentially vamping to fill the void, "invented when the train was already running". There was also an unusual second round of hands-on previews conducted remotely (the studio had to "invent how to do remote hands-ons", Grabowski says, as the wider industry grappled with Covid-19 lockdowns) to appease the clamour for more hands-on time.
By the time it got to Cyberpunk 2077's final, full launch, the mood on the PR and marketing side was "like after running a marathon," Grabowski says. Not only because of how long it took - almost three full years of drawing out a campaign - but in the rush of having finally accomplished something. The sense was that, like The Witcher 3, the team was onto another winner, and that the extended effort would be worth it.
"It was really easy to get caught up in the hype," Momot says. "But also, the stuff that we had back then - like all the materials, all the promos, all the pieces of comms - it was really, really good, and people were getting really excited. I think we felt that excitement, internally, as well.
"Sure there were some concerns," he adds. "As usual with every project that we have, there are obviously those reality-checkers, right? Like, 'Will we be able to deliver, is it really going to happen?' But it was exciting."
If the game's pre-launch marketing felt like a marathon, development itself might be more accurately described as a perpetually extended sprint, with a Bloomberg report suggesting some developers had endured company-enforced overtime for more than a year. And yet strangely, across everyone I spoke to at CD Projekt Red, the mood before launch was almost unanimously described as one of optimism.
"Just before the release, I was really optimistic, and I was really proud of what the team had built," Sasko says. "I knew the game had a great story, great characters, and that's what I was focusing on, I was lead of the quest team at the time."
Perhaps ominously, he adds, "Partially, also, I hadn't been completely aware of how all of the components worked together."
What happened next has been well documented. At first, Cyberpunk 2077's hype spilled over into the initial moments of its release. Early reviews - limited to just PC code, and after less than a week of access to it - ranged from lukewarm to emphatically positive, and in those first hours and days - before the scale of Cyberpunk 2077's disastrously buggy state, in particular on consoles, was revealed to the outside world - the CD Projekt Red team truly believed they'd pulled it all together, right at the death.
Marcin Momot describes a period of initial positivity, as the game first went live. In fact for a brief moment it seemed like an extraordinary success, smashing the record for concurrent players in a single-player game on Steam, with over a million playing at once. The next closest at the time was Fallout 4, in 2015, with less than half that figure.
Momot went through his usual "ritual", as he describes it to me, of playing the game at the moment it goes live around the world, to be a part of the collective experience. He finished work at "two or three" in the morning, having needed to check everything went live properly, and plus, "you have a lot of emotions going through you, you read comments, you see what people are doing," meaning amongst the adrenaline and hubbub of a major launch, it takes a while to clock off. Right after that, in the middle of the night, he played the game. Just for "20, 30 minutes," to be a part of it all - and went to sleep, in that moment, happy.
"Until the end, we were fighting, fighting, fighting, and then, yeah. What happened, happened."Radek Grabowski, similarly, describes a period of briefly enjoying the early praise, "the positive stuff in the reviews, people appreciating the storytelling, or the characters, the way we conveyed the Cyberpunk world in this game. That was heartwarming." There was already talk of bugs filtering back, he says, but this wasn't unexpected. "The gaming population… do not remember that The Witcher 3 launch wasn't near perfection," he recalls, and "required a couple of patches just to be at the state it's [in] now."
He's not wrong. Often overlooked in the now-fond recollections of The Witcher 3 was a noisy launch day drama around a "graphics downgrade" from the visuals flaunted in early trailers to those of the final release. Not knowing the intricacies of games development, Grabowski says, he was "expecting us to amend those technical issues in the future" once again with Cyberpunk 2077, just the same as the studio did with The Witcher 3. "I was thinking: we're gonna fix it."
Others shared Grabowski's thoughts, and that same hope the studio would pull things back as it had done with The Witcher 3, even those much closer to the game's code at launch. The Witcher 3's development - also challenging and full of crunch - and its last-gasp success ended up something of a curse, haunting the studio as it toiled on Cyberpunk 2077, even if staff didn't yet know it.
Charles Tremblay, at the time CD Projekt Red's lead engine programmer, was right there "in the trenches," joining the company during development of The Witcher 3, seeing it come together right at the end, and moving over to Cyberpunk from day one of its full production. Talking to me via video call, Tremblay, now CDPR's vice president of technology, sits in front of a row of five hulking pinball machines which he takes apart and fixes for fun, helping him to unwind in his spare time. One human-length pinball board from an in-progress repair sits propped up, landscape, on a workbench off to one side, a galaxy of detail in its mass of cables, levers and wires. "First of all," he says, "I was, in general, very happy about the tech we developed, and I'm still extremely proud about what we achieved."
At the time, the engine team's task was vast. The big issue they were wrestling with was the mechanical hard drives of that generation's consoles, he tells me - and indeed the drives of still plenty of older PCs. The engineers were "extremely struggling" with the amount of data they had to stream from the hard drive at a given moment. "The game was massive," he says. "Massive amount of content. But there's a physical limitation about how much I/O you can get from those consoles." (I/O refers to the reading and writing input and output of information to and from a drive.)
Yet the team remained steadfastly confident they'd sort the issue in time. "We were pretty tired - because again, it's going to the finish line, you're so focused. It was the same with The Witcher 3 - super focused, still working on the project." At one point, Tremblay says, before Covid hit in early 2020, they were "super confident" they were going to make it. "We got very good progress" at that point, he says, which likely contributed to the belief, at the moment of each consecutive delay, that each holdup was the final bit of time the team needed to crack it. Then came that pandemic, and at each moment the magic fix remained just slightly out of reach.
"Unfortunately, we did not manage to get in the best shape for launch," he says, but the team was, again, "very confident we could nail it for a patch." But to their dismay, still no solution came. "Until the end, we were fighting, fighting, fighting, and then, yeah. What happened, happened," he says, with some detectable frustration. "I think if you look at the PC people using SSDs," he notes, which have an order of magnitude faster read and write speeds, "it was much, much better for them. But unfortunately, on the mechanical drive, we did not manage to nail it down in the end."
Soon enough, the general public - playing on the then-current generation of PlayStation 4 and Xbox One consoles, and a much wider range of PC hardware than most of the enthusiast press had used for the game's restricted PC reviews - began to make progress through the game in earnest. And as much as the team hoped and earnestly believed it had done just enough to get the game over the line before fixing any lingering issues with a quick follow-up patch, the tide rapidly and dramatically turned. And those remaining issues were much bigger than anyone at CD Projekt Red had realised.
Image credit: Eurogamer
From an all-time high just before Cyberpunk 2077's eventual launch on 10th December 2020, CD Projket's share price plunged almost 50 percent in the week that followed. An avalanche of memes mocking the sea of bugs - characters T-posing, exposed genitals, environments popping in, cars slamming into suddenly-visible obstacles, players being flung across maps - met an equally powerful wave of fury from fans who felt, with some justification, they'd been deceived into buying an eminently faulty product.
"Slowly, slowly the information started creeping in," Paveł Sasko says, "that there are issues with this, with that." The moment was "really heartbreaking," he continues. "Personally it touched me a lot, but the hardest thing for me was the fact that a lot of people in my teams have been people that I hired myself, and trained myself or with other people, and the thing was, I know they hadn't ever shipped a game yet. Or they hadn't shipped a game that was broadly acclaimed as a huge success.
"When you work on triple-As," he continues, "you do it like, what, five, six, seven years? These are big games - sometimes 12 years, you know, depending on where you are. And sometimes you have that person who joins the team as a junior, works for years, gets to some position or maybe becomes a specialist, and then they ship the game, and then they don't see that acclaim, that love coming. I think that was the hardest part. Because I saw a lot of people in the team who have been believing, and doing their best, and the way they reacted, how hard it was for them to see that… I wanted them to be part of that success."
Some at the studio began to question CD Projekt's abilities; whether the studio could ever turn it around or if it was up to the task. "A lot of people had serious doubts," explains Momot, "and I think it's a no-brainer when a situation like that happens, when this is the reception you get for the game you've been working on for nearly a decade. It's bound to leave a mark on you."
"I think it was, actually, one of the worst moments of my life. I definitely wouldn't want to go back to that place. It was not a pretty place."Roughly a week after launch, Momot woke up to a phone call early in the morning from his boss. "The title of the episode, if there was a show about my life, would be 'The Phone Call'," he says. The call is about the game being removed from the PlayStation storefront by Sony, with the platform-holder also offering full refunds to anyone who bought it on PS4. The immediate plan was that "we have to get through this, we have to communicate this to people out there, and we have to own up to this," he says.
Momot describes his feelings then as "devastated. Hopeless. Just totally, totally - without any sort of aid, I think 'hopeless' is the one word I would use. Because it's like: okay, everyone is bashing you. The hits are coming from all angles. What can you do? You feel like you're really surrounded by all this hostility and dialogue that's really not going in the way you were hoping for it to go.
"You know the feeling if something bad happens to you, and you would like to just hide under a blanket, and you're sitting on the couch and really becoming one with the couch, going deeper and deeper? That's the feeling I had."
"The very first month I think was the toughest," explains Michał Nowakowski, CD Projekt's joint-CEO. (The company's other joint-CEO remains Adam Badowski, while Marcin Iwiński, another former joint-CEO and previously the public face of CD Projekt, stepped down in late 2022, taking a role as chairman of CD Projekt's joint supervisory board). Talking to me in the summer of this year, Nowakowski, in a smart leather jacket, leans forward over his laptop in a high-ceilinged room of the developer's main Warsaw office, flanked by large movie posters for The Goonies and It on a tastefully lime washed wall.
"It took us a moment to actually analyse the situation," he explains, pinning a slight delay in realisation on the impact of everyone working remotely during Covid-19. "It wasn't as obvious as it would be normally, where around launch we would all be in the office together, huddling - I remember The Witcher 3 launch when we were literally in the office still in the wee hours, like three, four AM checking out the reviews, talking, walking between rooms.
"But we knew that we were in deep trouble," he says, "and the decision that we needed to do something about it was an immediate reaction we had."
The backdrop at the time was the Christmas period, as Covid continued to rage. "Lots of festivities. It's almost cringey," Nowakowski says, "because you feel super sad inside and everybody - you know on the TV and around the whole world, yes it's Covid and everybody's at home, but it's joyful music. Home Alone is on the TV. And you're feeling like crap inside.
"I think it was, actually, one of the worst moments of my life to be honest, on a personal level. And I definitely wouldn't want to go back to that place. It was not a pretty place. And it was really, personally, hitting, you know, my soul. But I think it was more so [a dark place] - or on a similar level if not more - for everybody in this company. When you wake up and you read just the worst in the media, social media, and so on. It doesn't make you feel happy."
That's not to say he didn't understand the reasons for it, he adds. "We did wrong and you have to own up to it and make changes based on that." It was difficult for him, he says, "because you feel personally responsible. So for sure not great."
In the immediate aftermath, CD Projekt held large group meetings to internally communicate a plan - but even this was hampered by the pandemic. "In the past, whenever there was any situation or crisis, you're able to get people in one room," Nowakowski says. "If you're in a call and there's a hundred people, because of the number of people they cannot even switch on [their mics], because if everybody starts talking at the same time it's not even a conversation anymore." Leadership tried to put together a Slack channel for people to ask questions, but "it's not ideal, let's put it like that," he says. "I'd rather for us to be in a big room where a mic is being passed between people when we sit and talk, and everybody has a way to vent whatever sits in them."
In those immediate crisis meetings with the whole team, Nowakowski explains, studio leadership "basically tried to communicate to people how and why we're planning to do what we were planning to do, and what we believed might be the outcome, and where we want to try and take the company. We tried to say that very, very, very early on." But after a marathon development period and the huge pressure around launch, it was already - understandably - one ask too many for some.
"For some people of course there was no buy-in from the get go [after launch]," Nowakowski says. "They were just burned. And they had the right to it, I mean, it was emotional for everybody, for some more so than for others, for sure. Some people decided to leave. Some people stayed for a little bit, and decided to leave later on." Despite what might have been an expected exodus, however, "we actually didn't have as massive a turnover as I think it may have seemed from the outside. A lot of people stayed through."
The studio set about trying to find the reasons for such a catastrophic launch - and what, structurally, might have enabled those reasons to come about. The team broke for the Christmas and New Year holidays, which perhaps couldn't have come sooner given the months of continuous overtime for many. And then, in the new year of 2021, CD Projekt Red's developers set about uncovering what had gone so horribly wrong.
Across all of the developers I spoke to for this piece, it's surprising just how similar the suggestions are for what exactly went wrong and why. Time has passed already, of course, which inevitably compresses and simplifies recollections of the past (as of writing, it's now more than four years since Cyberpunk 2077's original public release date). Company-wide inquiries will also naturally have an effect on refining the narrative. Nevertheless, each response I get seems clearly tinged with personal experience. There is "no one answer," as I'm told time and time again. Still, there are several issues combined that continue to come up.
There are specific technical reasons for why Cyberpunk 2077 ran so dreadfully on some PCs and on consoles in particular - and why it continues to be less-than-stellar there even now - and likewise for many of the other, additional bugs which had causes of their own. And then behind those, enabling and exacerbating them, are what you might call structural problems with how CDPR operated and reacted - or failed to react - to challenges along the way.
"You have a choice: either you show a T-pose, or you hard crash. What do you prefer? We prefer not to hard crash."Amongst the entrails of his many pinball machines, with the tranquillity afforded by distance and the perspective of hindsight, Charles Tremblay outlines the technical causes of Cyberpunk 2077's launch day mess. The aforementioned streaming issue with mechanical hard drives was one - and was in fact cited by a somewhat shell-shocked looking Marcin Iwiński in a brief apology video in early 2021. "This is the number one issue," Tremblay says. "Number one issue for all the funny bugs that you saw, it's usually this."
This was mostly because of the game's size, and what it let you do. "If you drive very fast, you can stop the car, run in a building, now all of the quest has to be ready, everything has to be ready, animations be ready…" Tremblay continues. "We really, really tried to have, like, this seamless experience of not having fake loading screens, as much as possible… we don't have those fake doors, fake corridors, nothing. Everything is actually streamed - you go in the elevator, there is no fake loading. The whole system is always running, it's the same seamless system for everything." With all this to stream from the hard drive, but a set limit to how quickly that hard drive can read or write, the team had an almost insurmountable problem with getting it to work.
There were other issues, however, that contributed nearly as much. One was a problem with memory, known as a memory leak, where the amount of memory the game uses increases the longer it's left running - and which was especially problematic on consoles, where it's possible to suspend the game without fully closing it for dozens of hours at a time. "In theory, the game could run infinitely, and at some point we'd get some fragmentation issues or memory problems, where we now have no more memory to tackle some of the animations or something. And then you have a choice: either you show a T-pose, or you hard crash," whereby the game would shut down entirely and in the process wipe any progress since your last save. "What do you prefer?" Tremblay asks rhetorically. "We prefer not to hard crash." And so: many, many T-poses.
Beyond that, a litany of bugs borne of the combination of everything above happening at once: the streaming issues, the memory issues, and the sheer scale and complexity of what's going on behind the scenes in running the game. "It's still today a massive game," Tremblay says, "lots of complexity, lots of interconnected systems. So sometimes you have the streaming that's not working, memory is a problem, and then you have all the gameplay systems that expect [those to work fine]. And then when you have all those permutations of conditions, you multiply by one million people, then you have just one weird bug - you're like, 'Oh my god'."
It meant the studio found it "super useful" to get actual save files from people, "because, okay, we tested totally, we never saw this, I'm going to the JIRA bank" - the system where developers log and track bugs and their fixes - "yeah, I don't have this. Please, please, please, we need to reproduce this internally."
If Tremblay describes his and his team's engineering work as being in the trenches, Karolina Nieweglowska was right on the very front line. Nieweglowska started at CD Projekt Red as beta test director on The Witcher 2, and is now associate director of the player experience and safety team. It's an unusual title and an unusual role.
Nieweglowska describes it as a mixture of everything from technical support to communications, as she oversees the team which receives support tickets from players; analyses data; writes patch notes and system requirements. She also liaises with Tremblay's programming team, plus another team that specialises in communicating technical information, and then the PR, comms, and community teams who deal with players and fans directly. So when Charles Tremblay hears of a bug and needs more information, that information travels, in both directions, through Nieweglowska and her charge.
We're talking in person, earlier in 2024, at the same GDC as my second chat with Paveł Sasko. Nieweglowska, fresh off the podium of a session of her own, is slight and precise; careful, specific, and methodical with her wording so as to always be fully accurate. At one point, in trying to summarise how she describes her role and how she handles it, I suggest that it sounds a bit like how Star Trek's Scotty worked at the coal face and also reported back to the bridge, but that's not entirely right, as she notes.
"That's fair, that's a nice assessment. I like it. But I'm also thinking Uhura. You know, the communications officer. Not to take away from the PR team, but we have a pretty large volume of communications, like every day hundreds and hundreds of communications coming from a team which is embedded in the company. So it's not an unknowing support team which is outsourced somewhere far away and they don't know anything about what is happening in the company. They know everything. And they need to figure out what they can say and what they cannot say, and how to say it in such a way that our friends and all the other teams will be fine with that.
"I think the analogy that you're looking for might be some kind of merger - like in Voyager, when two characters got merged at some point." The character might've been wrong, but I was on the money with pinning her as a Trekkie. After working through a few ideas - "I've had 13 years to think about it" - she settles on one that meets the criteria: "a cell wall - in an organism you have a cell, and the wall of the cell knows what to let in, what to let out. We are in charge of letting those inputs in and figuring out how to make them into a constructive thing so the cell can work on it, and also all the time on what goes out."
How did it feel at the time, being in a technical support role on a launch like Cyberpunk 2077? "It felt busy," she says matter-of-factly, adding that the team was more prepared for launch than most others at the studio. "We already knew that there will be a big release, we were already expecting the unexpected - in our field I believe you need to hope for the best, prepare for the worst. So we were preparing… but then it hits, and people get to work."
Nieweglowska references her team's Slack channel from the game's launch day, on 10th December 2020, and describes "a flurry of activity" in jumping on tickets. "It was literally, 'Oh there's this AVX crash, what do we do?' Okay, so I talked to programmers; do we reply to players to just confirm it? Do we need to write something about it?
"Maybe I will disappoint you here," she jokes, "about the emotional aspect as it were. Of course like everyone in the company, it was very emotional for us, and difficult. The release was difficult, but support teams deal with constructive feedback on a daily basis, you have to be prepared."
She explains that PC, as a platform, had the most tickets by some way "because the PC has the greatest possible configuration abilities." It calls back to the issues Tremblay cited above: "it's the question of complexity. The number of possibilities is so big."
While Nieweglowska is profoundly stoic about her team's experience at Cyberpunk 2077's launch, she's also a symbol - not of what the company got wrong, but what it could have achieved had it done things differently. She serves as a link to the studio's systemic woes, and an example of how it ultimately learned to fix them: Nieweglowska's team was one of the only ones at the whole of CD Projekt Red to be actively and regularly communicating with anyone else.
Image credit: Eurogamer
The most obvious of CD Projekt's wider problems, at least to those outside of the company, was the pandemic. Tremblay cites it as a turning point in the team's progress on fixing the streaming issues, before the disruption of shifting to remote work in early 2020. Sasko described it as having "the worst possible timing," in the lead up to the game's initial late-spring launch. Likewise, it’s where the studio leadership pins a major portion of the struggles.
"I think I would start with Covid isolation," says joint-CEO Michał Nowakowski, when I ask him plainly what went wrong. "It's a lot of small things - and in hindsight everybody's very, very smart, obviously, and so are we in hindsight, super smart about it." But Covid's impact was undeniable.
"I was actually one of the people who publicly went on the record and said that we are a little bit more efficient than we were," Nowakowski says of the move to remote working, "which actually seemed to be true at a certain point. But I did not - and I think we did not, as an organisation - appreciate how much we lost. Of human interactions, and very intangible small things happening. In the corridor people exchanging thoughts, just casually talking about the project - which seems like they're 'not working' but they're actually exchanging valuable information.
"Everybody was working in their little universe, and we were struggling to figure out a good way to work together - and now we don't have much time, because we have a game to ship.""In order to get on a call," he continues, "you have to have an agenda, you have to have something specific to talk about - and it works for some things. For some matters, it's actually tricky… I think a lot was lost there. More than we probably thought at a given moment in time."
Meetings became "super awkward," Tremblay says, recounting the early-pandemic teleconferencing woes many experienced. "I think we didn't even have the concept of 'raising a hand' at the time," he jokes. Certain people began "dominating" calls because the natural flow of in-person interruptions didn't arise. Some people lacked cameras, adding to the confusion. "You had no idea if they were there, like 'Hey David, are you there? [long pause] 'Oh yeah I'm here'.
There was no bumping into colleagues in the kitchen, or standing together in front of a whiteboard to iterate fast on solutions. "Everybody was working in their little universe, and we were struggling to figure out a good way to work together - and now we don't have much time, because we have a game to ship."
More tangible losses came from staff dealing with less than stellar home internet speeds, compared to the company's cutting-edge infrastructure. "It takes the whole night to download anything," Tremblay says. The team tried streaming the game to their homes instead. There was "a lot of learning as we go", he concludes, which in terms of development time was "definitely not free."
Other issues, though, were directly of the studio’s own making. All fall under what Nowakowski describes to me plainly as "the way we developed games".
One of those, as Tremblay explains, was a historical approach of always prioritising pushing the graphical boundaries on PC, with other platforms following from there. "If you have an uber PC, you have a machine, and you give us good money?" he says. "Well come on, we want to have the best product we can for the machine." CD Projekt worked to the notion that if it could get something running on PC, console performance would eventually catch up if simply given more time.
Another was communication, an issue exacerbated by Covid and, notably, the studio's rapid growth. CD Projekt Red went from a team of around 330 to 350 developers on The Witcher 3 - including the studio's back office at the time - to upwards of 1000 people by the time of Cyberpunk 2077 (and now around 1200 or so today).
"When we switched to Cyberpunk, it was so many new things," explains Paveł Sasko. CD Projekt was suddenly handling a new IP, a shift to first-person perspective, new mechanics in shooting, stealth and driving. "All layered over with actual continuous growth of the studio," he continues. "Higher complexity always requires higher complexity of the tools, higher complexity of the process, communication, all of those things. In a way it was a consequence of this natural growth. When we've been growing and complexity was growing, there was a moment when it was really difficult to make it work with that big of a team." The knock-on consequence of that was one of "siloing", a word that repeatedly comes up in almost every conversation with CDPR.
Sarah Grummer, the quest designer behind Judy and Evelyn's hugely popular storyline in Cyberpunk 2077 and the opening mission of Phantom Liberty, described this as "the biggest problem" at the studio. "Siloing and a bunch of communication issues," as she put it, resulting in issues as rudimentary as two developers needing the same file, and accidentally "blocking" each other in working on it at the same time.
Sasko gives another example. "We were using 'waterfall'," he explains, naming a specific kind of development methodology. "In short, waterfall is when you just make one thing after another. So when you make a character, you draw a concept art, then you model, then you rig it, then you animate it, texture it, polish it, that's it. You make your hair or facial hair, whatever else is needed, and it's basically one after the other. A lot of games still use waterfall, especially smaller teams, because you can totally effectively build games like this."
But issues arise when working at scale, Sasko continues. "When we have a team of 20 people - a typical size of our dev teams - it's easy to just stick to communicating with each other, and make sure that, yes, we are creating the best quests, the best concept art, the best code. You think that everyone's kind of on the same page with you and you start creating silos." This was the case on Cyberpunk 2077, even as internal reviews tracked each team's progress. "We had a review focused on gameplay. A review focused on narrative. Review focused on art. A review focused on music." Lessons and progress from each one rarely overlapped.
Finally, subtly, and yet perhaps most fatally, was the combination of the studio's own ambition, and its resounding confidence in being able to deliver. Each delay came with the sincere belief that it would be the last - a leaked email to developers from Marcin Iwiński, after the second delay of three, expressly promised it would be "the last stretch". The sentiment remained that "we were not in a position that is going to be very different from where we were at launch with The Witcher 3," Nowakowski explains, citing that albatross again. "The feeling was that it was going to be a bit bumpy, but we're going to be able to fix it relatively fast - that was literally the feeling that was inside the company."
Naturally, plenty wonder why the studio didn't simply delay it again, for longer, to properly allow for things together. For one, as Nowakowski says, the breakdown in communication and outsized confidence meant the decision-makers simply felt they didn't need to. The second was sheer exhaustion.
"The team was really, really tired," Nowakowski says. "You could almost feel that people just wanted some sort of closure. If we pushed it further, probably - or at least that's how we felt - that would mean a lot of challenges for the team. It turned out that, because we didn't, it was probably even harder.
"We figured, a) it's going to probably be fine, and b) it could actually mean more damage for the team if we're gonna keep going. And we didn't realise we were in so much trouble, I think, most importantly. Because if the first part was clearly visible, I don't think we would have taken that decision - despite number one and number two. Nobody sane does something like that to a game that's been in development for so long, that we were betting on so hard."
Altogether, it made for a damaging cocktail that couldn't be unstirred. Ambitious plans for a seamless open world with no loading tricks; an initially simultaneous release across nine platforms (old generation consoles, 'pro' editions, new generations, PC and Stadia); visual fidelity to push the limits of high-end PCs; a series of new, systemic mechanics alongside a new IP; all combined with an approach where the team built its own engine as it went, almost tripled in size, and moved to emergency remote work at the height of a pandemic. Throughout all of this, CD Projekt continued to hold faith in the methods that had seen it through past launches that went down to the wire. Michał Nowakowski described it to me best, in labelling the studio's unshakable optimism as, simply put, "magical thinking".
Eventually, CD Projekt Red had at last realised what the problems were. Exhausted and crestfallen, the team took time to reset over Christmas and New Year. Then, at the beginning of 2021, those who didn't leave the company returned to work, and put the same unshakable faith that partially got them into this mess to another use: digging themselves back out.
Image credit: Eurogamer
One of CD Projekt Red's key solutions to its biggest technical problem came from bin bags.
As 2021 began, the studio's technical experts split into what Tremblay describes as "strike teams," to tackle the most essential bugs. They used a "simple Kanban approach" to prioritise what they tackled first (Kanban is a system for managing workflows, typically built around a visual board where items are moved between columns to indicate their stage in the process). The solutions to some of the biggest crashes "were not easy", he says, due to the nature of the engine the studio had built as it developed the game.
"We went from a quite traditional, 'single-threaded' engine," where tasks are executed one at a time, he says, "to 'fully parallelised,'" where multiple ones can be executed simultaneously by different cores of your machine. The upside is, as you'd expect from CD Projekt Red, a much higher ceiling to what the game can do. The downside is it's much more complicated to trace a solution when something goes wrong. "When you go fully parallelised, you have a lot of random crashes that are super hard to figure out, because you have everything together, working at the same time, interacting with each other," Tremblay explains (in his best attempt at terms I might understand).
The studio at least had telemetry data now, from the live version of the game and via Nieweglowska's team, that showed what players were up to. "And we started, slowly but surely, to change the way of working together," Tremblay explains.
The strike teams effectively tested out another broader solution to the studio's issues. Instead of the old, closed-off silos, interdisciplinary sub-teams across UI and UX, bug fixes, and performance would all pull towards the same overarching goal, such as the next-gen port. It worked well. Eventually, Tremblay and the team found a solution to the streaming issues.
"We have to face it and learn from it. If we didn't, I don't think there would be a place for us as a company in the future."After a lot of work optimising as best it could, the team had to start getting creative. "I did some tests," Tremblay tells me, whereby he’d surreptitiously dial down the quality in minor spots and see if anyone noticed. "I just subtly did some, you know, reduce the quality, and then just submit, and wait until someone panics," he said. Then, "'Okay, no one's panicking, therefore no one's actually realising.'" He'd run certain quality checks past the team - certain stickers, for instance, could be rendered in very low detail while others, such as those on an in-game bar counter right near the start, had to be "more crisp, because it endangered the experience of the gamers."
Enter the bin bags. "The trash bags you see on the side [of the road], you don't really focus on trash bags all the time," Tremblay says. "Like, no one actually stops and says, 'Hmm, this trash bag looks a bit trashy.' Well, we reduced the quality a little bit, and I think it was acceptable." The only problem was how long it took, with this process of tweaking, testing, and moving forward under the rigid limits of mechanical drive speeds ultimately lasting months.
Meanwhile, the company began the process of figuring out how to work together properly. Sarah Grummer mentions "coaches", who came in to look at working practices. Eventually, the studio settled on the 'agile' system, popular in the tech and programming world and indeed at many modern developers now. Football Manager developer Sports Interactive has switched to it in recent years, while Nintendo's internal studios on both the main Zelda and Mario series described something very similar in principle for their approaches to The Legend of Zelda: Tears of the Kingdom and Super Mario Bros. Wonder.
In essence, it involves splitting developers up into 'pods', where each contains all the essential disciplines to work on an entire section or project. "Let's say when you have a review with a quest director or game director," Sasko explains, "we sit together, and it's not a quest designer who's showing that piece of a quest, but it's actually a whole team. There's an environment artist, concept artist, VFX, cinematic animator, QA. There's like 22 disciplines, I think that we had at the end in that one pod."
Coupled with a range of small tweaks - reworked Slack channel settings to be more open, new reporting structures and email chains - it made for a slow, but significant reorganisation of how thousands of developers worked together.
Contentiously to some at the studio, CD Projekt Red also finally abandoned its tradition of building bespoke engines from scratch for each new game. Part of the reason for sticking with its own proprietary tech for so long, Nowakowski tells me, is that the studio felt "we had no other choice in the market; there were no engines available that were allowing us to realise what we wanted to, at the scale that we wanted to."
Now, the studio has formed a two-way partnership with Epic Games to work with Unreal Engine 5, providing a few of its own insights on the engine's development while also, crucially, now having the leeway to develop multiple games at once. This went hand-in-hand with what Nowakowski defined as "probably the most fundamental change [in] always working on the target platforms" rather than any changes in plans becoming a "flip the table moment."
In other words: the studio now develops on and reviews every platform's version of its games at once. Console versions "working super late" are now "unacceptable," Tremblay adds. "It's part of our process; we do the reviews on console so we know exactly where we are on all the platforms, [including] the lowest [spec] platform we have, rather than saying, 'PC's fine, we can go forward'." The process is applied across all of the studio's upcoming games. If one platform isn't working, the team stops until it gets it right.
Lastly, there was one big decision left to make: the dropping of the previous generation of consoles. The thinking was that with patch 1.6 in particular, "the things that we were adding on the technology level were simply no longer feasible for the previous gens," Nowakowski says. "We just realised, okay, we were giving people a free upgrade from the PS4 to the PS5, and we decided to hope for more people to jump at a certain point."
"We simply couldn't deliver the vision we wanted to on the PS4," he continues. "That's the reality of it. We had to either focus, or try and potentially fail with the past gens as well. And we didn't want to repeat that. I'm pretty sure it's a decision that hurt some people hoping to play [Phantom Liberty and 2.0] on past gen consoles, but it was the right decision, eventually."
With that final admission, it seemed like CD Projekt Red had finally come to understand its limits. In doing so, it might have saved itself. "We have to face it and learn from it," Nowakowski says. "If we didn't, I don't think there would be a place for us as a company in the future."
All of this sounds simple on paper. In some senses it is. "It sounds a bit stupid," Sarah Grummer admits to me. "It's like, 'Why didn't you do this before?!' If you look at this, it's such a natural thing to do! Why did we not see this before?"
The actual act of switching over took time. Change only began in earnest for the development of Phantom Liberty, with a reduced team size of around 350 people - a more focused scale of development used for The Witcher 3's expansions. And yet the difference in results couldn't have been more clear.
For a while, sentiment remained steadfastly the same. As part of the response to the nightmare of the main launch, the comms teams had turned down their "edgy" tone, and instead repositioned to be more direct and humble. Every trailer or fancy asset came paired with a "more down-to-earth" livestream featuring real developers talking through the changes and answering questions. When the team trialled another in-character series, this time in blogs in the voice of a Night City news anchor, people "hated it", Momot says. The studio had to wait until it had something to back up the noise. They had to "hit the gym," as he puts it, "and come back stronger, when you have something to show for it."
"That was the breaking point, where everything moved from: it's really, really dark and gloomy, to: okay, there's hope for us."The show-don't-tell approach was no doubt the right one, but it required time. Sentiment only first began to turn around the time of the game's next gen patch, labelled 1.5, which came almost two full years after the original launch, in 2022. Positivity was boosted again by patch 1.6 and the release of Netflix's tie-in Edgerunners anime. "That was a huge turnaround for us," Momot says. "That was the breaking point, where everything kind of moved from: it's really, really dark and gloomy, to: okay, there's hope for us. People are thinking the game is good, they're seeing our commitment and things that are improving."
Another year went by before Cyberpunk 2077 version 2.0 and its Phantom Liberty expansion debuted, simultaneously, in September 2023. By all accounts the expansion was fantastic, a twisting, neo-noir spy thriller dappled with surprise bursts of wider genres - such as one particularly brilliant turn towards horror - and another star performance, this time from Idris Elba. All together, with the updates of 2.0 including an entirely reworked police system, new skill tree, and fleshed-out cyberware mechanics, the difference was transformative. It bore out in the expansion's popularity, with the same attachment rate - around 22 to 24 percent - for Phantom Liberty's release, almost three full years after the base game's launch, as The Witcher 3's expansions had when they arrived after only three months.
The difference was just as profound for CD Projekt Red's developers. Tremblay describes a situation where the studio now has "way more open discussion about tech across all the groups." Previously, he says, "I think tech was quite a 'mystic' thing where, you know, 'they'll figure it out; we can do this and tech will figure it out later'" was the norm. "Now we have much more collaboration across different areas."
While it may not have been popular - and base console versions of the games still come with a warning on digital stores today - working purely with next-gen consoles for Phantom Liberty was a major boon on the technical side. "All the problems we had were gone," Tremblay says. Getting it running smoothly "took us something like six months, which is crazy, crazy fast."
The team was so taken aback by the difference, in fact, that it realised it had actually been if anything under ambitious with next-gen performance. "When we got the game, it ran at 50fps on PS5," he says, at which point the team joked, "all right, our job is done!" But after releasing the Series X and S versions, the S only ran at 30fps. "I don't know why, in our mind we were like, 'That's what people will be happy with,' the best quality we can get on Series S at 30fps.
"But people were like, 'Why? Why is it not 60fps?! And we looked at each other and said, 'Yeah why?!' So we just went okay, let's try." Soon enough it worked well enough to release a patch with the higher frame rate option to the public.
"Compared to Cyberpunk, where at launch you had this anxiety, and then it launched, and it was like, 'Shit', you're kind of heartbroken", says Grummer, "now, Phantom Liberty hits - and you have this anxiety but it's like, 'We actually did it."
It's "a very different sense of pride that you have for this," she says. "I don't know how to express it, but it's amazing. Both are amazing, it's just very different, feeling-wise." When it's put to her that it might feel more earned, she agrees. "You got kicked, and you were on the ground, and then you got yourself up and you're like 'Yes! Now I did it!'. That's incredibly gratifying."
Image credit: Eurogamer.
Every developer I spoke to at CD Projekt Red was unanimously happier with the new way of working. But the hope, too, is that it also leads to a more humane development cycle for the studio's big future projects than it was for Cyberpunk 2077.
"There was, I think, if you looked at the data, probably singular spots in the company where there was something you could probably pull under the definition of crunch," Nowakowski says, when I asked about Phantom Liberty's development. "But it was definitely not mass scale," he emphasises. "And I don't think it was a reason for any major complaints within the company either. So there was some overtime for sure happening for Phantom Liberty," he continues, "but the scale of it was miniscule compared to what it was with Cyberpunk or The Witcher 3. I would not, however, say it was zero."
"There's nothing we can say to convince anyone. There's just a moment when we need to show up."As for the company's future, keeping crunch to a minimum remains "the hope," he adds, for all of the upcoming games from the studio. "I hope we're going to be in a position where after the launch of the next big game, I'm going to be able to give you a similar answer as I just did. Realistically, I don't think I'm going to be able to give an answer that nobody took any overtime while developing, especially at the end of that game. But I would hope that it would be singular spots in the team, and singular spots that did not lead to somebody breaking their back.
"I think minimising that dark part of the development of the games is a key responsibility for us."
The developers themselves, for their part, remain steadfastly optimistic, even in the face of what will likely remain a continued attitude of scepticism towards the studio's future releases.
In the immediate aftermath of Cyberpunk 2077's disastrous launch, while the scale of the issue was still unfolding, there were a series of small crisis meetings held "at all levels of the company," Nowakowski explains. "At the level of the board, it was: a) the realisation of what happened, b) we self-acknowledged that what happened was a big, big challenge and a big hit for the company.
"None of us were actually even thinking about it in any financial terms," he continues, "but it's a problem for the future that we were hoping to build for this company, for its perception amongst fans. And that's something that, to be honest, we've probably lost forever. And yes, you can repair some things, but it's a certain perception of the company that's never going to come back. Is that good or bad? I don't know, but it's a fact."
Just as the studio had eroded the trust of a small, but no less vital part of its community in its ill-judged moments ahead of launch, it had undeniably lost that trust on a much wider scale with its launch. With a quieter determination, a re-directed concentration of its unwavering faith and forever ironclad self-belief, the developers of CD Projekt Red slowly turned it back around.
Marcin Momot, bottom left, surrounded by fellow CD Projekt Red colleagues as he prepares to send the "beep" tweet, in 2018. | Image credit: Marcin Momot / Eurogamer
Some, naturally, will probably never be fully convinced, as Nowakowski says. But the unanimous response to that suggestion amongst the developers of CD Projekt Red who I spoke to was one of polite defiance first; that these people can still be won around, if not with Phantom Liberty's success, and the years of quiet dedication to restoring the game to what it always could and should have been, then in showing it in the next game, or the one after that.
"I accept and understand that," says Sasko, "it might be a situation that never happens for some of them. But that's unfortunately the price we have to pay for what happened. But I hope that, throughout the work that we are doing, throughout the things we are showing, we can actually win some of those people around - and when they hear someone talking to them about the incredible experiences they can have in, let's say Phantom Liberty, or the next Witcher, next Cyberpunk, or next Hadar [CD Projekt's new IP], at the moment when they see there's that incredible value in that game, they will actually reach out for it, play it and enjoy it."
There was also, notably, a sense of typically stoic acceptance across everyone I spoke with: that the past simply can't be changed."There's nothing we can say to convince anyone," he adds. "There's just a moment when we need to show up."
Karolina Nieweglowska, meanwhile, has a typically logical suggestion. "I would say we worked on ourselves and we really looked at ourselves, and what we can improve or where we needed some changes," she says. "But in the end, people have to judge for themselves."
Her suggestion is simple enough. "Of course, you know, you can just check when the game is released. There will be reviews. I'm sure people like you will help with that. If someone is unsure, then they can wait and see. Maybe then they can decide to jump in."
![Anime Reborn Units Tier List [RELEASE] (November 2024)](https://www.destructoid.com/wp-content/uploads/2024/11/anime-reborn-units-tier-list.jpg)
