Network Operations  Brandon Williams Comments on the Evolution of UO Jan 28 2004 4:46PM   My name is Brandon Williams. I am a Sr. Unix Administrator for Electronic Arts/OSI. I am considered somewhat of an old timer around here in the OSI office, as I have been directly involved with building and supporting Ultima Online production and OSI development environments for the majority of the past seven years. I have seen UO go from a "proof of concept game," that was hopefully capable of lasting a few months, to a staggering alternate universe successfully capturing the interest of millions of avid players for over six years now. The evolution of UO is truly a remarkable story. If you have a moment, and an inkling of interest, please feel free to read on as this relatively dry Sys Admin tries to spin a tale. 1996 was a time of change, both for me and for a relatively small project in development at OSI code named "Multima." It was around this time when public beta for the project was getting ready to start. I was not yet a member of Operations, partially because there was no Operations team in place at that time. I was working part-time at OSI and majoring in computer science at school. As an avid gamer and office peon desperately looking for ways to get closer to game development, I volunteered to manage the influx of beta tester applications for Ultima Online. Little did I know, this would mean I would be opening thirty thousand letters, removing the two dollar shipping fee that applicants were sending to cover the cost of shipping their beta CD, and manually entering return shipping addresses into a simple flat database! The public's interest in UO's open beta program was overwhelming at times, but also extremely exciting. Around this time, a member of OSI's local IT staff, Mark Rizzo, broke out of his everyday duties to focus on the infrastructure that was to support UO. The initial plan was to run the entire service on a cluster of "powerful" dual processor Pentium Pro 200Mhz servers with 128MB of RAM--all from the "high tech" converted office/server room in our development studio! (Please excuse my facetiousness. It's just that looking back now; it's funny to think what we initially thought it would take to run the UO service.) With public beta ramping up, and a vast amount of work needing to be accomplished, Rizzo was able to justify the need for a junior-level sys admin position to help with the launch and support of UO. In hind site, I realize I am very fortunate to have jumped into this role full-time when I did. Rizzo was an exceptional mentor. I learned many aspects of sys admin work by working with him. Together we were discovering the trials and tribulations of launching and running an online service. Things were very different at that time. There was not an abundance of hardware venders in the market providing a state of the art X86 server platform. Instead, we were faced with building customized full tower case servers from scratch. ISPs were in an infant state. What is typically expected in a co-location center now was in the process of being defined when UO was getting ready to launch. Luckily for UO, big brass from the EA ranks were aware of the mammoth interest that was building for UO and had the foresight to open wide the checkbook and support the project as needed for an appropriate launch. In the very late stages of development, it became painfully apparent that the X86 platform was not going to efficiently support the game, and that we were not going to be able to support all of UO's customers in one single instance of the online world. This is when the idea of "a shard" was created. Fiction was put together to support this need, and hence, we now know of "Mondain the wizard" and the gem that shattered creating hundreds of like instances of the Britannia universe. Within the first few frenzied months of UO's launch, the Pacific, Baja, Napa Valley, and Sonoma shards were opened on the US West Coast; Great Lakes and Lake Superior were centrally located in Chicago; and the Chesapeake, Atlantic, and Catskills shards were located on the East Coast. All of these shards were running on more powerful (yet more expensive) Sun Sparc servers. With UO growing by leaps and bounds, EA and the UO support staff had to adapt quickly. It was not long before the Operations team began to hire on some of the industries best talent. Pretty soon, our operations team had software engineers specializing in login and billing systems, network architects specializing in Internet routing, web administrators mastering one of the most impressive dynamic web sites around, and, of course, more Unix administrators to support the growing platform needed by Ultima Online. Soon, EA began to tap foreign markets as well. Interest in UO from Japan, Europe, South Korea, Taiwan, and Australia was and still is staggering. To this day, our non-USA customer base makes up about half of our active players. Throughout the life of the service, Operations has added many shards, in several regions around the world. UO has benefited from several milestones and improvements in its life. Many champions have done great things for this game, whether it was a programmer developing distributed backups, significantly reducing backup times, or an architect sponsoring major hardware upgrades at various stages of this game's life. It is truly amazing how many developers and support staff UO has had over the years making the service what it is today. Looking forward, Operations knows there are many improvements to be made still yet on the infrastructure that supports UO. The day when shard crashes are a thing of the past is a goal of ours. Reworking the in game backup system to make reverts significantly rarer--and no longer than a few seconds when there's no way to stop them from occurring--is on the radar. Improving the server platform and our Internet connectivity to make latency a forgotten word is a utopia that our Operations team is working hard to achieve. Operations can not achieve all of these goals alone. Many other teams within the UO support structure and the Internet as a whole have a large say in these goals as well. They are common goals though, and they will be achieved. So that is my tale. (At least that is all I have time to tell right now.) Thank you for your interest and, more importantly, thank you so much for your continued dedication to the amazing world of Ultima Online! EA, OSI, and everyone here who directly supports UO know we're lucky to have such a dedicated and truly creative player base, and that you are a major factor in the continued success of this online universe. For that, my colleagues and I are extremely grateful to you. Keep having fun, and keep driving your high expectations. Hopefully together we can one day enjoy UO's 20th anniversary and beyond. Brandon Williams Sr. Unix Administrator Origin Systems  A Moment with Tom the Grey Apr 16 2003 2:15PM   Well, it is my turn to submit comments from my team, so hopefully I can entertain you with who I am and what I do. My name is Tom Carlile and I am the team lead for the Operations group here at OSI. Our group is responsible for administering all the Ultima Online shards and their backend infrastructure. Everything that is not specifically a development or internal database issue is handled by our group. I think a lot of people underestimate the complexity of the UO backend. Not only do we spend a lot of time preparing and protecting ourselves from hardware failures, but we're also the ones working through the night to get UO shards and subsystems back online when one of those outages does happen. It's rough hours sometimes, but bringing a crashed shard back up and watching its player count rise up to nearly 2000 in minutes can be a rewarding thing at 3 AM. We get our share of insults on the boards, but we still care about the players and sacrifice a lot in the name of keeping these imaginary worlds alive and well. When I am not off saving worlds from destruction and time-warps, I am usually spending time toward one of my other interests, such as sound synthesis. I've been teaching myself the fundamentals of substractive synthesis and music production for a couple years now. What began as a drum machine and keyboard synth sharing a MIDI clock has turned into a studio full of gear all interconnected and sequenced via computer. Recently I acquired some studio space at OSI, and have been primarily working on sound effects and environmental sounds for Ultima. It really can be a lot of fun trying to find or synthesize the right sound effect for some random game monster. My favorite music influences right now are Juno Reactor and Thievery Corporation. Working at OSI has given me a great opportunity to go beyond my forte of UNIX systems and network administration. Lately, I've been able to spend some work-time on designing sound and environmental effects. I’ve also been able to follow up on my interest in Japanese culture; since our Japanese market is so large, I have been able to travel to Japan many times in order to maintain and upgrade our systems there. There are not many UNIX sysadmins out there that get to do the things I do at work, and for that I consider myself very lucky. Back to the machines for me, and I hope this helped acquaint you with one of the many nameless and dedicated people behind Ultima Online, keeping the fantasy alive and well for over 5 years now. Tom "Tom the Grey" Carlile Sr. Unix Admin, Origin Systems  NetOps has a Deathwish - and he speaks! Apr 3 2002 2:27PM   This update comes to you from Sean "Deathwish" McMains. I started at Origin about 8 months ago as a web application developer after having become dissatisfied with life as a dot-com drone, ironically at another company in the Origin building. After moving downstairs to Origin proper, I was quickly taught to swim by being tossed in the deep water of maintaining our customer management and billing system. In spite of having been a long-time fan of OSI, I'd never actually played UO before coming to work here. Now that I've spent a good deal of time in the game and in the company, I'm convinced that for someone looking for deep gameplay in a Persistent State World there's nothing to compete with UO. (And with UO:3D, we're getting a bit of the eye candy too.) Since we've narrowed our focus solely to UO in the last couple months, we've been able to put even more effort into improving the game and its supporting systems. One recent improvement I'd like to bring to your attention involves the communication between the systems where you pay and the systems where you play. As you know, to play Ultima Online, you have to pay a certain amount monthly. The server that's responsible for maintaining that billing information is completely separate from the server that decides whether or not you're able to log into the game. So, years ago, a system to transfer information from one server to the other was built. Though this system worked fine under normal loads, it would on occasion get a bit behind when there was an exceptional amount of data to move. One instance of this occurred only last month, when we opened a new Japanese shard. The enthusiasm of the Japanese players was such that the shard filled to capacity within minutes of its opening, and stayed maxed out for days. We were thrilled to get such an enthusiastic response from our loyal fans, but unfortunately the database that serves the login servers began to show signs of strain. Though players were still able to log in to the game, our data transit system wasn't able to get its job done as efficiently as we hoped. Our team immediately came together and started moving the data manually to ensure that players who had paid for the UO service would be able to play. This, however, was obviously a short term solution. After analyzing the situation, several of our database experts -- Mike "AviStetto" Howard and Doug Mellencamp most notably -- decided on a design for a new data transfer system that should be more efficient and robust than the one we had been using. After presenting their plans to our supervisors, they decided to go ahead and build the new system, estimating that it would take two days of a couple of our guys' time. They set feverishly to work, and were ready to flip the switch right on schedule. We were all excited to see the results of the work, since the new system was simpler and would theoretically be much easier to maintain than the old one was. They made the switch, and were delighted to see the old system shuttling the data around as much as 30x as fast as was previously possible. Best of all, the very farthest behind it's ever gotten now in the couple of weeks that it's been operating is only about 8 minutes, and it averages much closer to 2 minutes -- far better than we'd ever seen with the old system. This is just one example of the kind of improvements that are being made to UO daily. They're not all big, glamorous things that you notice right away when you log in. But we hope that, as we continue to improve UO's foundations, that you'll eventually notice even fewer interruptions, and will just be able to get down to the serious business of GMing that character you've been working on. Sean "Deathwish" McCains Network Operations Origin Systems