Server Patch/Downtime

[dazman76]

*chuckle* Totally off-topic, but I like your sig Jackoba Indeed, there is no explanation for Blackpool

[yoram]

Apparently people are unaware that when you do some software patches on Linux, sometimes you have to rebuild the KERNEL, now this is a linux server FARM, so if you touch one kernel, you touch them all, and if they can do that in 4 hours, Lord help them, do NOT EXPECT the servers to be up in the 8 hour estimated timeframe.

Just asking for some more information is **** ing ?

Thanks for your information. But when you look at your answer I'd say the more reason to make a split between the patching of the system software and the game software.
Now when something goes wrong, it just looks more difficult to me to find the problem. Start with Linux let it rest a day or two and then do the patch for the game.

[zumwalt]

I don't want to flame here. But you just don't say we are down for a complete 'working day'.

You weren't just asking, you were wining, there is a difference, I can understand your frustration of not being allowed to play, but there are administrative facts your not considering.

The way there doing it is based on a judgement call they have made, also based probably primarily on what the patch is going to effect.

Let me break this down in lamens terms.

The code patch could effect things like packet synch, (reduce lag), coordinate map (ping pong characters, you see a mektoub running around but no rider, your party members are standing in one spot but in fact there attacking something a mile away), a few other 'technical' issues in core to characters.

The server patch would be done to prepare the hardware / firmware for the changes, including network card drivers, need to add some to the kernel because the cards were updated and the current kernel build doesn't support the drivers, this goes hand in hand with the packet synch utilizing the network cards drivers.

Maybe there adding more ram to increase the ability of the servers, possibly there adding more hard drive space so they have to update the partitions etc.

Its not cut and dry, it never is, and asking them to explain an 8 hour outage, when there are not 1 but 4 server farms, is asking a bit much.

They have alot of core issues to address, and these issues increased on them when there isp upgraded the routers. This gave them a thorn they were not ready for.

[xenofur]

what i find quite interesting is the fact that in general there are more american whine-threads about downtime as there are european, and we don't have the luck of a timezone-shift of 6 hours, when people here come back home at 18:00 with a patch scheduled to end at 16:00 they can usually expect it to take at least until 19:00
and that's just those that do work that late <_<

[yoram]

you were wining

Scouts honor. I really did not drink any wine.

[zumwalt]

****LOL****

[zukor]

****LOL****

Well done, zumwalt, just enough asterisks to get past the 10 character post minimum.

LOL.

Doctor Z.

[sankari]

Quit your *****ing, seriously, I say take the servers down for a week if it means when you turn them on they are bug free.

Apparently people are unaware that when you do some software patches on Linux, sometimes you have to rebuild the KERNEL, now this is a linux server FARM, so if you touch one kernel, you touch them all, and if they can do that in 4 hours, Lord help them, do NOT EXPECT the servers to be up in the 8 hour estimated timeframe.

Deal with the patch time, let them roll out bug fixes, and look forward to a smoother game, you can go to the mall, watch a movie, use real life time away from the game, you don't have to live in it.

Even if the servers were down for a week, would you honestly expect them to come back completely bug-free?

And is it wise to patch the code of the game, and at the same time do an upgrade to their systems? I certainly wouldn't do it in my job - but that's just me. /shrug

[zumwalt]

Thanks Doc

As far is it smart to do both, well, sometimes in the world of coding for a server side application, its not a matter of 'smart' its a matter of necessity.

Say you have coded some packet buffers that catch all inbound traffic to then distribute to the propper functions, however, your NIC is a POS and keeps dropping packets, so your CRC is blown.

This means your code could be flawless now, but the NIC is killing performance.

You put in a request to replace the NIC, when the new NIC comes in, you get the hardware drivers for it, and then find out that the packet buffer software you wrote doesn't support some of the new NICs result sets and passed data, so you have to rewrite the code.

You need to keep this as consistant as possible across the farm of servers since they work together as a team to keep the play world up and running.

So now your code is working with the new NIC, thus you have to replace the NIC on every server in the Farm, deploy the code, oh yea don't forget they actually have 5 seperate server farms.

Looking at this logically, (all of this is just an example btw), lets say that they have only 10 servers per farm, the master browser server is 1 of the 10, leaving 9 in the set for balance, but since there are 5 sets, thats 50 machines across atleast 2 countries, if not 3.

Lets just imagine that they use 1 person per 3 machines to replace the nics at about 15 minutes a nic with shut down and start up, another 5 minutes per machine for the new nic driver, thats 20 minutes per NIC (a long time), and lets just assume the software patch was made by a sheer genius, 1 minute for installing the patch, thats 21 minutes per machine used of man hours.

3 machines per hour approximately done per person, if there good and someone doesn't spill there cofee.

Financially thats 9 man hours, maybe 10, for 1 server farm, split between 4 people with the addition of the developer installing his/her patch.

Overall downtime for this scenario for 1 farm is approximately 1 hour, repeat this process across 5 server farms, makes 5 hours, or 45 man hours combined.

With this said, they are counting on there work to only cost them 1hour 30 minutes at most per farm to be done in 6 hours with time to spare.

This is just a generalization of a roleout to help people understand why things take time.

I have delt with enough server farms in my life to know, it never goes smooth the first time.