PDA

View Full Version : Tech Support Team?



-FN-
27-02-06, 18:20
Dear KK,

I know there's helpdesk@neocron.com - but to my knowledge that's just an account support email.

It's time to step up your tech support. I work for a relatively small (200-300 Employees, 1 Main Office, 3 Satellite) company. We have 3 IT guys (admittedly we should have more, and did, but that's another story). Internally we all have our own roles, but we are all jointly responsible for anything that is customer-base related. For example, if any of our servers start acting up, we're paged and having downtime is unheard of here (or we'd be fired :p). If there's a problem with a job that's being worked on, one of us who is designated 'on-call' is paged and we take care of the problem asap.

Here's what I don't get - how can there be an obvious database problem for 10 hours on Terra - but nobody at KK is able to do anything until someone "gets in the office"??? Buy a fucking office pager. Set up a KK Staff & Volunteer only email address that pages it so that GMs and KK Officials can notify someone who CAN fix it that there's a problem and get it fixed before half a day goes by. I personally don't give a shit that someone might be sleeping, or they're not 'on the clock'... but maybe that's just a difference in work ethic. I take a lot of pride in our small-town web services and how reliable and well-maintained they are. If one of our web-kiosks, or god forbid our dynamic subscription based website with 16,000 subscribers, went down, I'd see it as my ass, my reputation, my responsibility to fix it. It just has felt like (for years) that whoever runs the technical-side of things within Reakktor, doesn't want to do anything unless they're in their 9-5 work window and they have to. I'm not saying it *is* like that because I don't know how KK's Internal IT staff is managed, but this is not the first time a technical problem has gone untouched for an insane amount of time and I'm sure it won't be the last.

Personally, I don't even care about the rollback. I got one DEX level on a PE last night then played Planetside. But several SXR members did a 5 hour stint Rhino-ing, two people completely lommed their characters from one combat type to another, and one returned vet leveled completely new characters for several hours (since KK was nice enough to delete his account since it expired on Sep 03, 2005)... and this was all done during our 'prime' time - from 11pm - 4am EST. All of that progress is gone and a number of SXR members are pretty much throwing in the towel. I find that depressing and yes, it pisses me off. This all could've been avoided had someone at KK tended to this 10 hours before someone decided to roll into the office at 9am.

When you provide a service that customers pay for, and I'm not talking about development, or content, or anything like that, I'm talking about pure availability of reliable service, you should be a little more on the ball. If it's a physical hardware problem and you can't get into your datacenter until they open (which is a BS excuse anyway since real datacenters provide secure RSA access for their customers 24/7) - so be it. But when it's a database issue or something requiring a reboot that can be done remotely - it makes my stomach turn as an employee of the IT industry because this lack of response is an embarrasment to our field. I mean... HELLO, WE WORK WITH COMPUTERS - they can do everything FOR us. Monitoring, Maintenance, Notification - with today's technology, there is NO reason why it should take 10 hours for someone at KK to realize something needs to be looked at.

I have fewer and fewer reasons to login as the weeks go by, and now I've just lost about half a dozen more.

Deeply Disgusted,
-FN-

cRazy-
27-02-06, 18:34
My company is the same, we have a few Crontabs on our main servers which run overnight, myself and 3 Technicians are on 24/7 standby, I get all of the serverlogs E-mailed to my Mobile Phone and Computer every hour so if anything happens I can be there within minutes.

Reason being is that i'd rather drag myself out of bed and solve a problem if one of the Servers produced an error at sometime during the early morning then come back into the office the next day and realise for the last 8 or so hours our tests have bearly even started before they produced and error and locked up, which has happened twice so far (thankfully one of us fixed it within the same hour).

RogerRamjet
27-02-06, 18:35
helpdesk is the accounts thing?

Fuck, ive been emailing accounts for like a week.

Tratos
27-02-06, 18:37
So.... tech@neocron.com / tech@neocron.de then? ;)

-FN-
27-02-06, 18:40
helpdesk is the accounts thing?

Fuck, ive been emailing accounts for like a week.
My bad - helpdesk@ is for In-Game Related probems. You are correct - account@ is for Account Related Problems. And tech@ already exists for Technical Problems - but that's a public email. No support staff wants an email address that turns into an auto-notifcation page/text message that ANYONE can email :p

http://ng.neocron.com/index.php?id=55

xyl_az
27-02-06, 18:41
well they got probably 10-15 people as whole staff, half of them are probably working part time (like once in a week/month), another half is probably working 2-4 hours daily (or just at home). I'm guessing the only people who are working full job there are management team and they probably have no clue what a computer and how to use it.

dont expect too much

Brammers
27-02-06, 19:01
What I don't understand is why KK doesn't shut down a server when there is a problem sooner. On the 6th of December, everyones apartments decided to vanish at about 23:00GMT, and it wasn't until 11.00GMT that the server was shut down, and data restored.

Yet the server stayed up for at least 10 hours? What's the point? Even if Thanatos or SnowCrash. (Or whoever comes and fixes the data) there is no point running keeping a broken server running.

Put it another way - Broken server = high risk of rollback.

Players don't understand what broken apartments or messages like "CHARACTER DATA WAS DAMAGED TRYING TO RETRIEVE LAST AUTOMATIC BACKUP. PLEASE RELOG" means. Not everyone has OOC turned on to hear players complaining about error messages. Even last night I didn't spot the significance of that error message, and like everyone else, I carried on as normal.

No-one last night made the call to shut the server down. It doesn't look like they did in this case (http://forum.neocron.com/showthread.php?t=130389) or this case (http://forum.neocron.com/showthread.php?t=131581) either. Why not?

Also this is the 3rd rollback on Terra since December 2005. Thats 1 a month. There must be a reason for this, and this problem must have been investigated.

So where is the fix for the problem?

These are simple but hard questions, that have to be answered. If I was in your office I would expect these questions to be answered.

And also the community expects answers.

Immersion
27-02-06, 19:50
I'm confused as to why the rollback went so far back, and actually, why it happened at all. Did something happen in game? Because as far as I could tell it was just a login issue.
Was it just our vent, or did the whole issue only start at around 11 pm est? I could have lost track of time.. don't get me wrong, it happens. Why did we need to rollback the entire day anyway?
It makes for one sad sad gamer... (I was one of rhinoers spoken about by FN.. which wasn't 5 hours by the way.. about 1:45 pm to oh, maybe midnight. There was an hour break there... but pretty much constant. It was fun, nice bonding experience I guess)

But hey... don't worry about me. I've always got EQ to fall back on. I think I hear my elf calling for me... she's been awfully lonely without me :(

-FN-
27-02-06, 20:26
And also the community expects answers.
I don't expect anything. Not a response at least. I can only hope that this plants an idea within KK that they need to take more active response in issues like the ones you cited. And I feel bad for KK if they think 24/7 on-call employees aren't necessary - because that's just a common-sense business practice when you do exactly what they do, provide a persistent online service.

Zheo
27-02-06, 20:48
I work in a supermarket i admit thats not the most glamerous job however customer service is at the top of the list why? Ok lets look at the circle of corporate life:

Staff Provide 1st Class Service - Gains Customers - Customers Provide Money - Staff Want Money - Staff Provide 1st Class Service -

See the circle there? IF Staff feck up, customers leave and staff get no money this is bad for all...

So really KK have no real choice but to provide a good service otherwise people leave and they'll shut down.

So lets now discuss ideas for KK I have a few:

Better hardware/programming so theirs no rollbacks?

Fast Responce Team so they roll back only an hour or two at most.

More regular saving so they roll back at the exact point thus meaning less time lost gaming EG if ther server messes up at say 6pm they dont have to roll back to 2am the previous day.

anyone else?

Spermy
27-02-06, 20:52
3 Rollbacks in 3 months. A peaking population of 200 people. 200. Bugs that have existed for as long as the game has. Every patch breaks something else.
Memory leaks. Issues with EVERY payment vendor save PayByCash. Planfiles written and promises made, and then just thrown away, or put on the backburner so damn long the flame has gone out. The list is pretty much endless should you choose to go into detail.

People leave. The men in charge say, "you'll be back"

Some do. Some don't. More people have left in the past 3 months than have joined. If this continues, it's a slow painful spiral, towards the time when there are less than a handful of us left.

Neocron is in a coma. At this rate it will die, unless something changes seriously. The 2.1 update was supposed to be a revitalisation. It didn't do anything. It's frustrating.

For the comment that always seems to pop up.


We're still here.

You are barely so.

Do something.

Drastic.

james_finn
27-02-06, 22:04
TBH its this sort of rubbish that stops me logging in. I have active accounts and yet I havent logged in since a clan mate was doing a role play event. I dont want to loose time over rollbacks, and I certainly dont want to continue playing a game with as many bugs as beta (and yes I was in nc1 beta).

As for the fact the server stays up! Im in charge of customer service in a small shop, as the weekend supervisor Im the one whose gets a rollicking if customer service isnt up to scratch. There seems to be no ramifications for the fact that the server was running brokenly and wasnt fixed for 10 hours.

I'll echo -FN-'s statement


Buy a fucking office pager

Delphi

CMaster
27-02-06, 22:11
I've said this before (although not on here I don't think). The sucessful running of the servers is absolutley critical to the sucess of Neocron and Reakktor as a company. So why is there nobody (seemingly) contactable out of office hours to do anything about it? I know I'd hate for it to be my jonb, but there really should be somebody who is required to drop pretty much anything and rush to fix the servers.

Absolute lack of activity since christmas. Come on, keep patching little things up, put in an extra mini-run that took an hour worth of scripting. Don't make the players feel that the developers are letting the game rot away.
Emply some more english EGMs (after all, you don't pay them anyway, right?) Make it feel like Neocron is an exciting and happening place to be.

Make the game feel like its still alive damn it, rathen than slowly turning cold. Make the players believe that the devs care, because right now it appears like they barely give a damn.

Oh, and FN - have those people whos accounts have supposedly been deleted had discussions with account@neocron.com ? If not, I feel they should.

-FN-
27-02-06, 22:14
Oh, and FN - have those people whos accounts have supposedly been deleted had discussions with account@neocron.com ? If not, I feel they should.
I've asked them to - waiting to see how it pans out because now I'm curious.

elGringo
27-02-06, 22:19
...I certainly dont want to continue playing a game with as many bugs as beta (and yes I was in nc1 beta).

Sorry dude.. but thats just total BS.

Every now and again, I wish KK would set up a beta4 server, so all the people making false claims would be able to log and see wtf they're really refering to.

Nc1 Beta AND Nc1 early retail had far more severe bugs than nc2 has right now. Stateing anything else can only be due to selective perception or plain lies.

Of course, the very best time I, personally, had in NC was beta and early to mid retail... but thats in no way due to the games state in terms of bugs and severe problems.

Foo
27-02-06, 22:21
One of many reasons why i haven't logged on in nearly 2weeks . Glad i wasnt logged on when all this crap happend recently.

Oh and as for them saying " We will be patching loads inbetween the next Evo" :rolleyes: More BS? havent had a patch since xmas :(

-FN-
27-02-06, 22:51
Oh and as for them saying " We will be patching loads inbetween the next Evo" :rolleyes: More BS? havent had a patch since xmas :(
Again, to be fair, I don't ever recall them saying anything about multiple patches... but it is sad that a simple bugfix patch with a lot of the current issues (LE'd Runners and Soulclusters, Grim Chaser Loot, Vendor Purchase Spam, Item non-stacking, Missing Ceres Rifle Ammo Loot, Ceres Weapons doing TL80 damage, 1,000 English typos and errors in Missions, Ion Drones not doing any damage, etc) hasn't been produced. A single patch with just pure bug fixes would be nice.

cRazy-
27-02-06, 23:03
well they got probably 10-15 people as whole staff, half of them are probably working part time (like once in a week/month), another half is probably working 2-4 hours daily (or just at home). I'm guessing the only people who are working full job there are management team and they probably have no clue what a computer and how to use it.

dont expect too much

lol you cant sit there and say 'well they probably' etc when no one knows.

Ive seen pictures of the KK office, from what I saw there were many people sitting there infront of computers, one of them even had a edited model on the screen (presumably a modeller or artist). Oh and something I know from being in the industry, if you have a management team who dont know how to use a computer and your making a game, it will never see the light of day.

Im not having a go but, it just sort of amazed me how you were so specific about numbers and part time staff when none of us have a clue.

Foo
27-02-06, 23:04
They did , when they released the first info about evo2.2 and said it wont be released for 6 months or so . people asked " We have to wait 6months for a patch?" and they got replys " No we will be patching inbetween with fixes/content and what not". Nearly 3 months later and ...


/edit they said in the Planfile 5. Quoted by odin "It's a planfile folks. It's talking more about the next large revision. Of course there will be patches in the meantime."

Tratos
27-02-06, 23:07
Again, to be fair, I don't ever recall them saying anything about multiple patches... but it is sad that a simple bugfix patch with a lot of the current issues (LE'd Runners and Soulclusters, Grim Chaser Loot, Vendor Purchase Spam, Item non-stacking, Missing Ceres Rifle Ammo Loot, Ceres Weapons doing TL80 damage, 1,000 English typos and errors in Missions, Ion Drones not doing any damage, etc) hasn't been produced. A single patch with just pure bug fixes would be nice.
A patch is in the works it seems, how substantial is unknown though :(

Something does need to be done though things do look bleak.

xyl_az
27-02-06, 23:12
lol you cant sit there and say 'well they probably' etc when no one knows.

Ive seen pictures of the KK office, from what I saw there were many people sitting there infront of computers, one of them even had a edited model on the screen (presumably a modeller or artist). Oh and something I know from being in the industry, if you have a management team who dont know how to use a computer and your making a game, it will never see the light of day.

Im not having a go but, it just sort of amazed me how you were so specific about numbers and part time staff when none of us have a clue.

Something what i remember from beeing in the mmorpg/games industry (3.5 years) its quite normal situation. And if my assumptions are right (and they usually are) such scenario as i presented is quite *probable* (yeah, maybe a little exaggerated, but not so far from the truth)

Bishop Yutani
27-02-06, 23:47
Send in your resume.

[Edit]
Actually, i have more to say :) I wonder how many volunteers there are on the team. I can't remember where i read it, but it seems like a lot of the support staff are volunteers, so you have your reason why no one was responding late at night. Volunteers don't get beepers.

Also, i read a great post on RPG Codex a while ago and thought i'd pass it on. Its about the estimated costs of running an MMO. Very interesting.
http://rpgcodex.com/phpBB/viewtopic.php?t=10226

To my original reply: I thought and even better way than posting complaints to this forum to get shit changed was to send in my resume. I'm fully qualified, and -FN- and others, if you think you are too, i would suggest you follow suit.

james_finn
27-02-06, 23:59
Sorry dude.. but thats just total BS.

Every now and again, I wish KK would set up a beta4 server, so all the people making false claims would be able to log and see wtf they're really refering to.

Nc1 Beta AND Nc1 early retail had far more severe bugs than nc2 has right now. Stateing anything else can only be due to selective perception or plain lies.

Of course, the very best time I, personally, had in NC was beta and early to mid retail... but thats in no way due to the games state in terms of bugs and severe problems.

Im sorry I didnt mean the same number of bugs, I meant more that bugs were still there from beta, its not the quantity its the fact that some major bugs are still there after 3 years or so (cant remember the exact number lol).

Delphi

xyl_az
28-02-06, 00:00
Send in your resume.

No thx, dropped it 4 years ago and i have no intention on coming back. There are much better jobs out there, surprisingly enough

elGringo
28-02-06, 00:12
Oh and something I know from being in the industry, if you have a management team who dont know how to use a computer and your making a game, it will never see the light of day.

In that case you should be more than happy to note that KK's "new" CEO was a mapper and DEV for NC1. ;)



Im sorry I didnt mean the same number of bugs, I meant more that bugs were still there from beta,

Thats a different issue.. and true of course - nobody would honestly deny that ... but you've also got to see that some of those bugs are due to numerous different problems... e.g. its not one problem that causes a FRE, but a bunch of issues...
concerning this.. I have to repeat myself: I played nc2 closed beta many hours a day... every day... and I encountered 1 FRE during the entire closed beta ( that is "O N E" fatal error). Sadly the large open-beta patch ruined it again .. and right after the patch we were back to the known FRE issues.

..anyway - what I wanted to say is, NC is a lot more stable nowadays than it used to be .. it still carries arround a lot of old and new bugs .-.. nut it'll always be like that ... however, it runs BETTER than nc1 used to and A LOT better than all of the nc beta stages (cant tell for beta1 first hand... but b2 -> nc1 retail had more more problems and severe bugs than nc2 at the current patch level)

a4nic8er
28-02-06, 06:29
If it walks like a turky and gobbles like a turkey, we have to at least consider the possibility that we have a turkey on our hands (even if someone has stuck a bigger beak and some eagle feathers on it).


I wont be resubbing when my account expires (may try again after 2.2 comes out) - consider it a boycott of the current product.

IceStorm
28-02-06, 07:08
Neocron's servers aren't "business critical". It's an entertainment product. While you and I may be annoyed when the servers are down on our off time, in the grand scheme of things NC is just a game. There are many, many, many other games out there to play (like Silent Hill on ePSXe!).

Tostino
28-02-06, 07:13
I agree that 10 hrs of them not knowing that the database is compromised (or knowing but not doing anything) is way too much for a service that people are paying for.
On my Lineage2 server I have never had a roll back of more then 4 hrs (Only 3 so far after the server being up since september 05) and my Lineage2 server is a free service.
It's really not that good that KK can't keep their database stable. And there is no way that they should have so much time out of the day that a tech person is not in office, or available to come in if there is a problem.

IceStorm
28-02-06, 07:54
I agree that 10 hrs of them not knowing that the database is compromised (or knowing but not doing anything) is way too much for a service that people are paying for.You're not paying much. I pay $550/yr for five accounts (one's for a friend). That works out to about 6.3 cents per hour, for all five accounts.

NC is a fringe MMO. We all know this. We all know KK's run on part of Martin's allowance money. We all know there are 20 people, tops, that work at KK globally and that of those there are significantly less who have the rights/access/ability to "fix" database problems. We all know they don't sleep in cots next to the rack the servers are in. We all know the "tech" guys are in Europe.

Knowing all this, we choose to play NC. We really don't have any right to expect them to do anything about their tech support response times.

Apocalypsox
28-02-06, 08:11
Now guys, before we get into a spamfest you gotta think about it another way. Neocron is a small game, i admit. Now consider that KK has kept this game up and running for OVER 3 years now, and in the past id say year and a half, (since NC2) many people left about 3 months into NC2, so KK kinda has to run on minimum income. lets look at another circle.

Good Game -> Draws Customers -> customers pay for game -> game expands, gets better staff, can produce updates faster and more efficently and moderate the game better.

now heres NCs loop.

KK Produces great game -> attracts alot of customers at first -> lots of customers, on one server was more than we have on all servers now. -> here comes a big update, "Sequel" supposedly, going good for 3 months, most of the population drops out. -> Most of KKs funding for NC drops out -> lost jobs as cannot any longer pay to have them working -> the game falling into a permanent decline unless people get enough guts to handle a few roll backs in dark times, and a few bugs that are being fixxed (slowly at that, but it gets better every patch slightly)i wont continue anymore, but you guys get the picture. someone said this the other day in a thread i was looking at. i wont repeat it directly.

"Nc is on crutches and loosing blood, When are we going to the hospital?" i thought that was kinda funny :lol: anyways im sure you guys get the picture after all my ranting...just look at it from another point of view.

Nidhogg
28-02-06, 11:32
I know there's helpdesk@neocron.com - but to my knowledge that's just an account support email.
Actually that's not account support (which would be account@neocron.com). Also, if you have a technical problem you can email tech@neocron.com.

To address the crux of the post though, when something goes badly wrong enough to require a rollback there are very few people capable of dealing with that issue so if that happens "out of hours" there may well be a delay. That's not always the case because we will always try to reach the correct people and often succeed in dragging them out of bed (or what have you) but that's not always the case.

On top of that there's the difficulty in recognising the problem in the first place. When I logged off the night it happened it was 1AM UK (2AM in Germany) and I was aware of exactly one person who was having difficulty in logging in. The full extent of the problem only became apparent the next day and with the best will (and procedures) in the world there's little you can do in that situation.

Naturally we'd have preferred that it didn't happen at all but in this particular case I doub't it could have been handled much better than it was.

N

-FN-
28-02-06, 11:46
Actually that's not account support (which would be account@neocron.com). Also, if you have a technical problem you can email tech@neocron.com.
Hey :( I corrected myself a few posts after that! :p

Brammers
28-02-06, 11:52
On top of that there's the difficulty in recognising the problem in the first place. When I logged off the night it happened it was 1AM UK (2AM in Germany) and I was aware of exactly one person who was having difficulty in logging in. The full extent of the problem only became apparent the next day and with the best will (and procedures) in the world there's little you can do in that situation.

One poster put the error message up in this thread http://forum.neocron.com/showthread.php?t=132600 with 3 "me too" replies in the first 5 posts.


"CHARACTER DATA WAS DAMAGED TRYING TO RETRIEVE LAST AUTOMATIC BACKUP. PLEASE RELOG"

To me it sounds no-one around that night had the expertise to understand what the error message meant. Looks like some procedures need to be changed there.


Naturally we'd have preferred that it didn't happen at all but in this particular case I doub't it could have been handled much better than it was.

N

This is the 3rd rollback on Terra in the last 3 months. Granted, none of us like rollbacks, but why is data getting corrupted and AFAIK, only Terra seems to be getting hit by rollbacks and data problems.

IceStorm
28-02-06, 11:52
The full extent of the problem only became apparent the next day and with the best will (and procedures) in the world there's little you can do in that situation.An automated login agent that goes out at regular intervals, logs into the servers, verifies the login completed successfully, and verifies that the items/cash/location of the character are what they should be.

It probably wouldn't be hard to implement (seeing as how KK wrote the game), it would be easy enough to run from agent PCs at the office, in the USA, and at the colo, and it would allow you to see how login sessions are performing at any given time just by checking logs on the agent machines.

It's just a suggestion.

but why is data getting corrupted and AFAIK, only Terra seems to be getting hit by rollbacks.Because it has inhereted Saturn's cooties. We wouldn't have any problems if Terra was only populated by Plutonians. :-)

Nidhogg
28-02-06, 11:54
An automated login agent that goes out at regular intervals, logs into the servers, verifies the login completed successfully, and verifies that the items/cash/location of the character are what they should be.

It probably wouldn't be hard to implement (seeing as how KK wrote the game), it would be easy enough to run from agent PCs at the office, in the USA, and at the colo, and it would allow you to see how login sessions are performing at any given time just by checking logs on the agent machines.

It's just a suggestion.
Given the occasional unpredictability of logins I wouldn't want this utility waking me up in the middle of the night. ;)

N

IceStorm
28-02-06, 12:01
I didn't say it had to wake you up.

It's standard practice for health checks to be performed at the application layer in computing environments. Where I work we have numerous health checks done by application load balancers and external entities. When the checks fail, we can be alerted in one of several different ways, or not alerted depending on the desires of the groups doing the monitoring.

I realize you're making something of a joke, but keep in mind that it is possible to do better monitoring of NC than relying on the userbase's whining. Obviously, you don't have to, but if it was me I'd want to know how, say, 10 agents in the USA, Germany, and the Colo saw the servers over time. Automated trending is a nice thing to have.

Nidhogg
28-02-06, 12:20
But the point of this thread is that something is done as soon as a problem is detected. I don't see how better monitoring is helpful if you don't actually respond when it detects something.

N

IceStorm
28-02-06, 12:34
But the point of this thread is that something is done as soon as a problem is detected.First, it's not like you're the only person there. Alerting could be set up to wake someone else (an on-call of the week).

Second, part of trending and monitoring is to detect problems before users do. In that respect, agents might have seen a fault before you logged off. Assuming you (or an on-call's) routine included checking the agents before logging off, the problem may have been detected earlier.

This is all moot if KK's not interested in babysitting the servers 24x7. It's mostly intended to refute the idea that player whining is the only way to monitor the servers. :-)

Nidhogg
28-02-06, 13:13
First, it's not like you're the only person there. Alerting could be set up to wake someone else (an on-call of the week).

Second, part of trending and monitoring is to detect problems before users do. In that respect, agents might have seen a fault before you logged off. Assuming you (or an on-call's) routine included checking the agents before logging off, the problem may have been detected earlier.

This is all moot if KK's not interested in babysitting the servers 24x7. It's mostly intended to refute the idea that player whining is the only way to monitor the servers. :-)
First, I doubt anyone would want to be woken up every time a software agent failed to log in correctly. There are a hundred possible reasons for that, very few of which would require human intervention.

Second, as I said, we had only one complaint (that I was aware of) at that time but the damage was already done and at least one customer was aware of the problem. Everyone else (presumably including our theoretical software agent) was still able to log in successfully so our agent has failed to do its job right there.

My point is that these things aren't easy to manage but we do our best.

N

Bishop Yutani
28-02-06, 17:01
Good discussion guys.

Nidd, how many on staff are volunteers?

IceStorm
28-02-06, 17:04
First, I doubt anyone would want to be woken up every time a software agent failed to log in correctly.Monitoring in no way needs to be that simplistic.
There are a hundred possible reasons for that, very few of which would require human intervention.Well, that's the point. I'm not talking about a simple yes/no test, I'm talking about an agent that attempts login and returns results based on how far it got and logging those results over time. And I'm talking about running agents at multiple locations, including right in the colo on the same LAN segment(s) as the servers.
My point is that these things aren't easy to manage but we do our best.And my point is that KK relies on playerbase whining for the majority of its "server's down!" alerting. You don't necessarily have to. You can architect alternate means of simulating other players and verifying login state on a regular basis and trend it. You could also use that trending to help you form a reasonable alerting system.

Nidhogg
28-02-06, 17:31
Monitoring in no way needs to be that simplistic.
I'm merely commenting on your original proposal a few posts ago.

Sure, it's possible to run sophisticated detection routines (and we do monitor all manner of statistics from login rates to database cache hits). We also have a sophisticated load balancer that detects failures and transparently moves player sessions to other physical machines. It's rare that you ever feel any pain from server software or hardware problems because the load balancer will just mark that server as unavailable, move everyone to a working server, notify someone and the game continues but on fewer resources.

Bear in mind though that this is a game people play from around the world 24x7 so unlikely that you will be able to preempt your customer-base from finding out that there's a server outage before you do, even if you're checking from a dozen places around the world every minute of the day ;). Also, if you look back at the majority of "OMG SERVER DOWN111!!!ONE" threads you'll see that the servers weren't actually down at all and there was a problem out in the world cutting off a portion of the net.

In any case, we do perform pre-emptive maintenance. You're never even aware of the times when we manage imminent issues (hardware or software) except maybe with a small outage announcement in the case when we have to bring the servers down for any reason.

N

Durandal|AI
28-02-06, 17:56
Io Nidd

in which world are you?

have you seen how long it took you guys to manage the terra problem?

...

cRazy-
28-02-06, 18:09
Io Nidd

in which world are you?

have you seen how long it took you guys to manage the terra problem?

...


lol I dont really understand the whole point of that question....

Durandal|AI
28-02-06, 18:21
lol I dont really understand the whole point of that question....

they watch all servers

they take care of the problems immediately

but - why didnt they show a reaction for almost 8 hours as terra was fucked up again?

Nidhogg
28-02-06, 18:22
Io Nidd

in which world are you?

have you seen how long it took you guys to manage the terra problem?

...
I'm in the world that can come down on yours like a ton of bricks if you're not more polite. ;) If you're talking about the one this week then I've already talked about it earlier in this thread.

N

cRazy-
28-02-06, 18:28
Nidd just think on a more simple basis, employ an australian :p

That should be all 24 hour timezones covered.

L3Ts3L
28-02-06, 19:03
Come on KK just admit it, you guys are just making a last quick buck on us. Everyone left in NC is living in a dream world if you think NC is going to ever recover. You could say it's the beginning of the end. There's no coming back now for most players. The mmo business is booming. And new players have abselutely no idea why they downloaded 1.6 gig. Other mmo's that had atleast 10 times as much costomers have opended the doors. Like 1 year free game time. Don't you think it's time for such a move?

Bishop Yutani
28-02-06, 19:20
too late to go free, unless they advertise in game or something. If anything they need more money.... and a set of standard operating procedures.

RogerRamjet
02-03-06, 14:09
Sorry to drag this back up, but what times do the account@neocron.com people operate?

I get an email around 1AM, immediately reply to it and dont recieve an email for another 24 hours.

Are the staff in Europe or somewhere else (or maybe an insomniac ;p).

Ka0s^
04-03-06, 01:31
Trust me, Us guys have NO IDEA as to what goes on at KK and tbh you wont ever find out. Neocron has never been 100% right yet no matter how you call it we all still find time to come here now and again :)

Somebody said it earlier... Terra is always Fooked because it inherited Saturns bad habits... (anyone that goes back far enough will understand)

BTW hiya Nidd :lol:

Nixon
04-03-06, 04:44
3 Rollbacks in 3 months. A peaking population of 200 people. 200. Bugs that have existed for as long as the game has. Every patch breaks something else.
Memory leaks. Issues with EVERY payment vendor save PayByCash. Planfiles written and promises made, and then just thrown away, or put on the backburner so damn long the flame has gone out. The list is pretty much endless should you choose to go into detail.

People leave. The men in charge say, "you'll be back"

Some do. Some don't. More people have left in the past 3 months than have joined. If this continues, it's a slow painful spiral, towards the time when there are less than a handful of us left.

Neocron is in a coma. At this rate it will die, unless something changes seriously. The 2.1 update was supposed to be a revitalisation. It didn't do anything. It's frustrating.

For the comment that always seems to pop up.



You are barely so.

Do something.

Drastic.


WORD.

I came back to see is it worth it..

It ain't.