Service Update: Infrastructure Repairs & Relocation [Archive]

Trivaldi

29-09-12, 14:05

Hello,

The issues experienced at our current host have unfortunately reached a new level, which we will not tolerate. We have taken the decision that TITAN and our other infrastructure currently located in Germany will be relocated to a new data centre.

As this is not a minor piece of work it will take some time before the servers are available again. We are of course working to restore the service as soon as possible, however the hardware issue at our current data centre has now also caused some significant damage to our file systems. Therefore we must also concentrate on repairing this damage before bringing the service back online. Sadly this means we cannot give an estimate when the game servers will be back online at the time of writing.

UPDATE:
Expect the server to be available again at the beginning of the week. We know this is a lengthy downtime and we are sorry the process can not be shortened, but we believe that the long term stability of the servers and the riddance of regular server outtages arround US prime time is well worth the wait that comes with transition to a much better specced system in a, fingers crossed, much more reliable datacenter.
As soon as we can pin point a more precise timeframe for the return of the server availability, we'll let you know.
[-M]

Unfortunately we have gone from bad to worse recently, with DDoS attacks leading into issues at our data centre. We hope that this work today will result in the end of our recent availability issues, so that we can once again focus on the day to day operation of Neocron and improving that experience for you.

We appreciate this has been a difficult time for the community, we can only apologise for the inconvenience and hope you understand the difficulties we have faced recently.

Rest assured we won't be giving up.

Your Neocron Support Team.

Dropout

29-09-12, 14:09

Glad to hear that you dont take shit from a (clearly) useless data centre! Keep up the good Work.

Nose

29-09-12, 14:13

I love, how you won't give up, no matter the difficulties!

Keep up the good work.

Ivan Eres

29-09-12, 14:15

Everything's gonna be alright!

Never give up.

Thank you for the effort.

Volunteer Team < 3

Dru Blood

29-09-12, 14:18

Respect! Keep up the good work guys -> We all appreciate it!

It's a shame we cant play right now, but we've been here for over 10 years whats a few days more :)

Jordw

29-09-12, 14:35

Good call on the move :) Glad to see things still moving forward. I need my Neocrack though :p

Ratcattle

29-09-12, 14:51

You are my heroes! Thank you so much for all the fantastic work you do for us!

RogerRamjet

29-09-12, 14:58

Triv, pardon my ignorance, but what is a DDoS attack?

Doc Holliday

29-09-12, 15:03

Triv, pardon my ignorance, but what is a DDoS attack?

Distributed Denial Of Service. When a Botnet (lots of machines that access to the internet and are affected with certain spyware allowing a very naughty person to control them to do bad things) is used to send masses of information and/or sends lots of requests to a server for it to action and it overloads the server because the poor little hamster running it cant keep up.

thats the easy way of putting it. its when assholes want to break things.

RogerRamjet

29-09-12, 15:24

So people are actually attacking the servers? Jesus, people need hobbies.

Beckard

29-09-12, 15:46

Triv, pardon my ignorance, but what is a DDoS attack?
DoS is a description for a range of attacks that aim to bring down servers by overloading them
a good sum up can can be found here:
http://en.wikipedia.org/wiki/Denial-of-service_attack

Hell-demon

29-09-12, 15:49

Don't let the bastards grind you down.

Britney

29-09-12, 16:01

Hope the server gets moved to a good host for once like OVH or Leaseweb or kimsufi or anyone else with a proper support system in place and a good infrastructure. Maybe even with some good peering so they dont have lagspikes????

L0KI

29-09-12, 17:55

This really does sadden me, but it is for the best. Since Titan launched, it's been getting progressively worse. I just hope the downtime isn't too long, as I wouldn't want this fantastic community buzz that's currently in the air to dissipate.

Good luck to all of you; I'll definitely be here, patiently waiting every day.

Mokoi

29-09-12, 18:00

Expect the server to be available again at the beginning of the week.

We know this is a lengthy downtime and we are sorry the process can not be shortened, but we believe that the long term stability of the servers and the riddance of regular server outtages arround US prime time is well worth the wait that comes with transition to a much better specced system in a, fingers crossed, much more reliable datacenter.

As soon as we can pin point a more precise timeframe for the return of the server availability, we'll let you know.
Thanks for your support and patience on this matter.

L0KI

29-09-12, 18:02

Expect the server to be available again at the beginning of the week.

We know this is a lengthy downtime...

Beginning of the week?

That isn't lengthy downtime at all! I was worrying you'd say Mid-October. Great to know I'll be playing again in just a few days :)

Britney

29-09-12, 18:03

It would be nice to know which datacenter the servers are being moved to, and what specs the new server will have.

Or is that too much to ask?

Mokoi

29-09-12, 18:09

It would be nice to know which datacenter the servers are being moved to, and what specs the new server will have.

Or is that too much to ask?

here is a sneak peak of the new TITAN cluster:

https://www.youtube.com/watch?v=FY1XB0rrYes

... nah, just kidding ;)
I do not see a benefit in disclosing details regarding the exact specs, I can assure you however that it is a significant step up and probably the fastest machine ever to host a NC gameserver, now with a proper SLA and 24/7 support. I very much look forward to getting this thing on the grid.

Dropout

29-09-12, 18:20

now with a proper SLA and 24/7 support.
As every hosting company should have ;)

Libra

29-09-12, 18:22

Is there any way that Mars and Terra could be opened?

L0KI

29-09-12, 18:23

Is there any way that Mars and Terra could be opened?

I'm fairly sure the answer to that is no, and for good reason.

The Ottoman

29-09-12, 18:24

I'm starting to feel a little sick. I got a feeling I will be way to sick to work at the beginning of the week.

Nah, I'll wait till monday evening. :D

Vampslord

29-09-12, 18:30

Good job...

May i ask how you can afford all this? It's you gys paying ftom your own pocket?

Mokoi

29-09-12, 18:35

Good job...

May i ask how you can afford all this? It's you gys paying ftom your own pocket?

We are all unpaid volunteers and have daytime jobs that pay for much of the current infrastructure.
As this is a) not a sustainable model in the long run b) a bigger budget would help the game and c) we have the desire to put NC on a solid foundation again, action is being taken to ensure an alternate way of funding.
More on this will follow soon™.

Kane Gregory

29-09-12, 18:55

Mokoi could u trill ur chief to be a neocron fanboy? Maybe he will give u a pay increase =D

Doc Holliday

29-09-12, 18:59

We are all unpaid volunteers and have daytime jobs that pay for much of the current infrastructure.
As this is a) not a sustainable model in the long run b) a bigger budget would help the game and c) we have the desire to put NC on a solid foundation again, action is being taken to ensure an alternate way of funding.
More on this will follow soon™.

TYVM. u just made my day. The fact we can start and help pay and hopefully increase the wonderful things you guys are doing is truly music to my ears.

ps. why would it matter where the datacenter is. surely thats better kept private imo. you know. just in case.

EnZyme X

29-09-12, 18:59

http://www.kickstarter.com/

Kane Gregory

29-09-12, 19:04

Old idea and not possible with the requirements which kickstarter demands.

William Antrim

29-09-12, 19:15

How much do these things cost usually to host? Seriously we talking hundreds or thousands a month? Any ball park ideas anyone?

Britney

29-09-12, 19:38

Well seeing NC was run on 3 or 4 servers about 200 a month i'd say ballpark for a good bunch

Omnituens

29-09-12, 20:11

I do not see a benefit in disclosing details regarding the exact specs, I can assure you however that it is a significant step up and probably the fastest machine ever to host a NC gameserver

Moving from a 386 SX do the DX model then yeah? Fancy.

William Antrim

29-09-12, 20:17

Dollars, £ or Euros Britney?

Britney

29-09-12, 20:43

Dollars, £ or Euros Britney?

Euro's but ballpark figure might be more dependent on the specs of course.

I actually think you could run a neocron server decently on a cheap windows server vps as it's a 10 year old server code now or something? So what was awesome high end back then is what we throw away in the trash without even looking now. I highly doubt it even takes a load of bandwidth from what i've seen playing it the data to and from the server isn't THAT intensive.

But we can only guess cause no one ever released any specs, it seems neocron is just cursed in some arcane way by Crahn.

It does come down to the fact that the Strato server was seemingly doing it's job well till one of the harddisks decided to be a bitch. Then there's no clue why Strato didn't replace it, i assume any self-respecting datacenter runs everything in raid 1 which means you have two copies of everything and can just replace a single disk with no significant downtime!
Again that's mostly guesswork because the real reasons for the server trouble weren't fully disclosed that i can think of.

And the lag-spikes too, were if purely caused by network infrastructure, very unprofessional of Strato.

Anyway moral of the story: I hope we can get a real word on the servers it's not like some kind of arcane trade secret, and Strato sucks donkey balls i wouldn't buy a pen with their name on it!

And lastly: Mokoi and Zoltan and the rest are doing a good job, i just wish they have some better luck hosting-wise this time around!

L0KI

29-09-12, 20:57

How much do these things cost usually to host? Seriously we talking hundreds or thousands a month? Any ball park ideas anyone?

I pay around £45-50 a month for each of my hosted i7 Sandybridge servers (a fraction more for my xeons). This included a ridiculous amount of throughput. A large proportion of my servers are in Germany due them actually being slightly cheaper to host there.

I would assume the setup is something like this :

TERRA (on its own box in Canada)
MARS (on its own box in Canada)
TITAN (on its own box in Germany)
Hosting (web), account management, login system (all on another server in Germany)

So, I'd assume the actual cost of hosting the two servers is £100-200 a month, depending on the amount of traffic/bandwidth, etc.

I'm guessing of course, and I am certainly not trying to down-play it. We, as a community need to cover whatever he's paying, as it shouldn't be coming out of his own pocket.

rune100

29-09-12, 21:00

Seeing as I haven't played any subscribed MMO for years, I would not mind "donating" 10 euro a month. Hopefully it will be possible soon.

Mokoi

29-09-12, 21:04

200 EUR doesn't cut it if you remove all the private infrastructure that is currently being utilized.
We're looking at a couple of hundred euros as the bare minimum required to keep nc afloat with reliable hosting and a proper infrastructure for support services, backup and the like.

However, the real cost lies in personnel expenses of which we do not have any right now, thanks to a bunch of truly awesome guys and gurls (no self-praise intended here, I am just really impressed by what some of the other team members put into this community in terms of work, dedication and heart).
While I personally have no intention to make any money from this project or get paid for work contributed, my opinion (and that is not the announcement of any official roadmap, plan or intent) is that in order to offer a good gaming experience and the quality of service we desire, it would help to be able to at least put one techie and one support person on some sort of payroll again. If and when this is doable, we shall see... as always, soon™

Kame

29-09-12, 21:23

Since you guys probably aren't renting private bays, running your own hardware, I can only wonder why you didn't have the 2 hosting company's services overlap to migrate with less downtime than say, a weekend ?

WuTangClan

29-09-12, 21:25

I think stratos went kaput, kame.

Xiphias

29-09-12, 21:53

Dribble Joy

29-09-12, 21:56

Wow, I thought servers with traffic like this would be way more than that. Like... tens of thousands of pounds per month.

I suppose this isn't 2002 anymore.....

jvasquez33

29-09-12, 22:19

Why can't we play on Terra or Mars since they are in a completely different country with their own servers?

The Real Xtro

29-09-12, 22:38

Well things can't be helped nobody like downtime but hey a weekend offline is better if it ends the problems daily,would love to of been a fly on the wall when you dicided to tell the other peeps to go fuck themselves

Thanks for never giving up i really don't wanna be a panda already quit my guild in wow its amazing how a 10 year old game is defended so strongly ....... nice to hear there is gonna be improvments too can't wait to hear what you have in mind

dWintermut3

29-09-12, 23:27

Thank you so much to the GM and support staff for an absolutely epic effort and never giving up.

L0KI

29-09-12, 23:32

Why can't we play on Terra or Mars since they are in a completely different country with their own servers?

As I said (albeit, a guess), they're likely connected. The account system is probably hosted in Germany, despite the physical game servers being hosted in another part of the world.

Brooklyn

29-09-12, 23:32

I really hope this extended down time can fix some of the issues at hand. However NC has a history of not cooperating. Thanks to all the GM's. If you guys can ever get things stable I would be willing to pay a monthly sub again.

Also just kill off the old servers and concentrate time/resources on Titan.

-B

Mokoi

29-09-12, 23:47

terra and mars are up and crawling.
the master server, which is the gatekeeper to neocron if you like, is currently off the grid.
thus you cannot join any of the servers.

sorry to disappoint you, but we hope to make up for it with great performance and reliability in the near future. :)

jvasquez33

30-09-12, 00:09

As I said (albeit, a guess), they're likely connected. The account system is probably hosted in Germany, despite the physical game servers being hosted in another part of the world.

Ah, would have thought that each server location would have its own account hosting service so this wouldn't happen lol.

Mokoi

30-09-12, 00:11

Ah, would have thought that each server location would have its own account hosting service so this wouldn't happen lol.

now I have seen a lot of reakktorlogic along the way and have an open mind for weird configs, but why on earth would you duplicate account management for each server when your customer subscriptions are not tied to a specific server but the service in general ;)

jvasquez33

30-09-12, 00:29

\\Fényx//

30-09-12, 00:40

So if one hosting center went down, it wouldn't bring them all down? lol
On a game where originally, and most likely the last time it was properly updated, was all run from the same location ?

jvasquez33

30-09-12, 00:53

I had no idea how they handled the game and their hosting servers, hence why I asked why they all were inaccessible with one location going down lol.

borl

30-09-12, 01:00

I guess we'll have to let them off the independently located backup auth servers until the current financial issues are a thing of the past? :)

Beckard

30-09-12, 01:06

now I have seen a lot of reakktorlogic along the way and have an open mind for weird configs, but why on earth would you duplicate account management for each server when your customer subscriptions are not tied to a specific server but the service in general ;)

Doesn't sound so farfetched to me. You can still host the same database in all regions through replication so that you have redundancy plus you could loadbalance mass login after "scheduled" maintenance or reboots. It's a valid design strategy for when you got several world servers spread around the world. I believe a number of larger mmo's are doing just that.

Ofcrouse Neocron does not have the player base nor revenue model to support this kind of design :)

Big B

30-09-12, 01:44

good luck zoltan keep up the good work m8 hope u suceed sooner than later :D

stonehaven25

30-09-12, 02:49

Thanks for all the work you guys put into NC and I appreciate it, so should everyone else like most people.

damien vryce

30-09-12, 03:18

i personally and i think 95% of the current player base and original old schoolers that are back which from what ive seen is ALOT of returning players and a significant amount of new players would agree that NC has SOOO much potential and always has ive never played another mmo thats captivated myself like this one its unique and one of the very first if not the first. nothing has the community like this game or the pvp crafting uniqueness taht we all love. after all isnt that what brings us back to the game time and time again. i certainly feel that this time around the mods/gms zoltan everyone is communicating well and much more interactive with the community and informing us on whats going on and trying to get the game back to where it was and beyond. much thanks to you all and as you all like to say sometime SOON:) we can show our support for you and the game through a possible donation program the fact that so many of us are willing to do this should show you that we believe in the game and want to see it soar to even greater hights than anyone ever imagined at least that is my hopes. thanks again special thanks to nidhogg and danae with helping me with my game infos.

borl

30-09-12, 03:26

Ben 'Zoltan' Wagner @GM_Zoltan (https://twitter.com/GM_Zoltan) (2m)
Okay, TERRA and MARS online for our neocrack addicts! TITAN will follow later!

Doc Holliday

30-09-12, 04:38

wow. that is awesome to hear. good job guys. Get this damn paypal thing already so we can send money. i would rather give you guys $15 a month than blizzard ;)

ancient

30-09-12, 08:46

I'm enjoying the candidness and openness of the GM's and Mods these days. I like how honest they are being with the community, If only it was like this 9 years ago I do feel as there that alot of bad blood that happened in the past would not have been necessary.

It always used to feel like a 'them and us' sort of atmosphere. I do feel like we are all apart of the same community now. I think it's truely awesome.

Thank you guys, honestly.

^^ This. I feel 100% the same and was actually going to post an almost identical post in this thread. I've got such good vibes about the "relaunch" of NC, even with all the problems we are running into atm! I sincerely think that NC could break new ground for it's self with this dev team, as like you say the openness and transparency is something we as players have never had until now. We are actually getting listened to and our ideas/suggestions are being taken into account like never before. I know many people what left purely because of the customer relation problems in previous years.

It's a shame it took what it did for things become like this, but regardless I'm happy we are where we are now. I even have hope that NC could eventually become better then it's ever been. We have an extremely supportive dev team who in the space of a month or so have done more for this game then has been done in years on 0 wages. Imagine what could be achieve when they have a bit of funding!

Doc Holliday

30-09-12, 12:36

Leider ist die Beschädigung der Daten etwas schwerwiegender als erwartet, es braucht noch ein bisschen Zeit bis wir alle Daten wiederhergestellt haben.
Ich kann euch aber beruhigen: Eure Charakterdaten sind natürlich gesichert, da hängt ein weitaus intensiveres Backupsystem dahinter :) Ihr sollten keinen Unterschied zum letzten Stand wahrnehmen.

Vorraussichtliche Zeit bis zum vollständigen Abschluss der Datenrettung: Montag. Ich denke bis dahin sollte auch der neue Server komplett fertig eingerichtet und integriert sein, es ist uns bewusst wie wichtig euch das Spiel ist :)

Dann drückt uns mal die Daumen, dass alles bis dahin glatt läuft.

Z'

Posted in the same thread on the german forum. Apologies if the translation doesnt quite read correctly but this is googles english translation below

Unfortunately, the corruption of the data is slightly more severe than expected, it still needs a bit of time until we have restored all data.
But I can assure you: Your character data are naturally hedged, as a far more intensive backup system is behind it you should not perceive any difference to last stand.

Estimated time period until the completion of data recovery: Monday. I think by then, the new server should be completely finished and furnished integrated, we are aware how important the game is to you

Then keep your fingers fingers crossed that everything is running smoothly so far.

So. At least we know its gonna be roughly monday now. even with terra and mars back.

Still patiently waiting. I feel like i got a sick 10 year old child in hospital and im waiting for the doctor to give me some news. ;)