Post-Mortem

[Samuel]

After every bit of downtime, it would be nice if we could get a post-mortem describing the issues faced and the solution(s).

If Mick doesn't want to get another sysadmin, then perhaps another option is for the sysadmins on here to weigh in their thoughts on a particular issue. It could potentially keep the site online more with the increased help.

It'll be helpful to the sysadmins since they might learn a thing or two about managing a forum. It'll also be helpful to the community because less people will be wondering what the hell happened, and why it happens over and over.

kthnx

 

[Chearful]

d3l3t3d does this already if you just ask him what happened.

 

[Samuel]

d3l3t3d does this already if you just ask him what happened.

A discussion between a few members about what happened can be a lot better than just a one-on-one discussion.

 

[Fire]

I've been asking mick to keep us updated ever since he took over. Yet to see it happen :/

If d3l3t3d was to keep us updated, I'm sure a lot of the hate for him, would go away.

Forum was fine under BeBosny, now he's system admin its not. What little we know points to it been his fault. If he could let us all know about the problems, people would see past that and understand better.

 

[d3l3t3d]

Hi All,

Thanks for the suggestion. I will start posting what has happened especially in light of this outage that we just had. The irony isn't lost on me that this has come up and the site then goes down for a large chunk of time.

As a heads up, in some instances if I post an issue and I believe that there are security implications around it, I won't be posting up solutions. The reason being that we were attacked directly by a member in the first 4 weeks of my engagement. So, I am very mindful of describing any security issues.

The other thing to consider is that I am not the only person in the team making changes. Given the way the staff are disbursed around the globe, we do sometimes have a left hand right hand thing going on. So, a feature maybe added or changed or updated, that on the surface doesn't look like, or shouldn't have a significant impact, but on scale is enough to cause an issue.

Anyway, in conjunction with the team, I will do my best to post information directly or via the team in regard to what is happening.

HTH

 

[Samuel]

Hi All,

Thanks for the suggestion. I will start posting what has happened especially in light of this outage that we just had. The irony isn't lost on me that this has come up and the site then goes down for a large chunk of time.

As a heads up, in some instances if I post an issue and I believe that there are security implications around it, I won't be posting up solutions. The reason being that we were attacked directly by a member in the first 4 weeks of my engagement. So, I am very mindful of describing any security issues.

The other thing to consider is that I am not the only person in the team making changes. Given the way the staff are disbursed around the globe, we do sometimes have a left hand right hand thing going on. So, a feature maybe added or changed or updated, that on the surface doesn't look like, or shouldn't have a significant impact, but on scale is enough to cause an issue.

Anyway, in conjunction with the team, I will do my best to post information directly or via the team in regard to what is happening.

HTH

If security issues are there with the solution, don't hold off from posting the issue itself (yes, I know you were talking about not posting the solution, but I just want to reinforce this). Who knows? There might be someone here who has solved that particular issue before and could offer some extremely helpful insight.

I've been hit by thousands of different types of attacks and I've managed to mitigate them. During my time at work, where some of our sites exceed half a million requests a day (most notably one of our solutions that requires PDAs to communicate with a web service, and thousands of staff across the UK are using it), I was still able to mitigate a few attacks that came up. I feel like posting a post-mortem would seriously help me (and others) in understanding what issues you're facing - because right now it just seems like a story. Perhaps I'll forgive your team and remove the -rep and offer some assistance/advice, too.

Hopefully this will improve MCM, thanks for replying.

 

[Mick]

Hi All,

Thanks for the suggestion. I will start posting what has happened especially in light of this outage that we just had. The irony isn't lost on me that this has come up and the site then goes down for a large chunk of time.

As a heads up, in some instances if I post an issue and I believe that there are security implications around it, I won't be posting up solutions. The reason being that we were attacked directly by a member in the first 4 weeks of my engagement. So, I am very mindful of describing any security issues.

The other thing to consider is that I am not the only person in the team making changes. Given the way the staff are disbursed around the globe, we do sometimes have a left hand right hand thing going on. So, a feature maybe added or changed or updated, that on the surface doesn't look like, or shouldn't have a significant impact, but on scale is enough to cause an issue.

Anyway, in conjunction with the team, I will do my best to post information directly or via the team in regard to what is happening.

HTH

Thanks d3l3t3d.

Accepted, thanks for the suggestion.