Regarding Recent Downtime (03/02/17 - 03/03/17)
- Thread starter Mick
- Start date
- Status
PebbleHost
High performance, consistent uptime and fast support. Minecraft hosting that just works.
Same here, not blaming anyone though
Not exactly, it also appears at times when you try opening a PM through a URL. And personal messages are not deleted.
..... part 2.
Step 1, reread part one and realise how badly I had written it. Apologies about that. I was torn between responding with something whilst being pressured to leave, and it being 6 am. I have subsequently altered part one. Although I can't promise this will be much better.
To elaborate further on on part one. Yes, we use cloudfront. Why, all of our services are in AWS as you are now aware. I made that choice, because I believed then as I still do now, that it was the best longer term platform for MCM. I genuinely do care what happens with the market as I think that the potential for all sorts of positive actions can stem from it. So, I attempted to make an informed decision about the spread of the user base and the potential to distribute access across the globe. Hoping that everyone, would have as equally a pleasant experience without anyone in particular suffering latency based on distance.
As brought up in another post, AWS does have base DDoS protection, it isn't brilliant, but their only other option is 3K a month. Which is just not a viable option. So, I had to build my own, so that took me a few weeks to iron out. Which, fingers crossed is now good. As we have had quite a few attacks with a large majority being absorbed with no impact on the site.
The other thing to note. But, first, I am attempting to be clear as I can be, with out divulging anything that would make us vulnerable. And, without bringing up a huge debate that has already happened. So, what I am about to say is purely technical. When given the site, there were a lot of things that I was not given and we couldn't access. Why, doesn't matter, it is just the way it was at the time. So, I didn't have access to the previous cloudflare account. So I simply couldn't direct the site through it. I still may not have used it if I could, but I couldn't, so I had no other option.
Some of the attacks that we have had, I believe have been reasonably substantial. So during January when the site was getting hit the hardest, we would get attacks that would last for about 15 mins in total. The site typically processed about 50% of it before starting to time out. To reiterate the security and secrecy thing. During that time, some of the DDoS attacks came from within. That took me by surprise, I did not think that would happen, I thought the community was too tight for that, naive on my part.
Naivety only enhanced by seeing the amount of people trying, and succeeding to rip each other off. Anyway, I digress. In some cases we absorbed over a million requests in 15 minutes. For comparison, our daily average is about 70K.
So after that had happened, and I got all of the cloudfront things tuned, There were more internal changes to modules and functions. Of which, all don't get run past me. I am not the only person with 100% influence on the site functionality. So that had some small impacts here and there.
But then we started to see some horsepower issues. Which to be honest, I thought was strange. Until I started to look at some of the monthly interaction stats.
Active users - http://prntscr.com/efxj52
User registrations - http://prntscr.com/efxmwb
Thread numbers - http://prntscr.com/efxni0
Gifted upgrades - http://prntscr.com/efxou4
Attachments - http://prntscr.com/efxqk9
and there are a few more. Basically we have been growing and interacting more. In some cases by a reasonable amount.
So I then started a plan to better that, which was what the maintenance window was for about a week and a bit ago. We did a DB change. This I am hoping will help us with load, distribution and scale. The change was good, but 5 days later, AWS did what it did.
I acknowledge that my story doesn't have ultra fine detail. For lots of different reasons. One of which, trying to not have this turn into a novel.
So, there have been quite a few things going on in the last 12 weeks. During which we have had christmas and holidays. I have also had a few other personal things going on, of which I understand, no one really cares about except me. So, I have tried to stay as clinical and on task as I can. However, given the subject matter of some of those things, it hasn't been the easiest thing to do.
So, in summary, I hope that has helped providing in part the info that wraps some clarity around things.
The current random cloudfront pop ups were something I was looking at when AWS went down, so I haven't had another chance since then to look at that, but will.
The basic part is that in the background the index page refreshes itself every 20 odd seconds with a live update. Which hasn't been an issue in the past, but is now throwing on occasion a 502 error. Which would suggest that CF can't talk to the server. But it throws it up in same time as a normal request is answered. So, I do know about it and am working on it.
Again, I hope that has helped.
Step 1, reread part one and realise how badly I had written it. Apologies about that. I was torn between responding with something whilst being pressured to leave, and it being 6 am. I have subsequently altered part one. Although I can't promise this will be much better.
To elaborate further on on part one. Yes, we use cloudfront. Why, all of our services are in AWS as you are now aware. I made that choice, because I believed then as I still do now, that it was the best longer term platform for MCM. I genuinely do care what happens with the market as I think that the potential for all sorts of positive actions can stem from it. So, I attempted to make an informed decision about the spread of the user base and the potential to distribute access across the globe. Hoping that everyone, would have as equally a pleasant experience without anyone in particular suffering latency based on distance.
As brought up in another post, AWS does have base DDoS protection, it isn't brilliant, but their only other option is 3K a month. Which is just not a viable option. So, I had to build my own, so that took me a few weeks to iron out. Which, fingers crossed is now good. As we have had quite a few attacks with a large majority being absorbed with no impact on the site.
The other thing to note. But, first, I am attempting to be clear as I can be, with out divulging anything that would make us vulnerable. And, without bringing up a huge debate that has already happened. So, what I am about to say is purely technical. When given the site, there were a lot of things that I was not given and we couldn't access. Why, doesn't matter, it is just the way it was at the time. So, I didn't have access to the previous cloudflare account. So I simply couldn't direct the site through it. I still may not have used it if I could, but I couldn't, so I had no other option.
Some of the attacks that we have had, I believe have been reasonably substantial. So during January when the site was getting hit the hardest, we would get attacks that would last for about 15 mins in total. The site typically processed about 50% of it before starting to time out. To reiterate the security and secrecy thing. During that time, some of the DDoS attacks came from within. That took me by surprise, I did not think that would happen, I thought the community was too tight for that, naive on my part.
Naivety only enhanced by seeing the amount of people trying, and succeeding to rip each other off. Anyway, I digress. In some cases we absorbed over a million requests in 15 minutes. For comparison, our daily average is about 70K.
So after that had happened, and I got all of the cloudfront things tuned, There were more internal changes to modules and functions. Of which, all don't get run past me. I am not the only person with 100% influence on the site functionality. So that had some small impacts here and there.
But then we started to see some horsepower issues. Which to be honest, I thought was strange. Until I started to look at some of the monthly interaction stats.
Active users - http://prntscr.com/efxj52
User registrations - http://prntscr.com/efxmwb
Thread numbers - http://prntscr.com/efxni0
Gifted upgrades - http://prntscr.com/efxou4
Attachments - http://prntscr.com/efxqk9
and there are a few more. Basically we have been growing and interacting more. In some cases by a reasonable amount.
So I then started a plan to better that, which was what the maintenance window was for about a week and a bit ago. We did a DB change. This I am hoping will help us with load, distribution and scale. The change was good, but 5 days later, AWS did what it did.
I acknowledge that my story doesn't have ultra fine detail. For lots of different reasons. One of which, trying to not have this turn into a novel.
So, there have been quite a few things going on in the last 12 weeks. During which we have had christmas and holidays. I have also had a few other personal things going on, of which I understand, no one really cares about except me. So, I have tried to stay as clinical and on task as I can. However, given the subject matter of some of those things, it hasn't been the easiest thing to do.
So, in summary, I hope that has helped providing in part the info that wraps some clarity around things.
The current random cloudfront pop ups were something I was looking at when AWS went down, so I haven't had another chance since then to look at that, but will.
The basic part is that in the background the index page refreshes itself every 20 odd seconds with a live update. Which hasn't been an issue in the past, but is now throwing on occasion a 502 error. Which would suggest that CF can't talk to the server. But it throws it up in same time as a normal request is answered. So, I do know about it and am working on it.
Again, I hope that has helped.
Last edited:
Thanks for information. I wish you the best of luck resolving the last few problems.
I appreciate your attempt to argue against all the attacks/allegations against you over the past weeks, however I feel as though you didn't really stay on topic and answer some of the main questions people have been asking. My main point for this post is your somewhat childish grammar (technical accuracy). What I mean by this is that you have jumped between points and have composed your argument badly, to the point where what you are saying becomes hard to interpret. A word of advice from me personally is to be more clear in what you have to say and make sure you maintain a professional stature if you want to be taken seriously, especially with what has been happening recently.
Firstly, I want to touch on your decision to use AWS. I may not be the most knowledgeable when it comes to very technical stuff, but what I do know is that this was a bad decision regarding your capability to use the service. I beg to differ with your "informed" idea to use AWS and expand the usability of the site for all clients, as the site under the administration of BeBosny was completely fine and I never once saw a complaint with the speed or uptime. This could mean several things, such as BeBosny being more experienced than you or his methods of running the site were more efficient, which seems the most likely reason because, as Mick has claimed, you and your team (which we are yet to see an appearance of) are professionals. With that said, making your own DDoS protection instead of employing an experienced and developed company to do so is another bad decision on your part. I highly doubt that your ways of protecting the site from attacks are better than a company such as Cloudflare.
The reason you were not given much is because your employer/you essentially stole the administration from BeBosny and violated the agreement Mick had. It's a given that you wouldn't be given much when you snatch the site from somewhere else. Linked with this is your point on not being given the Cloudflare account from BeBosny. He paid for it himself to improve the site for its users, and this was an informed decision. Of course, if you steal the administration from him he would not want to give you something he paid for himself, merely because you want it. There is nothing preventing you from getting Cloudflare, I'm sure it wouldn't damage the substantial profits Mick is receiving and would actually benefit the site.
You receiving DDoS attacks during January is out of the picture now; time has moved on and that is the past. What many people are angry about is the recent downtime (past 3 weeks?) which I can see you have not spoken about too much. DDoS attacks shouldn't be a problem; there is no excuse for you not being able to mitigate them if BeBosny can. "Horsepower" issues should also be out of the picture with Amazon AWS, this site is small in comparison to what else is hosted on the same service, such as Netflix (may be on a better plan), who receive a much larger number of requests per day than this site does. "Growing and interacting more" shouldn't be a problem in the slightest with my previous point in mind.
"Ultra fine detail" is a must when you are your employer have shown a translucent approach to the situation. You must write a substantial amount if that's what’s needed; you got the site into this hole, it's your job to satisfy the customer(s) and dig yourself out of it, completely. If you're not up to that, you might as well leave the job now; managing a community forum may not be for you.
Of course, people must take into consideration your personal problems, but when things are as bad as they are, you should show a certain level of dedication to rectify the issues discussed.
Overall, things haven't been great and I think you know that, however you should be a little less afraid and more transparent, thus people will take you more serious and appreciate what you do (if you do any good, we haven't seen too much yet). I wish you luck in fixing the site and I hope that what I have said has given you some insight into what I, and many other people think of you. I hope this also may give you some solutions to the problems you have been facing and suggest a few fresh ideas.
Last edited:
It might be a configuration issue, but AWS is a great platform. It is fast and some very large websites use the same platform.
A dynamic configuration correctly utilising AWS is going to be fast and efficient.
I'm personally looking to move more of my infrastructure to AWS. It's a great platform.[DOUBLEPOST=1488740804][/DOUBLEPOST]
For DDoS protection? Cloudflare is a mix of CloudFront and a DDoS protection provider and a WAF. It offers a lot of services but doesn't really excel at any of them. It isn't really a CDN or a DDoS protection provider.
Using Cloudflare wouldn't help with DDoS attacks, probably. The only plan with even fair DDoS protection is their business plan @ $200/mo (although I figure that isn't much for this site).
Last edited:
Literally like every time I come on this site it is the laggiest piece of shit. You can make all the excuses that you want but it just makes people not want to come here. If this system admin cannot fix the problems, Mick, then fire him and replace him with someone competent.
It's like you guys are intentionally trying to destroy the site.
now i'll press reply and wait 5 mins for my post to actually work
It's like you guys are intentionally trying to destroy the site.
now i'll press reply and wait 5 mins for my post to actually work
I'm usually not much of a person to cut people slack, but the sysadmin here isn't all too bad as far as the site being "the laggiest piece of shit" goes. If you use inspect element and monitor how long it takes for the site to load, the server optimisation is pretty good and requests/files are returned pretty quickly.
There are a few assets on this site that cause the site to hang and take a lot longer to load, on all pages. The shoutbox and "GFNNotify" are the main two culprits. The site's problems could be attributed to assets and add-ons, rather than a poor server configuration, from what I can see.
Also, a proper move to a platform like AWS, utilising cloud services properly, for a platform like XenForo 1 isn't the easiest thing in the world. So... gotta cut some slack there too.
https://tools.pingdom.com/#!/ejUmbY/http://www.mc-market.org (US location for the best possible results)
As you can see, there's a bunch of scripts loaded (these could be condensed down to fewer files, but with the (good) speed of the server it wouldn't make the biggest difference. Nevertheless, I would still use an add-on to condense these down. The biggest problem, ironically, is Google (analytics and ads). It's ironic because Google has been pushing for faster websites for quite some time now. But, as expected, their ads and analytics platforms are the biggest causes of performance issues.
There are better analytics platforms than Google Analytics but they come at a price (GA is free). Bigger sites tend to use these platforms instead because they don't slow down the page so hard. As for the ads, well, you can't really beat AdSense for the typical site.
To me, the site feels as it normally does (or faster). On paper though, yes, apparently it is pretty slow.
Might be worth noting that AdSense is loaded async, so it won't be hanging the page load (so that's why on paper it's longer to load the page than it actually feels like it is). However, even that async load will feel shit on old PCs, and even my 2014 (or 2015?) MacBook Pro.
Edit: One more thing. It's pretty amazing that Mick hasn't sacked this sysadmin as you all keep suggesting. I can't comment on his work performance, I really don't know any specifics (the same as everyone else here), but he's not doing the worst job. I'm pretty glad that, assuming he is actually doing a good job and the problems aren't his fault, that Mick can face up to people screaming for him to get sacked for months now. A lot of people would've sacked him under pressure. I was pretty sceptical of someone like Mick running this site (not that I care much for this site) but he's done a pretty good job. Should give him some credit every now and then for keeping your beloved marketplace up and running.
and for clarification, you in my post refers to the reader (not you, nara).
Last edited:
*Warning* Conspiracy theories below:
Honestly I don't think Mick is the best owner he could be, but everyone has their flaws. However I think he does still care about it. MCM must also make him a considerable amount of money, through account upgrades, ads etc. I know if it was me that owned MCM, I would be doing everything I could to keep it alive. d3l3t3d showed in his previous post MCM's user base has already started to suffer: http://prnt.sc/efxj52
There are only really two explanations I can come up with:
1. Mick is only 15. Which doesn't matter in terms of how he runs the forum, however we where previously told that it was his dad that hired d3l3t3d and his team. Therefore he might be behind more of what goes on behind the scenes than we might be aware of. It makes sense for him to be running that, as well as the legal side of things, since Mick isn't old enough to do so yet.
As an add-on theory to the above. Does Mick actually own MCM? We where all convinced BeBosny owned MCM then that turned out not to be the case. I'm not sure how much Mick paid for MCM when he bought it, but I can imagine it been a lot. More than someone of 14 (Which is must of been at the time) could really afford. Was it his dad that bought it for him? Therefore he technically owns it.
2. There is something we haven't been told and there is a reason why Mick and d3l3t3d are set on their current path. Maybe things will be better once the lag and downtime is sorted. There could be a very good reason why they are doing things the way they are. If there wasn't, surely Mick would have abandoned it, and started to find an alternative solution for hosting the forum. Mick might be a bit stubborn but I genuinely think that him seeing his forum suffer, would have driven him to hire some extra or alternative people to sort it out, if there wasn't any good reason for things been how they are.
There is a 3rd theory, though I feel that will attract rather mixed opinions, and feel is the less likely of the 3. So ill refrain from posting that yet. But it does put the blame on someone other than the people previously mentioned. Come to think of it, there are quite a few. Might make a full post on it, if I run out of things to do.
Anyhow that's just my thought process at the moment, not to be taken two seriously. But there's two ideas that might explain what's happening.
Honestly I don't think Mick is the best owner he could be, but everyone has their flaws. However I think he does still care about it. MCM must also make him a considerable amount of money, through account upgrades, ads etc. I know if it was me that owned MCM, I would be doing everything I could to keep it alive. d3l3t3d showed in his previous post MCM's user base has already started to suffer: http://prnt.sc/efxj52
There are only really two explanations I can come up with:
1. Mick is only 15. Which doesn't matter in terms of how he runs the forum, however we where previously told that it was his dad that hired d3l3t3d and his team. Therefore he might be behind more of what goes on behind the scenes than we might be aware of. It makes sense for him to be running that, as well as the legal side of things, since Mick isn't old enough to do so yet.
As an add-on theory to the above. Does Mick actually own MCM? We where all convinced BeBosny owned MCM then that turned out not to be the case. I'm not sure how much Mick paid for MCM when he bought it, but I can imagine it been a lot. More than someone of 14 (Which is must of been at the time) could really afford. Was it his dad that bought it for him? Therefore he technically owns it.
2. There is something we haven't been told and there is a reason why Mick and d3l3t3d are set on their current path. Maybe things will be better once the lag and downtime is sorted. There could be a very good reason why they are doing things the way they are. If there wasn't, surely Mick would have abandoned it, and started to find an alternative solution for hosting the forum. Mick might be a bit stubborn but I genuinely think that him seeing his forum suffer, would have driven him to hire some extra or alternative people to sort it out, if there wasn't any good reason for things been how they are.
There is a 3rd theory, though I feel that will attract rather mixed opinions, and feel is the less likely of the 3. So ill refrain from posting that yet. But it does put the blame on someone other than the people previously mentioned. Come to think of it, there are quite a few. Might make a full post on it, if I run out of things to do.
Anyhow that's just my thought process at the moment, not to be taken two seriously. But there's two ideas that might explain what's happening.
Last edited by a moderator:
I do not claim to know much about any of this sort of stuff. All I know is that literally almost every other website on the internet manages to somehow stay up without having problems almost every single day of the week, and I know this site did that just fine for months and months before Mick took over again. A system administrator is the person who is in charge of maintaining the site and ensuring that it runs properly - if it is not running properly, then the responsibility rests on that person. It may not be his fault, but that is what he is employed on this site for, and ultimately if I was in his shoes I'd be taking responsibility for the problems.
I don't think Mick is "brave" for not getting rid of this person, I think he's cheap and he doesn't want to pay for a competent system administrator and he doesn't want to pay to have these problems actually taken care of. I think your admiration and support of him is completely misplaced. How much downtime was there when BeBosny was in charge? I'd say pretty much none, and when there was downtime it was usually scheduled and he let everyone know what was going on. These are problems that Mick created when he took over, and they're problems that are HIS responsibility to fix. I honestly have no idea why you are praising someone for not fixing a problem that they themselves created.
Again, I know little about system administration, websites, etc, but I am 24 years old and I've been using the internet since I was a kid and I honestly can't think of any other site that I've ever frequently gone to that has had as many stability issues as MCM has had in the past few months. Almost every single time I come here lately it is nearly completely unusable and that honestly makes me just want to close my business because it's becoming too much of a headache to deal with.
I'm positive it drives people away from the site. If you were a first time visitor and you came here and it was laggy as shit, would you ever come back? I wouldn't.
This is completely unacceptable and I think it's very weird that you're defending them at all. If this had just started this weekend, I'd be right there with you saying, "Give them some time to fix it!" But it's been happening on and off for months, so the excuses are pretty much all gone. Mick needs to take responsibility, throw some fucking money at the site and hire someone who knows what they're doing, or else the site is going to die.
Last edited:
- Status
