tag:blogger.com,1999:blog-590774712948906419.post8698887327110013797..comments2023-10-27T00:32:39.200-07:00Comments on Pete Miron: How to Scale Backend Infrastructurepete mironhttp://www.blogger.com/profile/10573198508233227961noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-590774712948906419.post-85317029828146364962010-09-16T20:00:00.509-07:002010-09-16T20:00:00.509-07:00Great point, Mike.
One thing I didn't mention...Great point, Mike.<br /><br />One thing I didn't mention in the initial post is that an old boss of mine used to threaten to do "Axe Testing". Where he threatened to take a random box in the data center off line, and we needed to make sure the system stayed online. I've had a few boxes in AWS become totally inaccessible. We need to write a post on when bad things happen to good infrastructure.<br /><br />- both plugs in a server plugged into same (failed) UPS.<br />- Data Center power runs generator but AC does not. (how hot did those machines get?)<br />- DNS failure causes app server to take 60 seconds to find its localhost IP (127.0.0.1) - I believe that was an Oracle JDBC driver.<br />- Machines auto-negotiate network down to 10mbps. (personal fave.)<br />- Nasdaq shut off switch due to NetBIOS chatter on misconfigured server.<br />- Shower pump floods server closet and takes out T1 routers.<br /><br />Things will go offline in unpredictable, unimaginable ways.pete mironhttps://www.blogger.com/profile/10573198508233227961noreply@blogger.comtag:blogger.com,1999:blog-590774712948906419.post-27604928414386302502010-09-16T18:46:17.094-07:002010-09-16T18:46:17.094-07:00I'd like to highlight an implied concept, whic...I'd like to highlight an implied concept, which is usually taken for granted until something bad happens: Build your systems with the idea of robust infrastructure. When you build systems that can really scale you know that hardware and software will break. Make your systems robust enough to withstand multiple hardware/software failures at the same time. Just because you can reproduce a system quickly doesn't mean you don't have a giant single point of failure. Provisioning and configuration management is not enough. The App must understand how to survive when components go offline.candymanhttps://www.blogger.com/profile/05603480680534514316noreply@blogger.comtag:blogger.com,1999:blog-590774712948906419.post-33211476876486774742010-09-16T18:14:54.651-07:002010-09-16T18:14:54.651-07:00In memory caching is the logical extension of gett...In memory caching is the logical extension of getting your db in ram. We use memcached pretty extensively. The market data systems I've worked on kept all realtime quotes in in-memory hashtables.<br /><br />I've not personally used reddis, but was discussing it today as a possible replacement for a homegrown workflow system. If you use it, let me know what you find.pete mironhttps://www.blogger.com/profile/10573198508233227961noreply@blogger.comtag:blogger.com,1999:blog-590774712948906419.post-9734960226377089562010-09-16T17:14:18.447-07:002010-09-16T17:14:18.447-07:00Nice write up Pete, thanks for sharing.
I'm c...Nice write up Pete, thanks for sharing.<br /><br />I'm curious what your opinion on in memory cacheing systems like redis or memcached is?mnogahttps://www.blogger.com/profile/14189487864917039737noreply@blogger.com