As a test I was asked the following question

“describe how the internet works. Be detailed. Ignore the physical layer. You have 1 hour and no reference materials.”

Wow. It was kind of fun though. Below the jump is How The Internet Works by Me. Typos and all. Total stream of consciousness writing. Honestly, reference materials would have just slowed me down.

If you think it’s easy, get out a timer, and go – just don’t read mine first. ;-)

Continue reading “Testing”

Scaling Wide

One of the questions that I am asked often is “how do I scale my web application?” or “we’re expecting 200 times the traffic tomorrow, what do I do?”. The second question is usually too late.

The first however, when asked at an appropriate time, can be what saves you from asking the second. A good knowledge of best practices to start with, and then a rigorous testing and introspection process will do more to inform you “when” then any thing else.

Continue reading “Scaling Wide”

Me on Web App Scaling

I have have a couple of things published on Web App Scaling in the recent past. Here is a list:

  1. A Quick Primer On Sharding For Ruby On Rails: I discuss application sharding techniques at a very high level.
  2. Masters Interview at New Relic: I talk about all kinds of scaling concepts.
  3. Filesystem Sharding Tactics and Processes: A look at how to shard filesystems. Many of the ideas can be used for sharding just about anything.

For those that do not know. I’d like to take a second and define sharding – even though I did not invent the term. Sharding is taking a system and breaking it up into smaller pieces, each with their own dedicated resources, to improve performance. It’s that simple.

Anyways, enjoy!

Filesystem scaling in high traffic web applications

One of the more interesting problems that high traffic, high load web applications face is how to scale the filesystem. As load increases, vertical scaling (bigger servers) simply does not work. You have to add more application servers, and more layers to effectively serve the needs of the users. One of the issues that many such applications have to overcome is how to serve assets (pictures, podcasts, video) to all of the edge servers that a user might access the application from, and allow users to upload user generated content.

Continue reading “Filesystem scaling in high traffic web applications”