Pragmatic Programming Techniques: Scalable System Design
"Scalability" is not equivalent to "Raw Performance"
Understand environmental workload conditions that the system is design for
Dimension of growth and growth rate: e.g. Number of users, Transaction volume, Data volume
Measurement and their target: e.g. Response time, Throughput
Understand who is your priority customers
Rank the importance of traffic so you know what to sacrifice in case you cannot handle all of them
Scale out and Not scale up
Keep your code modular and simple
Don't guess the bottleneck, Measure it
Bottlenecks are slow code which are frequently executed. Don't optimize slow code if they are rarely executed
Write performance unit test so you can collect fine grain performance data at the component level
Setup a performance lab so you can conduct end-to-end performance improvement measurement easily
Plan for growth
Do regular capacity planning. Collect usage statistics, predict the growth rate
Common Techniques
Server Farm (real time access)
Incoming requests will be dispatched by the load balancer to different machines and hence the workload is spread and shared across the servers in the farm.
Data Partitioning
By nature, data is stateful. So there must be a deterministic mechanism to dispatch data request to the server that host the data
Data partitioning mechanism also need to take into considerations the data access pattern. Data that need to be accessed together should be staying in the same server. A more sophisticated approach can migrate data continuously according to data access pattern shift.
Map / Reduce (Batch Parallel Processing)
Content Delivery Network (Static Cache)
This is common for static media content. The idea is to create many copies of contents that are distributed geographically across servers.
User request will be routed to the server replica with close proximity
Cache Engine (Dynamic Cache)
This is a time vs space tradeoff. Some executions may use the same set of input parameters over and over again. Therefore, instead of redo the same execution for same input parameters, we can remember the previous execution's result.
==> some times, the input may be different, but shares a lot of common parameter, we can query the data source using the common parameters, and cache them, do other filtering in code. ==> if we can.
Resources Pool
DBSession and TCP connection are expensive to create, so reuse them across multiple requests
Calculate an approximate result
Instead of calculate an accurate answer, see if you can tradeoff some accuracy for speed.
If real life, usually some degree of inaccuracy is tolerable
Filtering at the source
Try to do more processing upstream (where data get generated) than downstream because it reduce the amount of data being propagated
Asynchronous Processing
In callback mode, the caller need to provide a response handler when making the call. The call itself will return immediately before the actually work is done at the server side. When the work is done later, response will be coming back as a separate thread which will execute the previous registered response handler. Some kind of co-ordination may be required between the calling thread and the callback thread.
In polling mode, the call itself will return a "future" handle immediately. The caller can go off doing other things and later poll the "future" handle to see if the response if ready. In this model, there is no extra thread being created so no extra thread co-ordination is needed.
Implementation design considerations
Use efficient algorithms and data structure.
Analyze your concurrent access scenarios when multiple threads accessing shared data. Carefully analyze the synchronization scenario and make sure the locking is fine-grain enough. Also watch for any possibility of deadlock situation and how you detect or prevent them. A wrong concurrent access model can have huge impact in your system's scalability. Also consider using Lock-Free data structure (e.g. Java's Concurrent Package have a couple of them)
Analyze the memory usage patterns in your logic. Determine where new objects are created and where they are eligible for garbage collection. Be aware of the creation of a lot of short-lived temporary objects as they will put a high load on the Garbage Collector.
However, never trade off code readability for performance. (e.g. Don't try to bundle too much logic into a single method). Let the VM handle this execution for you.
http://highscalability.com/blog/2008/2/13/whats-your-scalability-plan.html
Move MySQL to a separate server. This frees up resources (CPU, disk, memory). What you want to run on this server depend on its capabilities. Maybe run a memcached server on it.
Move to a distributed memory cache using memcached.
Add a MySQL master/slave configuration.
If more webservers are needed us LVS on the front end as a load balancer.
Read full article from Pragmatic Programming Techniques: Scalable System Design
"Scalability" is not equivalent to "Raw Performance"
Understand environmental workload conditions that the system is design for
Dimension of growth and growth rate: e.g. Number of users, Transaction volume, Data volume
Measurement and their target: e.g. Response time, Throughput
Understand who is your priority customers
Rank the importance of traffic so you know what to sacrifice in case you cannot handle all of them
Scale out and Not scale up
Keep your code modular and simple
Don't guess the bottleneck, Measure it
Bottlenecks are slow code which are frequently executed. Don't optimize slow code if they are rarely executed
Write performance unit test so you can collect fine grain performance data at the component level
Setup a performance lab so you can conduct end-to-end performance improvement measurement easily
Plan for growth
Do regular capacity planning. Collect usage statistics, predict the growth rate
Common Techniques
Server Farm (real time access)
Incoming requests will be dispatched by the load balancer to different machines and hence the workload is spread and shared across the servers in the farm.
Data Partitioning
By nature, data is stateful. So there must be a deterministic mechanism to dispatch data request to the server that host the data
Data partitioning mechanism also need to take into considerations the data access pattern. Data that need to be accessed together should be staying in the same server. A more sophisticated approach can migrate data continuously according to data access pattern shift.
Map / Reduce (Batch Parallel Processing)
Content Delivery Network (Static Cache)
This is common for static media content. The idea is to create many copies of contents that are distributed geographically across servers.
User request will be routed to the server replica with close proximity
Cache Engine (Dynamic Cache)
This is a time vs space tradeoff. Some executions may use the same set of input parameters over and over again. Therefore, instead of redo the same execution for same input parameters, we can remember the previous execution's result.
==> some times, the input may be different, but shares a lot of common parameter, we can query the data source using the common parameters, and cache them, do other filtering in code. ==> if we can.
Resources Pool
DBSession and TCP connection are expensive to create, so reuse them across multiple requests
Calculate an approximate result
Instead of calculate an accurate answer, see if you can tradeoff some accuracy for speed.
If real life, usually some degree of inaccuracy is tolerable
Filtering at the source
Try to do more processing upstream (where data get generated) than downstream because it reduce the amount of data being propagated
Asynchronous Processing
In callback mode, the caller need to provide a response handler when making the call. The call itself will return immediately before the actually work is done at the server side. When the work is done later, response will be coming back as a separate thread which will execute the previous registered response handler. Some kind of co-ordination may be required between the calling thread and the callback thread.
In polling mode, the call itself will return a "future" handle immediately. The caller can go off doing other things and later poll the "future" handle to see if the response if ready. In this model, there is no extra thread being created so no extra thread co-ordination is needed.
Implementation design considerations
Use efficient algorithms and data structure.
Analyze your concurrent access scenarios when multiple threads accessing shared data. Carefully analyze the synchronization scenario and make sure the locking is fine-grain enough. Also watch for any possibility of deadlock situation and how you detect or prevent them. A wrong concurrent access model can have huge impact in your system's scalability. Also consider using Lock-Free data structure (e.g. Java's Concurrent Package have a couple of them)
Analyze the memory usage patterns in your logic. Determine where new objects are created and where they are eligible for garbage collection. Be aware of the creation of a lot of short-lived temporary objects as they will put a high load on the Garbage Collector.
However, never trade off code readability for performance. (e.g. Don't try to bundle too much logic into a single method). Let the VM handle this execution for you.
http://highscalability.com/blog/2008/2/13/whats-your-scalability-plan.html
