How Twitter Uses Redis To Scale

http://highscalability.com/blog/2014/9/8/how-twitter-uses-redis-to-scale-105tb-ram-39mm-qps-10000-ins.html
Scaling Redis at Twitter
http://www.csdn.net/article/2014-09-10/2821615-how-twitter-uses-redis-to-scale-105tb-ram-39mm-qps-10000-ins

• Redis is a brilliant idea because it takes underutilized resources on servers and turns them into valuable service.
• Twitter specialized Redis with two new data types that fit their use cases perfectly. So they got the performance they needed, but it locked them into an older code based and made it hard to merge in new features. I have to wonder, why use Redis for this sort of thing? Just create a timeline service using your own datastructures. Does Redis really add anything to the party?
• Summarize large chunks of log data on the node, using your local CPU power, before saturating the network.
• If you want something that’s high performance separate the fast path, which is the data path, away from the slow path, which is the command and control path. 

• Redis drives Timeline, Twitter’s most important service. Timeline is an index of tweets indexed by an id. Chaining tweets together in a list produces the Home Timeline. The User Timeline, which consists of tweets the user has tweeted, is just another list.
• Why consider Redis instead of Memcache? The Network Bandwidth Problem and The Long Common Prefix Problem.
• The Network Bandwidth Problem.
  • Memcache didn’t work as well as Redis for the timeline. The problem was dealing with fanout.
  • Twitter read and writes happen incrementally and they are fairly small, but the timelines themselves are fairly large.
  • When a tweet is generated it needs to be written to all relevant timelines. The tweet is a small piece of data that is attached to some data structure. On read it’s desirable to load a small batch of tweets. On a scroll down another batch is loaded.
  • The hometime line can be largish, what is reasonable for a viewer to read in one set. Maybe 3000 entries, for example. Which means for performance reasons accessing the databases should be avoided.
  • A read-modify-write cycle  for incremental writes, and small reads, on large objects (the timeline), is too expensive and creates a network bottleneck.
  • On a gigalink at 100K+ reads and writes per second, if the average object size is more than 1K, the network becomes the bottleneck.

• A dedicated caching cluster under utilizes CPUs. For simple cases, in-memory key-value stores are CPU light. 1% of CPU time  on a box can handle more than 1K requests per second for small key values. Though for different data structures the result can be different.

• Hotkeys are a problem so they are a building a tiered caching solution with client side caching that will automatically cache hotkeys.

Hybrid List

• Added Hybrid List to Redis for more predictable memory performance.
• Timeline is a list of Tweet IDs, so it’s a list of integers. Each ID is small.
• Redis supports two list types: ziplist and linklist. Ziplist is space efficient. Linked list is flexible, but as a doubly linked list has the overhead of two pointers per key, which given the size of the ID is very high overhead.
• To use memory efficiently ziplists are used exclusively.
• A Redis ziplist threshold is set to the max size of a Timeline. Never store a bigger Timeline than can be stored in a ziplist. This means a product decision, how many tweets can be in a Timeline, are linked to a low level component (Redis). Generally not desirable.
• Adding to and deleting from a ziplist is inefficient, especially with a very large list. Deleting from a ziplist uses memmove to move data around, to make sure the list is still contiguous. Adding to a ziplist requires a memory realloc call to make enough space for the new entry.
• Potential high latency for write operations due to Timeline size. Timelines vary a lot in size. Most users don’t tweet very much, so their User Timeline is small. Home Timelines, especially those involving celebreties can be huge. When updating a large timeline and the cache runs out of heap, which is often the case when using a cache, a very large number of very small timelines will be evicted before there’s enough contiguous RAM to handle one big ziplist. As all this cache management takes time, a write operation can have a high latency.
• Since writes are fanned out to a lot of timelines there’s a higher chance to be caught in a write latency trap as memory is used for expanding the timelines.
• It’s hard to create a SLA for write operations given the high variability of write latencies.
• Hybrid List is a linked list of ziplists. A threshold is set of how big each ziplist can be in bytes. In bytes because to memory efficient it helps to allocate and deallocate blocks of the same size. When a list goes over it is spilled into the next ziplist. A ziplist is not recycled until the list is empty, which means it is possible, through deletion, to have each ziplist have only one entry. In practice, tweets aren’t deleted all that often.
• Before Hybrid List a workaround was to expire larger timelines more quickly, which freed up memory for other timelines, but was expensive when a user went to view their timeline.

BTree

• Added BTree to Redis to support range queries on hierarchical keys to return a list of results.
• In Redis the way to deal with secondary keys or fields is a hash map. To have sorted data in order to perform a range query a sorted set is used. Sorted set orders by a score which is a double, so an arbitrary secondary key or an arbitrary name can’t be used for the sorting. Since hash map uses a linear search it’s not great if there are a lot of secondary keys or fields.
• BTree is the attempt fix the shortcomings of hash map and sorted set. It’s better to just have one data structure that does what you want. It’s easier to understand and reason about.
• Borrowed the BSD implementation of BTree and added it to Redis to create a BTree. Supports key lookup as well as range query. Has good lookup performance. The code is relatively simple. The downside is BTree is not memory efficient. It has a lot of meta data overhead due to the pointers.

Cluster Management

• A cluster is using more than one instance of Redis for a single purpose. If a data set is larger than a single Redis instance can handle or throughput is higher than what a single instance can handle, the key space will need to be partitioned so the data can be stored in more than one shard, across a set of instances. Routing is taking a key and figuring out which shard the data for the key is on.
• Thinks cluster management is the number one reason Redis adoption hasn’t exploded. When a cluster is available there’s no reason not to migrate all cache use cased to Redis.
• Tricky to get Redis cluster right. People use Redis because as a data structure server the idea is to perform frequent updates. But a lot of Redis operations are not idempotent. If there’s a network glitch a retry is required and the data can be corrupted.
• Redis cluster favors having a centralized manager dictating the global view. With memcache a lot clusters use a client side approach based on consistent hashing. If there’s inconsistent data, so be it. To provide really good services, a cluster needs features like detecting which shard is down and then replaying operations to get back in sync. After a long enough period spent down cache state should be cleaned up. Corrupted data in Redis is hard to detect. When there’s a list and it’s missing a chunk, it’s hard to tell.
• Twitter has multiple attempts at building a Redis cluster. Twemproxy which is not used by Twitter internally, it was built for Twemcache and Redis support was added. Two more solutions were based on proxy style routing. One was associated with the Timeline service and not meant to be general. The second was a generalization of the Timeline solution that provided cluster management, replication, and shard repairing.
• Three options in a cluster: servers talk to each other to reach agreement of what a cluster looks like; use a proxy; or do client side cluster management where the clients form a quorum.
• Didn’t go with a server approach because the philosophy is to keep servers simple, dumb and fast.
• Didn’t go with the client because changes are hard to propagate. Approximately 100 projects in Twitter use a cache cluster. Changing anything in the client would have to be pushed to 100 clients it could take years for changes to propagate. Quick iteration means it’s almost impossible to put code in the client.
• Went with a proxy style routing approach and partitioning for two reasons. A cache service is a high performance service. If you want something that’s high performance separate the fast path, which is the data path, away from the slow path, which is the command and control path. If cluster management is merged into the server it complicates the code for Redis, which is a stateful service, any time you want to fix a bug or provide an upgrade to the cluster management code, the stateful Redis service must be restarted too, which will potentially throw away a bunch of data. A rolling restart of a cluster is painful.
• There was a concern using the proxy approach that another network hop is inserted between the client and the server. Profiling showed the extra hop is a myth. At least in their ecosystem. Latency to through the Redis server was less than .5 milliseconds. At Twitter most of the backend services are Java based and use Finagle to talk to each other. When going through the Finagle path the latency was close to 10 milliseconds. So the extra hop isn’t the problem. Inside the JVM is the problem. Outside the JVM you can do pretty much whatever you want, unless of course you go through another JVM.
• Failure of a proxy doesn’t matter much. On the data path introducing a proxy layer isn’t so bad. The client doesn’t care which proxy they talk to. If a proxy fails after a timeout the client goes to another proxy. No sharding is happening at the proxy level, they are all stateless. To scale throughput simply add more proxies. The tradeoff is additional cost. The proxy layer is allocated resources just to do the forwarding. Cluster management, sharding, and doing the view of the cluster happens outside the proxies. The proxies don’t have to agree with each other.
• Twitter has instances that have 100K open connections and it works fine. There’s just overhead to pay. There’s no reason to close connections. Just keep them open, it improves latency.
• Cache clusters are used as a look-aside cache. The caches themselves are not responsible for data replenishment. The client is responsible for fetching a missing key from storage then caching it. If a node goes down the shard is moved to another node. The failed machine is flushed when it comes back so no data is left around. All this is done by the cluster leader. A central viewpoint is really important to keep a cluster in a state that’s easy to understand.

Scale demands predictability.

• Tail latencies matter. When you do fanouts to a lot of shards, when one is slow your entire query will be slow.
• Deterministic configuration is operationally important.

• Super low latency services don’t play well with Mesos today, so these jobs are isolated from other jobs.
• Knowing your resource usage at runtime is really helpful. 

• Push computation to the data. If you look at relative network speeds, CPU speeds, and disk speeds, it makes sense to do computation before going to disk and do computation before going to the network. An example is summarizing logs on a node before they are pushed to a centralized monitoring service. LUA in Redis another way to apply computation close to the data.

• LUA is not production ready in Redis today. On demand scripting means service providers can’t guarantee their SLA. A loaded script can do anything. What service provider would want to take the risk of blowing their SLA because of someone elses code? A deployment model would be better. It would allow for code review and benchmarking, so resource usage and performance could be properly calculated.
• Redis as the next high performance stream processing platform. It has pub-sub and scripting

Twitter Architecture

http://liangzirensheng.blog.163.com/blog/static/189765437201101653936223/
【1】万事开头易
Twitter的核心业务逻辑，在于Following和Be followed。[5]
进入Twitter个人主页，你会看到你following的那些作者，最近发表的微博客。所谓微博客，就是一则短信，Twitter规定，短信的长度不得超过140个字。短信不仅可以包含普通文字信息，也可以包含URL，指向某个网页，或者照片及视频等等。这就是following的过程。
当你写了一则短信并发表以后，你的followers会立刻在他们的个人主页中看到你写的最新短信。这就是be followed的过程。
实现这个业务流程似乎很容易。
1. 为每一个注册用户订制一个Be-followed的表，主要内容是每一个follower的ID。同时，也订制一个Following的表，主要内容是每一个following作者的ID。
2. 当用户打开自己的个人空间时，Twitter先查阅Following表，找到所有following的作者的ID。然后去数据库读取每一位作者最近写的短信。汇总后按时间顺序显示在用户的个人主页上。
3. 当用户写了一则短信时，Twitter先查阅Be-followed表，找到所有followers的IDs。然后逐个更新那些followers的主页。
如果有follower正在阅读他的Twitter个人主页，主页里暗含的JavaScript会自动每隔几十秒，访问一下Twitter服务器，检查正在看的这个个人主页是否有更新。如果有更新，立刻下载新的主页内容。这样follower就能读到最新发表的短信了。
从作者发表到读者获取，中间的延迟，取决于JavaScript更新的间隔，以及Twitter服务器更新每个follower的主页的时间。?
从系统架构上来说，似乎传统的三段论(Three-tier architecture [6])，足够满足这个业务逻辑的需要。事实上，最初的Twitter系统架构，的确就是三段论。

2006年5月Twitter刚上线的时候，为了简化网站的开发，他们使用了Ruby-On-Rails工具，而Ruby-On-Rails的设计思想，就是三段论。
1. 前段，即表述层(Presentation Tier) 用的工具是Apache Web Server，主要任务是解析HTTP协议，把来自不同用户的，不同类型的请求，分发给逻辑层。
2. 中段，即逻辑层 (Logic Tier）用的工具是Mongrel Rails Server，利用Rails现成的模块，降低开发的工作量。
3. 后段，即数据层 (Data Tier) 用的工具是MySQL 数据库。
先说后段，数据层。
Twitter 的服务，可以概括为两个核心，1. 用户，2. 短信。用户与用户之间的关系，是追与被追的关系，也就是Following和Be followed。对于一个用户来说，他只读自己“追”的那些人写的短信。而他自己写的短信，只有那些“追”自己的人才会读。抓住这两个核心，就不难理解 Twitter的其它功能是如何实现的[7]。
围绕这两个核心，就可以着手设计Data Schema，也就是存放在数据层(Data Tier)中的数据的组织方式。不妨设置三个表[8]，
1. 用户表：用户ID，姓名，登录名和密码，状态（在线与否）。
2. 短信表：短信ID，作者ID，正文（定长，140字），时间戳。
3. 用户关系表，记录追与被追的关系：用户ID，他追的用户IDs (Following)，追他的用户IDs (Be followed)。
再说中段，逻辑层。
当用户发表一条短信的时候，执行以下五个步骤，
1. 把该短信记录到“短信表” 中去。
2. 从“用户关系表”中取出追他的用户的IDs。
3. 有些追他的用户目前在线，另一些可能离线。在线与否的状态，可以在“用户表”中查到。过滤掉那些离线的用户的IDs。
4. 把那些追他的并且目前在线的用户的IDs，逐个推进一个队列(Queue)中去。
5. 从这个队列中，逐个取出那些追他的并且目前在线的用户的IDs，并且更新这些人的主页，也就是添加最新发表的这条短信。
以上这五个步骤，都由逻辑层(Logic Tier)负责。前三步容易解决，都是简单的数据库操作。最后两步，需要用到一个辅助工具，队列。队列的意义在于，分离了任务的产生与任务的执行。
队列的实现方式有多种，例如Apache Mina[9]就可以用来做队列。但是Twitter团队自己动手实现了一个队列，Kestrel [10,11]。Mina与Kestrel，各自有什么优缺点，似乎还没人做过详细比较。
不管是Kestrel还是Mina，看起来都很复杂。或许有人问，为什么不用简单的数据结构来实现队列，例如动态链表，甚至静态数组？如果逻辑层只在一台服务器上运行，那么对动态链表和静态数组这样的简单的数据结构，稍加改造，的确可以当作队列使用。Kestrel和Mina这些“重量级”的队列，意义在于支持联络多台机器的，分布式的队列

表述层的主要职能有两 个，1. HTTP协议处理器(HTTP Processor)，包括拆解接收到的用户请求，以及封装需要发出的结果。2. 分发器(Dispatcher)，把接收到的用户请求，分发给逻辑层的机器处理。如果逻辑层只有一台机器，那么分发器无意义。但是如果逻辑层由多台机器组成，什么样的请求，发给逻辑层里面哪一台机器，就大有讲究了。逻辑层里众多机器，可能各自专门负责特定的功能，而在同功能的机器之间，要分摊工作，使负载均衡。
访问Twitter网站的，不仅仅是浏览器，而且还有手机，还有像QQ那样的电脑桌面工具，另外还有各式各样的网站插件，以便把其它网站联系到 Twitter.com上来[12]。因此，Twitter的访问者与Twitter网站之间的通讯协议，不一定是HTTP，也存在其它协议。
三段论的Twitter架构，主要是针对HTTP协议的终端。但是对于其它协议的终端，Twitter的架构没有明显地划分成三段，而是把表述层和逻辑层合二为一，在Twitter的文献中，这二合一经常被称为“API”。

2006年5月Twitter刚上线的时候，Twitter架构与Figure 1差距不大，不一样的地方在于加了一些简单的缓存(Cache)。即便到了现在，Twitter的架构依然可以清晰地看到Figure 1 的轮廓。

【3】Cache == Cash
网站在回应用户请求时的反应速度，是影响用户体验的一大因素。而影响速度的原因有很多，其中一个重要的原因在于硬盘的读写(Disk IO)。
Table 1 比较了内存(RAM)，硬盘(Disk)，以及新型的闪存(Flash)，在读写方面的速度比较。硬盘的读写，速度比内存的慢了百万倍。所以，要提高网站的速度，一个重要措施是尽可能把数据缓存在内存里。当然，在硬盘里也必须保留一个拷贝，以此防范万一由于断电，内存里的数据丢失的情况发生。

Twitter 工程师认为，一个用户体验良好的网站，当一个用户请求到达以后，应该在平均500ms以内完成回应。而Twitter的理想，是达到200ms- 300ms的反应速度[17]。因此在网站架构上，Twitter大规模地，多层次多方式地使用缓存。Twitter在缓存使用方面的实践，以及从这些实践中总结出来的经验教训，是Twitter网站架构的一大看点。


1. Vector Cache and Row Cache
具体来说，Twitter工程师认为最重要的列是IDs。即新发表的短信的IDs，以及被频繁阅读的热门短信的IDs，相关作者的IDs，以及订阅这些作者的读者的IDs。把这些IDs存放进缓存 (Stores arrays of tweet pkeys [14])。在Twitter文献中，把存放这些IDs的缓存空间，称为Vector Cache [14]。
Twitter工程师认为，读取最频繁的内容是这些IDs，而短信的正文在其次。所以他们决定，在优先保证Vector Cache所需资源的前提下，其次重要的工作才是设立Row Cache，用于存放短信正文。
命中率(Hit Rate or Hit Ratio)是测量缓存效果的最重要指标。如果一个或者多个用户读取100条内容，其中99条内容存放在缓存中，那么缓存的命中率就是99%。命中率越高，说明缓存的贡献越大。
设立Vector Cache和Row Cache后，观测实际运行的结果，发现Vector Cache的命中率是99%，而Row Cache的命中率是95%，证实了Twitter工程师早先押注的，先IDs后正文的判断。
Vector Cache和Row Cache，使用的工具都是开源的MemCached [15]。
2. Fragment Cache and Page Cache
前文说到，访问Twitter网站的，不仅仅是浏览器，而且还有手机，还有像QQ那样的电脑桌面工具，另外还有各式各样的网站插件，以便把其它网站联系到Twitter.com上来[12]。接待这两类用户的，是以Apache Web Server为门户的Web通道，以及被称为“API”的通道。其中API通道受理的流量占总流量的80%-90% [16]。
所以，继Vector Cache和Row Cache以后，Twitter工程师们把进一步建筑缓存的工作，重点放在如何提高API通道的反应速度上。
读者页面的主体，显示的是一条又一条短信。不妨把整个页面分割成若干局部，每个局部对应一条短信。所谓Fragment，就是指页面的局部。除短信外，其它内容例如Twitter logo等等，也是Fragment。如果一个作者拥有众多读者，那么缓存这个作者写的短信的布局页面(Fragment)，就可以提高网站整体的读取效率。这就是Fragment Cache的使命。
对于一些人气很旺的作者，读者们不仅会读他写的短信，而且会访问他的主页，所以，也有必要缓存这些人气作者的个人主页。这就是Page Cache的使命。
Fragment Cache和Page Cache，使用的工具也是MemCached。
观测实际运行的结果，Fragment Cache的命中率高达95%，而Page Cache的命中率只有40%。虽然Page Cache的命中率低，但是它的内容是整个个人主页，所以占用的空间却不小。为了防止Page Cache争夺Fragment Cache的空间，在物理部署时，Twitter工程师们把Page Cache分离到不同的机器上去。
3. HTTP Accelerator
解决了API通道的缓存问题，接下去Twitter工程师们着手处理Web通道的缓存问题。经过分析，他们认为Web通道的压力，主要来自于搜索。尤其是面临突发事件时，读者们会搜索相关短信，而不理会这些短信的作者，是不是自己“追”的那些作者。
要降低搜索的压力，不妨把搜索关键词，及其对应的搜索结果，缓存起来。Twitter工程师们使用的缓存工具，是开源项目Varnish [18]。
比较有趣的事情是，通常把Varnish部署在Web Server之外，面向Internet的位置。这样，当用户访问网站时，实际上先访问Varnish，读取所需内容。只有在Varnish没有缓存相应内容时，用户请求才被转发到Web Server上去。而Twitter的部署，却是把Varnish放在Apache Web Server内侧[19]。原因是Twitter的工程师们觉得Varnish的操作比较复杂，为了降低Varnish崩溃造成整个网站瘫痪的可能性，他们便采取了这种古怪而且保守的部署方式。
Apache Web Server的主要任务，是解析HTTP，以及分发任务。不同的Mongrel Rails Server负责不同的任务，但是绝大多数Mongrel Rails Server，都要与Vector Cache和Row Cache联系，读取数据。Rails Server如何与MemCached联系呢？Twitter工程师们自行开发了一个Rails插件(Gem)，称为CacheMoney。
虽然Twitter没有公开Varnish的命中率是多少，但是[17]声称，使用了Varnish以后，导致整个Twitter.com网站的负载下降了50%

4】抗洪需要隔离
如果说如何巧用Cache是Twitter的一大看点，那么另一大看点是它的消息队列(Message Queue)。为什么要使用消息队列？[14]的解释是“隔离用户请求与相关操作，以便烫平流量高峰 (Move operations out of the synchronous request cycle, amortize load over time)”。
其中洪峰时刻，Twitter网站每秒钟收到350条新短信，这个流量洪峰维持了大约5分钟。根据统计，平均每个Twitter用户被120人 “追”，这就 是说，这350条短信，平均每条都要发送120次 [16]。这意味着，在这5分钟的洪峰时刻，Twitter网站每秒钟需要发送350 x 120 = 42,000条短信。
面对洪峰，如何才能保证网站不崩溃？办法是迅速接纳，但是推迟服务。打个比方，在晚餐高峰时段，餐馆常常客满。对于新来的顾客，餐馆服务员不是拒之门外，而是让这些顾客在休息厅等待。这就是[14] 所说的 “隔离用户请求与相关操作，以便烫平流量高峰”。
如何实施隔离呢？当一位用户访问Twitter网站时，接待他的是Apache Web Server。Apache做的事情非常简单，它把用户的请求解析以后，转发给Mongrel Rails Sever，由Mongrel负责实际的处理。而Apache腾出手来，迎接下一位用户。这样就避免了在洪峰期间，用户连接不上Twitter网站的尴尬局面。
虽然Apache的工作简单，但是并不意味着Apache可以接待无限多的用户。原因是Apache解析完用户请求，并且转发给 Mongrel Server以后，负责解析这个用户请求的进程(process)，并没有立刻释放，而是进入空循环，等待Mongrel Server返回结果。这样，Apache能够同时接待的用户数量，或者更准确地说，Apache能够容纳的并发的连接数量(concurrent connections)，实际上受制于Apache能够容纳的进程数量。Apache系统内部的进程机制参见Figure 5，其中每个Worker代表一个进程。
Apache能够容纳多少个并发连接呢？[22]的实验结果是4,000个，参见Figure 6。如何才能提高Apache的并发用户容量呢？一种思路是不让连接受制于进程。不妨把连接作为一个数据结构，存放到内存中去，释放进程，直到 Mongrel Server返回结果时，再把这个数据结构重新加载到进程上去。
事实上Yaws Web Server[24]，就是这么做的[23]。所以，Yaws能够容纳80,000以上的并发连接，这并不奇怪。但是为什么Twitter用 Apache，而不用Yaws呢？或许是因为Yaws是用Erlang语言写的，而Twitter工程师对这门新语言不熟悉 (But you need in house Erlang experience [17])。

Figure 5. Apache web server system architecture [21]
Courtesy http://www.alibuybuy.com/wp-content/uploads/2010/11/3b81_4071355801_db6c8cd6c0_o.png



2. 浏览器上传作者新写的短信(Tweet)，Apache收到短信后，把短信连同作者ID，转发给Mongrel Rails Server。然后Apache进程进入空循环，等待Mongrel的回复，以便更新作者主页，把新写的短信添加上去。
3. Mongrel收到短信后，给短信分配一个ID，然后把短信ID与作者ID，缓存到Vector MemCached服务器上去。
同时，Mongrel让Vector MemCached查找，有哪些读者“追”这位作者。如果Vector MemCached没有缓存这些信息，Vector MemCached自动去MySQL数据库查找，得到结果后，缓存起来，以备日后所需。然后，把读者IDs回复给Mongrel。
接着，Mongrel把短信ID与短信正文，缓存到Row MemCached服务器上去。
4. Mongrel通知Kestrel消息队列服务器，为每个作者及读者开设一个队列，队列的名称中隐含用户ID。如果Kestrel服务器中已经存在这些队列，那就延用以往的队列。
对应于每个短信，Mongrel已经从Vector MemCached那里知道，有哪些读者追这条短信的作者。Mongrel把这条短信的ID，逐个放进每位读者的队列，以及作者本人的队列。
5. 同一台Mongrel Server，或者另一台Mongrel Server，在处理某个Kestrel队列中的消息前，从这个队列的名称中解析出相应的用户ID，这个用户，既可能是读者，也可能是作者。
然后Mongrel从Kestrel队列中，逐个提取消息，解析消息中包含的短信ID。并从Row MemCached缓存器中，查找对应于这个短信ID的短信正文。
这时，Mongrel既得到了用户的ID，也得到了短信正文。接下去Mongrel就着手更新用户的主页，添加上这条短信的正文。
6. Mongrel把更新后的作者的主页，传递给正在空循环的Apache的进程。该进程把作者主页主动传送(push)给作者的浏览器。
如果读者的浏览器事先已经登录Twitter网站，建立连接，那么Apache给该读者也分配了一个进程，该进程也处于空循环状态。Mongrel 把更新后的读者的主页，传递给相应进程，该进程把读者主页主动传递给读者的浏览器。
【6】流量洪峰与云计算
由此可见，Twitter流量的波动十分可观。对于Twitter公司来说，如果预先购置足够的设备，以承受流量的变化，尤其是重大事件导致的洪峰流量，那么这些设备在大部分时间处于闲置状态，非常不经济。但是如果缺乏足够的设备，那么面对重大事件，Twitter系统有可能崩溃，造成的后果是用户流失。
怎么办？办法是变买为租。Twitter公司自己购置的设备，其规模以应付无重大事件时的流量压力为限。同时租赁云计算平台公司的设备，以应付重大事件来临时的洪峰流量。租赁云计算的好处是，计算资源实时分配，需求高的时候，自动分配更多计算资源。

但是租来的计算资源怎么用，又是一个大问题。查看一下[35]，不难发现Twitter把租赁来的计算资源，大部分用于增加Apache Web Server，而Apache是Twitter整个系统的最前沿的环节。
为什么Twitter很少把租赁来的计算资源，分配给Mongrel Rails Server，MemCached Servers，Varnish HTTP Accelerators等等其它环节？在回答这个问题以前，我们先复习一下前一章“数据流与控制流”的末尾，Twitter从写到读的6个步骤。
这6个步骤的前2步说到，每个访问Twitter网站的浏览器，都与网站保持长连接。目的是一旦有人发表新的短信，Twitter网站在500ms 以内，把新短信push给他的读者。问题是在没有更新的时候，每个长连接占用一个Apache的进程，而这个进程处于空循环。所以，绝大多数Apache 进程，在绝大多数时间里，处于空循环，因此占用了大量资源。
事实上，通过Apache Web Servers的流量，虽然只占Twitter总流量的10%-20%，但是Apache却占用了Twitter整个服务器集群的50%的资源[16]。所以，从旁观者角度来看，Twitter将来势必罢黜Apache。但是目前，当Twitter分配计算资源时，迫不得已，只能优先保证Apache的需求。

在第四章“抗洪需要隔离”中，我们曾经打过一个比方，“在晚餐高峰时段，餐馆常常客满。对于新来的顾客，餐馆服务员不是拒之门外，而是让这些顾客在休息厅等待”。对于Twitter系统来说，Apache充当的角色就是休息厅。只要休息厅足够大，就能暂时稳住用户，换句行话讲，就是不让用户收到 HTTP-503的错误提示。
稳住用户以后，接下去的工作是高效率地提供服务。高效率的服务，体现在Twitter业务流程6个步骤中的后4步。

【7】作为一种进步的不彻底
不彻底的工作方式，对于架构设计是一种进步。
当一个来自浏览器的用户请求到达Twitter后台系统的时候，第一个迎接它的，是Apache Web Server。第二个出场的，是Mongrel Rails Server。Mongrel既负责处理上传的请求，也负责处理下载的请求。Mongrel处理上传和下载的业务逻辑非常简洁，但是简洁的表象之下，却蕴含着反常规的设计。这种反常规的设计，当然不是疏忽的结果，事实上，这正是Twitter架构中，最值得注意的亮点。

分割业务流程的做法，并不是Twitter的首创。事实上，三段论的架构，宗旨也是分割流程。Web Server负责HTTP的解析，Application Server负责业务逻辑，Database负责数据存储。遵从这一宗旨，Application Server的业务逻辑也可以进一步分割。

很显然，Twitter的做法，属于事件驱动一派。事件驱动的好处在于动态调用资源。当某一个工作的负担繁重，成为整个流程中的瓶颈的时候，事件驱动的架构可以很方便地调集更多资源，来化解压力。对于单个机器而言，多线程和事件驱动的两类设计，在性能方面的差异，并不是非常明显。但是对于分布式系统而言，事件驱动的优势发挥得更为淋漓尽致。
Twitter把业务流程做了两次分割。一，分离了Mongrel与MySQL数据库，Mongrel不直接插手MySQL数据库的操作，而是委托 MemCached全权负责。二，分离了上传和下载两段逻辑，两段逻辑之间通过Kestrel队列来传递控制指令。

Twitter Finagle

Netty at Twitter with Finagle
Within a service-oriented architecture, services spend most of their time waiting for responses from other upstream services. Using an asynchronous library allows services to concurrently process requests and take full advantage of the hardware.

Netty isn’t tied to any particular protocol. Adding to it is as simple as creating the appropriate event handlers. 

Clients need to connect to and load balance across a cluster of servers. All services need to export metrics (request rates, latencies, etc) that provide valuable insight for debugging service behavior

Finagle relies on Netty for IO multiplexing providing a transaction-oriented framework on top of Netty’s connection-oriented model.

Finagle emphasizes modularity by stacking independent components together. Each component can be swapped in or out depending on the provided configuration.
Finagle’s Abstraction

Finagle’s core concept is a simple function (functional programming is the key here) from Request to Future of Response.

1	type Service[Req, Rep] = Req => Future[Rep]

This simplicity allows for very powerful composition. Service is a symmetric API representing both the client and the server. Servers implement the service interface. The server can be used concretely for testing or Finagle can expose it on a network interface. Clients are provided an implemented service that is either virtual or a concrete representation of a remote server.

For example, we can create a simple HTTP server by implementing a service that takes an HttpReq and returns a Future[HttpRep] representing an eventual response:

1 2 3 4 5

val s: Service[HttpReq, HttpRep] = new Service[HttpReq, HttpRep] {   def apply(req: HttpReq): Future[HttpRep] =     Future.value(HttpRep(Status.OK, req.body)) } Http.serve(":80", s)

A client is then provided with a symmetric representation of that service:

1 2 3

val client: Service[HttpReq, HttpRep] = Http.newService("twitter.com:80") val f: Future[HttpRep] = client(HttpReq("/")) f map { rep => transformResponse(rep) }

This example exposes the server on port 80 of all interfaces and consumes from twitter.com port 80. However we can also choose not to expose the server and instead use it directly:

?

1	server(HttpReq("/")) map { rep => transformResponse(rep) }

Here the client code behaves the same way but doesn’t require a network connection. This makes testing clients and servers very simple and straightforward.

there is a need for application agnostic functionality as well. Timeouts, authentication, and statics are a few examples. Filters provide an abstraction for implementing application-agnostic functionality.

Filters receive a request and a service with which it is composed.

?

1	type Filter[Req, Rep] = (Req, Service[Req, Rep]) => Future[Rep]

Filters can be chained together before being applied to a service.

?

1 2 3 4

recordHandletime andThen traceRequest andThen collectJvmStats andThen myService

This allows for clean abstractions of logic and good separation of concerns. Internally, Finagle heavily uses filters. Filters enhance modularity and reusability. They’ve proved valuable for testing as they can be unit tested in isolation with minimal mocking.

Failure Management

At scale, failure becomes common rather than exceptional;

Finagle can balance load across a cluster of hosts implicitly using latency as a heuristic. A Finagle client locally tracks load on every host it knows about. It does so by counting the number of outstanding requests being dispatched to a single host. Given that, Finagle will dispatch new requests to hosts with the lowest load and implicitly the lowest latency.

Failed requests will cause Finagle to close the connection to the failing host and remove it from the load balancer. In the background, Finagle will continuously try to reconnect. The host will be re-added to the load balancer only after Finagle can re-establish a connection. Service owners are then free to shut down individual hosts without negatively impacting downstream clients. Clients also keep per-connection health heuristics and remove the connection if it’s deemed unhealthy.

the flow of data is defined and we leave the concurrency to Finagle. We don’t have to manage thread pools or worry about race conditions. 

Recently the internal structure of Finagle has been updated to be more modular, paving the way for an upgrade to Netty 4.

Your Server as a Function by Marius Eriksen provides more insight into Finagle’s philosophy.

https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rpc-system

Sophisticated network servers and clients have many moving parts: failure detectors, load-balancers, failover strategies, and so on. These parts need to work together in a delicate balance to be resilient to the varieties of failure that occur in a large production system.

Finagle provides a robust implementation of:

• connection pools, with throttling to avoid TCP connection churn;
• failure detectors, to identify slow or crashed hosts;
• failover strategies, to direct traffic away from unhealthy hosts;
• load-balancers, including “least-connections” and other strategies; and
• back-pressure techniques, to defend servers against abusive clients and dogpiling.

Additionally, Finagle makes it easier to build and deploy a service that

• publishes standard statistics, logs, and exception reports;
• supports distributed tracing (a la Dapper) across protocols;
• optionally uses ZooKeeper for cluster management; and
• supports common sharding strategies.

Finagle is flexible and easy to use because it is designed around a few simple, composable primitives: Futures, Services, and Filters.

Futures can be combined and transformed in interesting ways, leading to the kind of compositional behavior commonly seen in functional programming languages.

val service = new Service[HttpRequest, HttpResponse] {

  def apply(request: HttpRequest) =

    Future(new DefaultHttpResponse(HTTP_1_1, OK))

}

val address = new InetSocketAddress(10000)

val server: Server[HttpRequest, HttpResponse] = ServerBuilder()

  .name("MyWebServer")

  .codec(Http())

  .bindTo(address)

  .build(service)

Filters are a useful way to isolate distinct phases of your application into a pipeline. For example, you may need to handle exceptions, authorization, and so forth before your Service responds to a request.

http://www.infoq.com/cn/news/2014/05/twitter-finagle-intro

https://en.wikipedia.org/wiki/Back_pressure

Back pressure refers to pressure opposed to the desired flow of a fluid in a confined place such as a pipe. It is often caused by obstructions or tight bends in the confinement vessel along which it is moving, such as piping or air vents.