再看预算控制模块
http://oldblog.antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html
Redis is different than other database solutions in many ways: it uses memory as main storage support and disk only for persistence
a Redis data set can't be bigger than available memory, so if you have some big data application and a mostly-reads access pattern, Redis is not the right pick.
Slow latest items listings in your home page
We can make both the home page box and the comments time line page with pagination fast using a simple Redis pattern:
Show a leaderboard with the top #100 scores.
Show the user its current global rank.
ZADD leaderboard <score> <username>
To get the top 100 users by score is as easy as ZREVRANGE leaderboard 0 99.
Similarly to tell the user its global rank you just do ZRANK leaderboard <username>
Note that you can do more than this, for instance it is trivial to show the user the scores of users "near" his position, that is, to show the portion of the leaderboard that includes the score of our user.
Order by user votes and time
score = points / time^alpha
Counting stuff
With atomic increments you can take all your counts, reset them atomically with GETSET if needed, put expires in your counters, so that you can take the count of events only if the time difference between those events is less then a given amount of seconds.
INCR user:1
EXPIRE user:1 60
Unique N items in a given amount of time
For instance I want to know the number of unique registered users, or IP addresses, that accessed a given article in an online newspaper.
Every time I get a new pageview I just do the following:
SADD page:day1:<page_id> <user_id>
Of course instead of day1 you may want to use the first second of today, as unix time, like: time()-(time()%3600*24), or something like that.
Want to know the number of unique users? Just do SCARD page:day1:<page_id>.
Need to test if a specific user already accessed that page? Just do SISMEMBER page:day1:<page_id>.
Pub/Sub
QueuesRedis has blocking variants of list pop commands that will block if a list is empty.
Caching
http://www.slideshare.net/itamarhaber/redis-use-patterns-devcontlv-june-2014
Redis (REmote DIctionary Server)
Redis makes u think:
about how data is stored
about how data is accessed
about efficiency
about performance
Pattern: Avoiding Calls to the DB
Denormalization
Non relational no foreign keys, no referential integrity constraints
Thus, data normalization isn't practical
Be prepared to have duplicated data
Pattern: Lists of Items
def view_product(uid, product):
redis.lpush('user:' + uid + ':viewed', product)
redis.ltrim('user:' + uid + ':viewed', 0, 9)
…
def get_last_viewed_products(uid):
return redis.lrange('user:' + uid + ':viewed', 0, -1)
Key Points About Key Names
the convention is to use colons (':') to separate the parts of the key's name
Your schema is your keys' names so keep them in order
Queues
def enqueue(queue, item):
redis.lpush(queue, item)
def dequeue(queue):
return redis.rpop(queue)
# or use brpop for blocking pop
Is Redis ACID? (mostly) Yes!
Redis is (mostly) single threaded, hence every operation is
Atomic
Consistent
Isolated
WATCH/MULTI/EXEC allow something like transactions (no rollbacks)
Server-side Lua scripts ("stored procedures") also behave like transactions
Durability is configurable and is a tradeoff between efficiency and safety
Pattern: Searching
Motivation: finding keys in the database, for example all the users
How #1: use a LIST to store key names
How #2: the *SCAN commands
def do_something_with_all_users():
first = True
cursor = 0
while cursor != 0 or first:
first = False
cursor, data = redis.scan(cursor, 'user:*')
do_something(data)
Pattern: Relationships
Motivation: Redis doesn't have foreign keys, you need to maintain them
> SADD user:1:friends 3 4 5 // Foo is social and makes friends
> SCARD user:1:friends // How many friends does Foo have?
> SINTER user:1:friends user:2:friends // Common friends
> SDIFF user:1:friends user:2:friends // Exclusive friends
> SUNION user:1:friends user:2:friends // All the friends
ZSETs (Sorted Sets)
When the scores are identical, members are sorted alphabetically
Lexicographical ranges are also supported:
ZLEXCOUNT, ZRANGEBYLEX
http://www.yunweipai.com/archives/7325.html
Redis Partitioning即Redis分区，简单的说就是将数据分布到不同的redis实例中，因此对于每个redis实例所存储的内容仅仅是所有内容的一个子集。分区（Partitioning）不仅仅是Redis中的概念，几乎是所有数据存储系统都会涉及到的概念
客户端实现
代理实现即客户端将请求发往代理服务器，代理服务器实现了Redis协议，因此代理服务器可以代理客户端和Redis服务器通信。代理服务器通过配置的分区schema来将客户端的请求转发到正确的Redis实例中，同时将反馈消息返回给客户端
这几天看了一下redis集群，老大让看看把这部分数据迁到集群中。一看吓一跳：算钱相关的，居然redis用作存储，这胆子也是够肥的!
数据量其实很小，不过之前还是业务层做的hash分到多个redis实例中的，但是没有主从。如果redis挂了，预算就完全不受控，然后就没法跟客户交待了。
首先明确一点：redis是一个AP的系统，不保证C，而且...是会丢数据的。如果切换到redis集群可以解决的是容错和可用性问题。但是不会解决的是一致性以及数据不丢。
redis为什么会丢数据？从单机的角度，写aof是周期性的，只有fsync刷盘之后，数据才是真正的持久化了，期间进程挂掉或者突然断什么等意外都可能导致数据丢失。升到集群之后，由于是先回复请求，再异步写副本的，数据也会丢。比如说master回复client写ok，然后master挂了，数据还没有写到slave。接着slave提升为master，这次写就丢失了。
对redis集群做了一个简单的测试，跟单个实例相比，读的性能基本上没有损失。但是写操作性能降低很大。
因为要多一份异步写副本操作，平均到每个实例的处理能力下降了好多。比如在我们内网机器上测试单实例的set操作，不开pipeline，无slave，大概是7-8w的qps。测试3主3从的redis集群，整集群的吞吐量大概才12w，平均到每个实例不到4w了。3个master的CPU跑满之后，slave的CPU大概在40~50%的样子。
回头再看我们的业务，每投出一次广告，都要修改预算数据。而读是周期性(比如10秒)由dispatch读出，再走消息队列同步到竞价服务那边。所以，这是一个写远高于读场景。
总结一下，换到集群是满足了可用性。但是一致性方面，不适合算钱的业务。另外业务模型上面主要是写操作对redis集群也不算啥优势，虽然以现在的的压力完全没任何问题。做肯定是能做，不过这里并不是一个比较适合redis集群的场景。
http://oldblog.antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html
Redis is different than other database solutions in many ways: it uses memory as main storage support and disk only for persistence
a Redis data set can't be bigger than available memory, so if you have some big data application and a mostly-reads access pattern, Redis is not the right pick.
Slow latest items listings in your home page
We can make both the home page box and the comments time line page with pagination fast using a simple Redis pattern:
- Every time a new comment is added, we add its ID into a Redis list: LPUSH latest.comments <ID>.
- We also trim the list to a given length, so that Redis will just hold the latest 5000 items: LTRIM latest.comments 0 5000.
- Every time we need to get a range of items for our latest comments usages, we call a function that will do the following (in pseudo code):
FUNCTION get_latest_comments(start,num_items):
id_list = redis.lrange("latest.comments",start,start+num_items-1)
IF id_list.length < num_items
id_list = SQL_DB("SELECT ... ORDER BY time LIMIT ...")
END
RETURN id_list
END
Leaderboards and related problemsShow a leaderboard with the top #100 scores.
Show the user its current global rank.
ZADD leaderboard <score> <username>
To get the top 100 users by score is as easy as ZREVRANGE leaderboard 0 99.
Similarly to tell the user its global rank you just do ZRANK leaderboard <username>
Note that you can do more than this, for instance it is trivial to show the user the scores of users "near" his position, that is, to show the portion of the leaderboard that includes the score of our user.
Order by user votes and time
score = points / time^alpha
Counting stuff
With atomic increments you can take all your counts, reset them atomically with GETSET if needed, put expires in your counters, so that you can take the count of events only if the time difference between those events is less then a given amount of seconds.
INCR user:1
EXPIRE user:1 60
Unique N items in a given amount of time
For instance I want to know the number of unique registered users, or IP addresses, that accessed a given article in an online newspaper.
Every time I get a new pageview I just do the following:
SADD page:day1:<page_id> <user_id>
Of course instead of day1 you may want to use the first second of today, as unix time, like: time()-(time()%3600*24), or something like that.
Want to know the number of unique users? Just do SCARD page:day1:<page_id>.
Need to test if a specific user already accessed that page? Just do SISMEMBER page:day1:<page_id>.
Pub/Sub
QueuesRedis has blocking variants of list pop commands that will block if a list is empty.
Caching
http://www.slideshare.net/itamarhaber/redis-use-patterns-devcontlv-june-2014
Redis (REmote DIctionary Server)
Redis makes u think:
about how data is stored
about how data is accessed
about efficiency
about performance
Pattern: Avoiding Calls to the DB
Denormalization
Non relational no foreign keys, no referential integrity constraints
Thus, data normalization isn't practical
Be prepared to have duplicated data
Pattern: Lists of Items
def view_product(uid, product):
redis.lpush('user:' + uid + ':viewed', product)
redis.ltrim('user:' + uid + ':viewed', 0, 9)
…
def get_last_viewed_products(uid):
return redis.lrange('user:' + uid + ':viewed', 0, -1)
Key Points About Key Names
the convention is to use colons (':') to separate the parts of the key's name
Your schema is your keys' names so keep them in order
Queues
def enqueue(queue, item):
redis.lpush(queue, item)
def dequeue(queue):
return redis.rpop(queue)
# or use brpop for blocking pop
Is Redis ACID? (mostly) Yes!
Redis is (mostly) single threaded, hence every operation is
Atomic
Consistent
Isolated
WATCH/MULTI/EXEC allow something like transactions (no rollbacks)
Server-side Lua scripts ("stored procedures") also behave like transactions
Durability is configurable and is a tradeoff between efficiency and safety
Pattern: Searching
Motivation: finding keys in the database, for example all the users
How #1: use a LIST to store key names
How #2: the *SCAN commands
def do_something_with_all_users():
first = True
cursor = 0
while cursor != 0 or first:
first = False
cursor, data = redis.scan(cursor, 'user:*')
do_something(data)
Pattern: Relationships
Motivation: Redis doesn't have foreign keys, you need to maintain them
> SADD user:1:friends 3 4 5 // Foo is social and makes friends
> SCARD user:1:friends // How many friends does Foo have?
> SINTER user:1:friends user:2:friends // Common friends
> SDIFF user:1:friends user:2:friends // Exclusive friends
> SUNION user:1:friends user:2:friends // All the friends
ZSETs (Sorted Sets)
When the scores are identical, members are sorted alphabetically
Lexicographical ranges are also supported:
ZLEXCOUNT, ZRANGEBYLEX
Redis Partitioning即Redis分区，简单的说就是将数据分布到不同的redis实例中，因此对于每个redis实例所存储的内容仅仅是所有内容的一个子集。分区（Partitioning）不仅仅是Redis中的概念，几乎是所有数据存储系统都会涉及到的概念
我们为什么要分区
我们为什么要分区？分区的动机是什么？通常来说，Redis分区的好处大致有如下两个方面：
- 性能的提升，单机Redis的网络I/O能力和计算资源是有限的，将请求分散到多台机器，充分利用多台机器的计算能力可网络带宽，有助于提高Redis总体的服务能力。
- 存储的横向扩展，即使Redis的服务能力能够满足应用需求，但是随着存储数据的增加，单台机器受限于机器本身的存储容量，将数据分散到多台机器上存储使得Redis服务可以横向扩展。
范围分区
所谓范围分区，就是将一个范围内的key都映射到同一个Redis实例中，加入数据集还是上面提到的用户数据，具体做法如下：
我们可以将用户ID从0到10000的用户数据映射到R0实例，而将用户ID从10001到20000的对象映射到R1实例，依次类推。
这种方法虽然简单，但是在实际应用中是很有效的，不过还是有问题：
- 我们需要一张表，这张表用来存储用户ID范围到Redis实例的映射关系，比如用户ID0-10000的是映射到R0实例……。
- 我们不仅需要对这张表进行维护，而且对于每种对象类型我们都需要一个这样的表，比如我们当前存储的是用户信息，如果存储的是订单信息，我们就需要再建一张映射关系表。
- 如果我们想要存储的数据的key并不能按照范围划分怎么办，比如我们的key是一组uuid，这个时候就不好用范围分区了。
因此，在实际应用中，范围分区并不是很好的选择，不用担心，我们还有更好的方法，接下来认识下哈希分区。
哈希分区
哈希分区跟范围分区相比一个明显的优点是哈希分区适合任何形式的key，而不像范围分区一样需要key的形式为object_name:<id>，而且分区方法也很简单，一个公式就可以表达：
id=hash(key)%N
其中id代表Redis实例的编号，公式描述的是首先根据key和一个hash函数（如crc32函数）计算出一个数值型的值。接着上面的例子，我们的第一个要处理的key是user:1，hash（user:1）的结果是93024922。
然后哈希结果进行取模，取模的目的是计算出一个介于0到3之间的值，因此这个值才可以被映射到我们的一台Redis实例上面。比如93024922%4结果是2，我们就会知道foobar将要被存储在R2上面。
当然除了上面提到的两种分区方法，还有很多其他的方法。比如一种从哈希分区演进而来的consistent hashing分区，相信信息可以参考我的另一篇文章《memcached分布式实现原理》，其已经被redis client和proxies实现了
客户端实现
代理实现即客户端将请求发往代理服务器，代理服务器实现了Redis协议，因此代理服务器可以代理客户端和Redis服务器通信。代理服务器通过配置的分区schema来将客户端的请求转发到正确的Redis实例中，同时将反馈消息返回给客户端
查询路由
查询路由是Redis Cluster实现的一种Redis分区方式：
查询路由Redis分区示意图
查询路由的过程中，我们可以将查询请求随机的发送到任意一个Redis实例，这个Redis实例负责将请求转发至正确的Redis实例中。Redis集群实现了一个通过和客户端协作的hybrid来做查询路由。
尽管Redis分区到现在为止，so far so good，但是Redis分区有一些致命的缺点，这导致一些Redis功能在分区的环境下并不能很好地工作，我们来看看：
- 多键操作是不被支持的，比如我们将要批量操作的键被映射到了不同的Redis实例中。
- 多键的Redis事务是不被支持的。
- 分区的最小粒度是键，因此我们不能将关联到一个键的很大的数据集映射到不同的实例。
- 当应用分区的时候，数据的处理是非常复杂的，比如我们需要处理多个rdb/aof文件，将分布在不同实例的文件聚集到一起备份。
- 添加和删除机器是很复杂的，例如Redis集群支持几乎运行时透明的因为增加或减少机器而需要做的rebalancing,然而像客户端和代理分区这种方式是不支持这种功能的。
持久存储用还是缓存
尽管数据分区对于Redis来说无论是数据持久化存储还是缓存，在概念上都是一样的，然而对于数据持久化存储还是有一个很大的限制。当我们使用Redis来作为持久化存储的时候，每一个key必须一直被映射到同一个Redis实例。而当Redis被当做缓存使用的时候，对于这个key，如果一个实例不能用了，这个key还可以被映射到其他的实例中。
Consistent hashing实现通常使得当一个key被映射到的实例不能用的时候将这个key映射到其他实例成为可能。类似，如果增加了一台机器，一部分的key将会被映射到这台新的机器上，我们需要了解的两点如下：
- 如果Redis被用来当做缓存，且要求容易增加或删除机器，使用consistent hashing是非常简单的。
- 如果Redis被用来当做（持久）存储，一个固定的key到实例的映射是需要的，因此我们不能够再灵活的添加或删除机器。否则，我们需要在增加或删除机器的时候系统能够rebalace，当前Redis Cluster已经支持。
六、Pre-Sharding
通过上面的介绍，我们知道Redis分区应用起来是有问题的，除非我们只是使用Redis当做缓存，否则对于增加机器或删除机器是非常麻烦的。
然而，通常我们Redis容量变动在实际应用中是非常常见的，比如今天我需要10台Redis机器，明天可能就需要50台机器了。
鉴于Redis是很轻量级的服务（每个实例仅仅占用1M），对于上面的问题一种简单的解决办法是：
我们可以开启多个Redis实例，尽管是一台物理机器，我们在刚开始的时候也可以开启多个实例。我们可以从中选择一些实例，比如32或64个实例来作为我们的工作集群。当一台物理机器存储不够的时候，我们可以将一般的实例移动到我们的第二台物理机上，依次类对，我们可以保证集群中Redis的实例数不变，又可以达到扩充机器的目的。
怎么移动Redis实例呢？当需要将Redis实例移动到独立的机器上的时候，我们可以通过下面步骤实现：
- 在新的物理机上启动一个新的Redis实例。
- 将新的物理机作为要移动的那台的slave机器。
- 停止客户端。
- 更新将要被移动的那台Redis实例的IP地址。
- 对于slave机器发送SLAVEOF ON ONE命令。
- 使用新的IP启动Redis客户端。
- 关闭不再使用的那个Redis实例。
