Distributed Misc

http://www.cnblogs.com/javanerd/p/6444536.html

第二章 系统模型

描述分布式系统的三种模型

• Physical models : 用机器，网络，硬件等语言去描述整个系统。
• Architectural models : 用计算、计算任务、计算单元等语言去描述整个系统。（比上一中抽象）
• Fundametal models : 用抽象的方式去描述整个系统，主要包括三个方面：interaction models( 系统间的通信)，failure models （描述故障），security models

Architectural models

在这个模型中，考察四个方面：

• 在分布式系统中，通讯的实体（entity）是什么？
• 实体间的通信方式。（ interprocess communication - socket 调用，remote invocation - rpc ，indirect communication - 消息队列）
• 各个实体的在分布式系统的中的角色。
• 各个实体在物理层面，对应的是什么设施。

Fundamental models

用基本的模型主要为了解决两个问题

• 对于分布式系统中的一切假设去建模
• 对于一些假设去证明是否可能，或者不可能。

interaction models

主要影响系统间通信的两个方面：

1. 通信性能受限
2. 没有一个全局时间

failures models

主要关注系统是否故障，故障可以分类

1. Omission failures（无响应，未履行的故障）
2. 1.1 Process omission：进程挂了，在异步分布式系统中，这种故障检测只能靠timeout，但是timeout有可能只是因为系统响应慢了。
3. 1.2 Communication omission failures
4. Arbitrary failures（Byzantine failure）：随机的failures
5. Timing failures： 只在同步的分布式系统中有，指的是各种时间没有在bound中。例如消息延迟了。

https://en.wikipedia.org/wiki/Byzantine_fault_tolerance
在分布式计算中，不同的计算机通过通讯交换信息达成共识而按照同一套协作策略行动。但有时候，系统中的成员计算机可能出错而发送错误的信息，用于传递信息的通讯网络也可能导致信息损坏，使得网络中不同的成员关于全体协作的策略得出不同结论^[2]，从而破坏系统一致性^[3]。拜占庭将军问题被认为是容错性问题中最难的问题类型之一。

一组拜占庭将军分别各率领一支军队共同围困一座城市。为了简化问题，将各支军队的行动策略限定为进攻或撤离两种。因为部分军队进攻部分军队撤离可能会造成灾难性后果，因此各位将军必须通过投票来达成一致策略，即所有军队一起进攻或所有军队一起撤离。因为各位将军分处城市不同方向，他们只能通过信使互相联系。在投票过程中每位将军都将自己投票给进攻还是撤退的信息通过信使分别通知其他所有将军，这样一来每位将军根据自己的投票和其他所有将军送来的信息就可以知道共同的投票结果而决定行动策略。

系统的问题在于，将军中可能出现叛徒，他们不仅可能向较为糟糕的策略投票，还可能选择性地发送投票信息。假设有9位将军投票，其中1名叛徒。8名忠诚的将军中出现了4人投进攻，4人投撤离的情况。这时候叛徒可能故意给4名投进攻的将领送信表示投票进攻，而给4名投撤离的将领送信表示投撤离。这样一来在4名投进攻的将领看来，投票结果是5人投进攻，从而发起进攻；而在4名投撤离的将军看来则是5人投撤离。这样各支军队的一致协同就遭到了破坏。

由于将军之间需要通过信使通讯，叛变将军可能通过伪造信件来以其他将军的身份发送假投票。而即使在保证所有将军忠诚的情况下，也不能排除信使被敌人截杀，甚至被敌人间谍替换等情况。因此很难通过保证人员可靠性及通讯可靠性来解决问题。

假始那些忠诚（或是没有出错）的将军仍然能通过多数决定来决定他们的战略，便称达到了拜占庭容错。在此，票都会有一个默认值，若消息（票）没有被收到，则使用此默认值来投票。

Scalability Misc

Scalability Testing
https://msdn.microsoft.com/en-us/library/aa292189(v=vs.71).aspx
Scalability testing is an extension of performance testing. The purpose of scalability testing is to identify major workloads and mitigate bottlenecks that can impede the scalability of the application.

Use performance testing to establish a baseline against which you can compare future performance tests. As an application is scaled up or out, a comparison of performance test results will indicate the success of scaling the application. When scaling results in degraded performance, it is typically the result of a bottleneck in one or more resources.

https://en.wikipedia.org/wiki/Scalability_testing

Scalability testing, is the testing of a software application to measure its capability to scale up or scale out in terms of any of its non-functional capability.

The main goals of scalability testing are to determine the user limit for the web application and ensure end user experience, under a high load, is not compromised. One example is if a web page can be accessed in a timely fashion with a limited delay in response. Another goal is to check if the server can cope i.e. Will the server crash if it is under a heavy load?

Increment loads

Mini Inverted Index - LintCode

https://github.com/zxqiu/leetcode-lintcode/blob/master/system%20design/Inverted_Index.javaCreate an inverted index with given documents.
 Notice
Ensure that data does not include punctuation.
 Example

Given a list of documents with id and content. (class Document)
[
  {
    "id": 1,
    "content": "This is the content of document 1 it is very short"
  },
  {
    "id": 2,
    "content": "This is the content of document 2 it is very long bilabial bilabial heheh hahaha ..."
  },
]
Return an inverted index (HashMap with key is the word and value is a list of document ids).
{
   "This": [1, 2],
   "is": [1, 2],
   ...
}
解：
这道题只需要把每个document中的每个词作为key放进HashMap中，并且把对应的document id组成一个List放进HashMap的value中。
注意需要查重，避免同样的id多次放进某一个key的List里面。

 * Definition of Document:
 * class Document {
 *     public int id;
 *     public String content;
 * }
 */
public class Solution {
    /**
     * @param docs a list of documents
     * @return an inverted index
     */
    public Map<String, List<Integer>> invertedIndex(List<Document> docs) {
        Map<String, List<Integer>> ret = new HashMap<String, List<Integer>>();
        if (docs == null) {
            return ret;
        }
       
        for (Document doc : docs) {
            String[] words = doc.content.split(" ");
           
            for (String word : words) {
                if (word.length() == 0) {
                    continue;
                }
                if (ret.containsKey(word)) {
                    if (ret.get(word).contains(doc.id)) {
                        continue;
                    }
                    ret.get(word).add(doc.id);
                } else {
                    List<Integer> list = new ArrayList<Integer>();
                    list.add(doc.id);
                    ret.put(word, list);
                }
            }
        }
       
        return ret;
    }

Mini Cassandra - LintCode

https://github.com/zxqiu/leetcode-lintcode/blob/master/system%20design/Mini_Cassandra.java
Cassandra is a NoSQL storage. The structure has two-level keys.
Level 1: raw_key. The same as hash_key or shard_key.
Level 2: column_key.
Level 3: column_value
raw_key is used to hash and can not support range query. let's simplify this to a string.
column_key is sorted and support range query. let's simplify this to integer.
column_value is a string. you can serialize any data into a string and store it in column value.
implement the following methods:
insert(raw_key, column_key, column_value)
query(raw_key, column_start, column_end) // return a list of entries
Example
insert("google", 1, "haha")
query("google", 0, 1)
>> [（1, "haha")]
解：
使用HashMap来定位raw_key，每个raw_key对应一个TreeMap。
利用TreeMap自排序特征（基于红黑树）来对column_key排序。
/**
 * Definition of Column:
 * public class Column {
 *     public int key;
 *     public String value;
 *     public Column(int key, String value) {
 *         this.key = key;
 *         this.value = value;
 *    }
 * }
 */
public class MiniCassandra {
   
    HashMap<String, TreeMap<Integer, String>> store;

    public MiniCassandra() {
        store = new HashMap<String, TreeMap<Integer, String>>();
    }
   
    /**
     * @param raw_key a string
     * @param column_start an integer
     * @param column_end an integer
     * @return void
     */
    public void insert(String raw_key, int column_key, String column_value) {
        TreeMap<Integer, String> data;
        if (store.containsKey(raw_key)) {
            data = store.get(raw_key);
        } else {
            data = new TreeMap<Integer, String>();
        }
       
        data.put(column_key, column_value);
        store.put(raw_key, data);
    }

    /**
     * @param raw_key a string
     * @param column_start an integer
     * @param column_end an integer
     * @return a list of Columns
     */
    public List<Column> query(String raw_key, int column_start, int column_end) {
        List<Column> ret = new ArrayList<Column>();
       
        if (!store.containsKey(raw_key)) {
            return ret;
        }
       
        TreeMap<Integer, String> data = store.get(raw_key);
        for (Map.Entry<Integer, String> entry : data.subMap(column_start, true, column_end, true).entrySet()) {
            ret.add(new Column(entry.getKey(), entry.getValue()));
        }
       
        return ret;
    }

Mini Memcache

https://github.com/zxqiu/leetcode-lintcode/blob/master/system%20design/Memcache.java
Implement a memcache which support the following features:
get(curtTime, key). Get the key's value, return 2147483647 if key does not exist.
set(curtTime, key, value, ttl). Set the key-value pair in memcache with a time to live (ttl). The key will be valid from curtTime to curtTime + ttl - 1 and it will be expired after ttl seconds. if ttl is 0, the key lives forever until out of memory.
delete(curtTime, key). Delete the key.
incr(curtTime, key, delta). Increase the key's value by delta return the new value. Return 2147483647 if key does not exist.
decr(curtTime, key, delta). Decrease the key's value by delta return the new value. Return 2147483647 if key does not exist.
It's guaranteed that the input is given with increasingcurtTime.
Clarification
Actually, a real memcache server will evict keys if memory is not sufficient, and it also supports variety of value types like string and integer. In our case, let's make it simple, we can assume that we have enough memory and all of the values are integers.
Search "LRU" & "LFU" on google to get more information about how memcache evict data.
Try the following problem to learn LRU cache:
http://www.lintcode.com/problem/lru-cache
Example
get(1, 0)
>> 2147483647
set(2, 1, 1, 2)
get(3, 1)
>> 1
get(4, 1)
>> 2147483647
incr(5, 1, 1)
>> 2147483647
set(6, 1, 3, 0)
incr(7, 1, 1)
>> 4
decr(8, 1, 1)
>> 3
get(9, 1)
>> 3
delete(10, 1)
get(11, 1)
>> 2147483647
incr(12, 1, 1)
>> 2147483647
*/


public class Memcache {
    private class Data {
        int value;
        int ttl;
        int editTime;
       
        public Data(int value, int ttl, int editTime) {
            this.value = value;
            this.ttl = ttl;
            this.editTime = editTime;
        }
    }
   
    HashMap<Integer, Data> cache;

    public Memcache() {
        cache = new HashMap<Integer, Data>();
    }

    public int get(int curtTime, int key) {
        int value = 2147483647;
        if (!cache.containsKey(key)) {
            return value;
        }
       
        Data data = cache.get(key);
        if (data.ttl == 0 || data.ttl + data.editTime - 1 >= curtTime) {
            value = data.value;
        }
       
        return value;
    }

    public void set(int curtTime, int key, int value, int ttl) {
        Data data = new Data(value, ttl, curtTime);
        cache.put(key, data);
    }

    public void delete(int curtTime, int key) {
        if (cache.containsKey(key)) {
            cache.remove(key);
        }
    }
   
    public int incr(int curtTime, int key, int delta) {
        int value = 2147483647;
        if (!cache.containsKey(key)) {
            return value;
        }
       
        Data data = cache.get(key);
        if (data.ttl == 0 || data.ttl + data.editTime - 1 >= curtTime) {
            data.value += delta;
            value = data.value;
            cache.put(key, data);
        } else {
            cache.remove(key);
        }
       
        return value;
    }

    public int decr(int curtTime, int key, int delta) {
        int value = 2147483647;
        if (!cache.containsKey(key)) {
            return value;
        }
       
        Data data = cache.get(key);
        if (data.ttl == 0 || data.ttl + data.editTime - 1 >= curtTime) {
            data.value -= delta;
            value = data.value;
            cache.put(key, data);
        } else {
            cache.remove(key);
        }
       
        return value;
    }

GFS Client

https://github.com/zxqiu/leetcode-lintcode/blob/master/system%20design/GFS_Client.java
Implement a simple client for GFS (Google File System, a distributed file system), it provides the following methods:
    read(filename). Read the file with given filename from GFS.
    write(filename, content). Write a file with given filename & content to GFS.
There are two private methods that already implemented in the base class:
    readChunk(filename, chunkIndex). Read a chunk from GFS.
    writeChunk(filename, chunkIndex, chunkData). Write a chunk to GFS.
To simplify this question, we can assume that the chunk size is chunkSize bytes. (In a real world system, it is 64M). The GFS Client's job is splitting a file into multiple chunks (if need) and save to the remote GFS server. chunkSize will be given in the constructor. You need to call these two private methods to implement read & write methods.
 Example
GFSClient(5)
read("a.txt")
>> null
write("a.txt", "World")
>> You don't need to return anything, but you need to call writeChunk("a.txt", 0, "World") to write a 5 bytes chunk to GFS.
read("a.txt")
>> "World"
write("b.txt", "111112222233")
>> You need to save "11111" at chink 0, "22222" at chunk 1, "33" at chunk 2.
write("b.txt", "aaaaabbbbb")
read("b.txt")
>> "aaaaabbbbb"
解：
由于没有提供Master Server，需要Client自己记录每个文件名对应的所有Chunk Server。
分析Example，发现Chunk Server Index只需要将文件按照chunkSize切块，然后Index从0到块数减1就可以了。
如此，只需要HashMap只需要记录切了几块，然后从第0块写到最后一块，读的时候也从0块读到最后一块即可。
/* Definition of BaseGFSClient
 * class BaseGFSClient {
 *     private Map<String, String> chunk_list;
 *     public BaseGFSClient() {}
 *     public String readChunk(String filename, int chunkIndex) {
 *         // Read a chunk from GFS
 *     }
 *     public void writeChunk(String filename, int chunkIndex,
 *                            String content) {
 *         // Write a chunk to GFS
 *     }
 * }
 */
public class GFSClient extends BaseGFSClient {
    Map<String, Integer> name2Index;
    private int chunkSize;

    public GFSClient(int chunkSize) {
        this.chunkSize = chunkSize;
        name2Index = new HashMap<>();
    }
   
    // @param filename a file name
    // @return conetent of the file given from GFS
    public String read(String filename) {
        if (!name2Index.containsKey(filename)) {
            return null;
        }
       
        String ret = "";
        for (int i = 0; i < name2Index.get(filename); i++) {
            ret += readChunk(filename, i);
        }
       
        return ret;
       
    }

    // @param filename a file name
    // @param content a string
    // @return void
    public void write(String filename, String content) {
        int chunkNum = content.length() / chunkSize;
        chunkNum += (content.length() % chunkSize == 0) ? 0 : 1;
       
        name2Index.put(filename, chunkNum);
       
        for (int i = 0; i < chunkNum; i++) {
            int start = i * chunkSize;
            int end = Math.min(content.length(), (i + 1) * chunkSize);
            writeChunk(filename, i, content.substring(start, end));
        }
    }

Heart Beat

https://github.com/zxqiu/leetcode-lintcode/blob/master/system%20design/Heart_Beat.java
Heart Beat
In the Master-Slave architecture, slave server will ping master in every k seconds to tell master server he is alive. If a master server didn't receive any ping request from a slave server in 2 * k seconds, the master will trigger an alarm (for example send an email) to administrator.
Let's mock the master server, you need to implement the following three methods:
    initialize(slaves_ip_list, k). salves_ip_list is a list of slaves' ip addresses. k is define above.
    ping(timestamp, slave_ip). This method will be called every time master received a ping request from one of the slave server. timestamp is the current timestamp in seconds. slave_ip is the ip address of the slave server who pinged master.
    getDiedSlaves(timestamp). This method will be called periodically (it's not guaranteed how long between two calls). timestamp is the current timestamp in seconds, and you need to return a list of slaves' ip addresses that died. Return an empty list if no died slaves found.
You can assume that when the master started, the timestamp is 0, and every method will be called with an global increasing timestamp.
 Example
initialize(["10.173.0.2", "10.173.0.3"], 10)
ping(1, "10.173.0.2")
getDiedSlaves(20)
>> ["10.173.0.3"]
getDiedSlaves(21)
>> ["10.173.0.2", "10.173.0.3"]
ping(22, "10.173.0.2")
ping(23, "10.173.0.3")
getDiedSlaves(24)
>> []
getDiedSlaves(42)
>> ["10.173.0.2"]
解：
用一个HashMap维护每个slave和对应的最后一次出现的timestamp。
当有新的ping时，更新这个map。
当取dead slave时，对比当前timestamp和map中每个slave的timestamp，找出超时的即可。
    Map<String, Integer> slaves;
    int interval;

    public HeartBeat() {
        slaves = new HashMap<>();
    }

    // @param slaves_ip_list a list of slaves'ip addresses
    // @param k an integer
    // @return void
    public void initialize(List<String> slaves_ip_list, int k) {
        if (slaves_ip_list == null) {
            return;
        }
       
        for (String ip : slaves_ip_list) {
            slaves.put(ip, 0);
        }
        interval = k;
    }

    // @param timestamp current timestamp in seconds
    // @param slave_ip the ip address of the slave server
    // @return nothing
    public void ping(int timestamp, String slave_ip) {
        if (!slaves.containsKey(slave_ip)) {
            return;
        }
       
        slaves.put(slave_ip, timestamp);
    }

    // @param timestamp current timestamp in seconds
    // @return a list of slaves'ip addresses that died
    public List<String> getDiedSlaves(int timestamp) {
        List<String> ret = new ArrayList<>();
       
        for (Map.Entry entry : slaves.entrySet()) {
            if (timestamp - (Integer)entry.getValue() >= 2 * interval) {
                ret.add((String)entry.getKey());
            }
        }
       
        return ret;
    }

Company Airbnb

http://techstacks.io/airbnb
- Ruby, Rails

https://medium.com/airbnb-engineering
http://airbnb.io/events/tech-talks/

https://www.slideshare.net/RohanKhude/technology-stack-behind-airbnb
Airbnb is using Elastic Load Balancing, which automatically distributes incoming traffic between multiple Amazon EC2 instances.

To easily process and analyze 50 Gigabytes of data daily, Airbnb uses Amazon Elastic MapReduce (Amazon EMR)

Airbnb moved its main MySQL database to Amazon Relational Database Service.
Airbnb chose Amazon RDS because it simplifies much of the time-consuming administrative tasks typically associated with databases

Syntactically Awsome Style Sheet
Sass is an extension of CSS3, adding nested rules, variables, mixins, selector inheritance, and more.

Sendgrid – Transactional Email  Delivering your transactional and marketing email through one reliable platform.

MixPanel – Funnel Analysis Analytics • It tracks user interactions with web and mobile applications • Funnel analysis helps you identify where people drop off so you can increase your conversion rates • The funnel analyses "are an effective way to calculate conversion rates on specific user behaviors'

Amazon Route 53 – DNS Management  DNS data is typically deployed on multiple physical servers.  The main purposes of DNS management software are: to reduce human error when editing complex and repetitive DNS data.  Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service.

BrainTree – Payment Services  Subsidiary of Paypal

Twilio – Voice and SMS • Twilio offers developers a API for phone services to make and receive phone calls, and send and receive text. • Airbnb uses Twilio to build Voice Connect click to call, bridging the communication gap between guests and would be hosts.

Nexmo – Voice and SMS  API for SMS, Voice and Phone Verification.
Urban Airship - Mobile Push Messaging Improve customer engagement with push notifications, location- based marketing and analytics.
The push notifications services enables international users to receive notifications regardless of their location or carrier.  Users then receive push notifications for both new messages received on the platform as well as new reservations, delivered directly to their mobile device

Visual Website Optimizer - A/B Testing Analytics • VWO is the easiest A/B, Split and Multivariate testing tool. • To optimizing your website for increased conversion rate and sales. • A/B testing (sometimes called split testing) is comparing two versions of a web page to see which one performs better. • You compare two web pages by showing the two variants (let's call them A and B) to similar visitors at the same time. The one that gives a better conversion rate, wins!

Aerosolve - A machine learning package built for humans. Created by airbnb Airbnb's Price Tips feature helps users figure out what to charge by using machine learning. n this dynamic pricing feature, we show hosts the probability of getting a booking (green for a higher chance, red for a lower chance), or predicted demand, and allow them to easily price their listings dynamically with a click of a button.

Fig. Learning to rank images. On the left, image ordering trained from professional photographer ratings. On the right, image ordering trained from organic books, clicks and impressions Aerosolve - Image Analysis Algorithms

Scout – Performance Monitoring  Scout is a simple hosted server monitoring service.  Rather than installing and configuring monitoring scripts on servers  Scout features a point-and-click plugin setup  Scout currently monitors the servers of AirBnB and The New York Times.

Sentry - Exception Monitoring  Real-time crash reporting for your web apps, mobile apps, and games.  Sentry notifies you when your users experience errors in your web and mobile apps.

New Relic – Performance Monitor SaaS Application Performance Management for Ruby, PHP, .Net, Java, Python, and Node.js Apps. A tool for building and distributing development environments

Chef - Server Configuration and Automation  Chef enables you to manage and scale cloud infrastructure with no downtime or interruptions.  Freely move applications and configurations from one cloud to another.  Chef is integrated with all major cloud providers including Amazon EC2, VMWare, IBM Smartcloud, Rackspace, OpenStack, Windows Azure, HP Cloud, Google Compute Engine, Joyent Cloud and others.

Mocha - Javascript Testing Framework Simple, flexible, fun javascript test framework for node.js & the browser. Mocha tests run serially, allowing for flexible and accurate reporting, while mapping uncaught exceptions to the correct test cases.

Logstash - Log Management  Logstash is a tool for managing events and logs.  You can use it to collect logs, parse them, and store them for later use.

kibana - Monitoring Tools Kibana is an open source, browser based analytics and search dashboard for Elasticsearch. Airbnb - Collect logs with logstash and visualize them with kibana.
https://fossbytes.com/top-airbnb-open-source-projects-you-must-know/
https://prestodb.io/

Airflow is use to create workflows as directed acyclic graphs (DAGs) of tasks. It is very similar to Apache Oozie for Hadoop. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

Aerosolve is a machine learning package/library that enables humans to partner with a machine in a symbiotic way exceeds the capabilities of humans or machines alone. This Airbnb open source project Aerosolves library is meant to be used with sparse, interpretable features such as those that commonly occur in search (search keywords, filters) or pricing (number of rooms, location, price). 

Synapse is Airbnb’s new system for service discovery. Synapse solves the problem of automated fail-over in the cloud, where failover via network re-configuration is impossible. The end result is the ability to connect internal services together in a scalable, fault-tolerant way.

Polyglot.js is a tiny I18n helper library written in JavaScript, made to work both in the browser and in CommonJS environments (Node). It provides a simple solution for interpolation and pluralization, based off of Airbnb’s experience adding I18n functionality to its Backbone.jsand Node apps. Polyglot has zero dependencies.

Nerve is a utility for tracking the status of machines and services. It runs locally on the boxes which make up a distributed system, and reports state information to a distributed key-value store.  The combination of Nerve and Synapse make service discovery in the cloud easy!

https://medium.com/airbnb-engineering/deeplinkdispatch-778bc2fd54b7
https://github.com/airbnb/DeepLinkDispatch

Deep links provide a way to link to specific content on either a website or an application. These links are indexable and searchable, and can provide users direct access to much more relevant information than a typical home page or screen. In the mobile context, the links are URIs that link to specific locations in the application.

At Airbnb, we use these deep links frequently to link to listings, reservations, or search queries. For example, a typical deep link to a listing may look something like this:

airbnb://rooms/8357

This deep link directly bypasses the home screen in the application and opens listing information for the Mushroom Dome cabin.

Android supports deep links through declaration in the Manifest. You can add an intent filters which define a mapping between deep link schemas and Activities. Subsequently, any URI with the registered scheme, host, and path will open up that Activity in the app.

While convenient for simple deep link usage, this traditional method becomes burdensome for more complicated applications. For example, you could use the intent filter to specify the path pattern, but it’s somewhat limiting. You can’t easily indicate the parameters that you would expect in the URI that you are filtering for. For complex deep links, you are likely to have to write a parsing mechanism to extract out the parameters, or worse, have such similar code distributed amongst many Activities.

DeepLinkDispatch is designed to help developers handle deep links easily without having to write a lot of boilerplate code and allows you to supply more complicated parsing logic for deciding what to do with a deep link. You can simply annotate the Activity with a URI in a similar way to other libraries. Looking at the example deep link URI from above, you could annotate an activity like so, and declare an “id” parameter that you want the application to parse:

@DeepLink(“rooms/{id}”)
public class SomeActivity extends Activity {

After annotating a particular Activity with the deep link URI that the activity should handle, DeepLinkDispatch will route the deep link automatically and parse the parameters from the URI. You can then determine whether the intent was fired by a deep link and extract out the parameters declared in the annotation. Here’s an example:

if (getIntent().getBooleanExtra(DeepLink.IS_DEEP_LINK, false)) { 
  Bundle parameters = getIntent().getExtras();
  String someParameter = parameters.getString("id");
  ...
}

At its core, DeepLinkDispatch generates a simple Java class to act as a registry of what Activities are registered with which URIs, and what parameters should be extracted. DeepLinkDispatch also generates a shim Activity which tries to match any deep link with an entry in the registry– if it finds a match, it will extract the parameters and start the appropriate Activity with an Intent populated with the parameters.

Additionally, DeepLinkDispatch is intended to provide greater insight into deep link usage. Android by default does not give much insight into what deep links are being used nor what deep links are failing. DeepLinkDispatch provides callbacks in the Application class for any deep link call, either successful or unsuccessful, allowing developers to track and correct any problematic links firing at the application.

public class SomeApp extends App implements DeepLinkCallback {
  @Override public void onSuccess(String uri) {
    // Handle or track a successful deep link here
  }

  @Override public void onError(DeepLinkError error) {
    // Handle or track and error here
  }
}

Security Breach

http://coolshell.cn/articles/17607.html

安全问题从来都是需要多方面一起努力，但是安全问题最大的短板就是在用户这边。这次的这个事，说白了，就是用户没有给MongoDB设置上用户名和口令，然后还把服务公开到了公网上。

那么，怎么会有这么多的对外暴露的MongoDB？看了一下Shodan的报告，发现主要还是来自公有云平台，Amazon，Alibaba，Digital Ocean，OVH，Azure 的云平台上有很多这样的服务。不过，像AWS这样的云平台，有很完善的默认安全组设置和VPC是可以不把这样的后端服务暴露到公有云上的，为什么还会有那么多？

这么大量的暴露在公网上的服务是怎么回事？有人发现（参看这篇文章《It’s the Data, Stupid!》 ），MongoDB历史上一直都是把侦听端口绑在所有的IP上的，这个问题在5年前（2011年11月）就报给了MongoDB (SERVER-4216)，结果2014年4月才解决掉。所以，他觉得可能似乎 MongoDB的 2.6之前的版本都会默认上侦听在0.0.0.0 。

1）一是技术人员下载了mongod的软包，一般来说，mongodb的压缩包只有binary文件 ，没有配置文件 ，所以直接解开后运行，结果就没有安全认证，也绑在了公网上。也许，MongoDB这么做的原因就是为了可以快速上手，不要在环境上花太多的时间，这个有助于软件方面的推广。但是，这样可能就坑了更多的人。

2）因为MongoDB是后端基础服务，所以，需要很多内部机器防问，按道理呢，应该绑定在内网IP上，但是呢，可能是技术人员不小心，绑在了0.0.0.0的IP上。

关于公网的IP。一般来说，公有云平台上的虚拟主机都会有一个公网的IP地址，老实说，这并不是一个好的方法，因为有很多主机是不需要暴露到公网上的，所以，也就不需要使用公网IP，于是，就会出现弹性IP或虚拟路由器以及VPC这样的虚拟网络服务，这样用户在公有云就可以很容易的组网，也就没有必要每台机器都需要一个公网IP，使用云平台，最好还是使用组网方案比较好的平台。

关于安全组。在AWS上，你开一台EC2，会有一个非常严格的安全组——只暴露22端口，其它的全部对外网关闭。这样做，其实是可以帮用户防止一下不小心把不必要的服务Open到公网上。按道理来说，AWS上应该是帮用户防了这些的。但是，AWS上的MongoDB祼奔的机器数量是最多的，估计和AWS的EC2的 基数有关系吧（据说AWS有千万台左右的EC2了）

Google Tips Misc

https://support.google.com/assistant/thread/1841677

Try the following

https://techcrunch.com/2017/07/14/google-play-music-gets-more-personalized-with-new-release-radio-customized-to-your-tastes/

Go to

https://assistant.google.com/services/a/uid/0000004d5731f294

Scroll to bottom of web page and select

Stop Weather from remembering me

Google Voice
https://www.digitaltrends.com/mobile/how-to-record-calls-iphone/
Surprisingly, Google Voice, in addition to helping you filter out unwanted calls, will record incoming calls for the stellar price of zero dollars. The only setbacks are that Google doesn’t allow you to record outgoing calls — only incoming ones — and you have to port your phone number over to Google to get access to the recording feature. This makes it rather inconvenient if you’re hoping to record any conversations that you need to initiate, or if you like your current carrier.

Google Keeps
https://www.computerworld.com/article/3064305/22-top-tips-for-google-keep-on-android.html
4. Archive aggressively

8. Collect your spoken thoughts

Take a note on the go by speaking it into Keep and letting the app transcribe your words. You can start recording in one of three ways: tap the microphone icon in Keep's bottom toolbar, tap the plus icon in the lower-left corner and then select "Recording" while editing a note, or tap the microphone icon in Keep's home screen widget. Then, just yammer away. When you're finished, Keep will place its transcription in the body of the note and also attach an audio recording of your voice for good measure (and/or mortification — yes, that really is what you sound like).

10. Transform your scribbles into words

https://www.blog.google/products/g-suite/8-tips-help-you-keep-google-keep/
1. Record voice notes.

2. Transcribe notes from pictures.

3. Create drawings and even search handwritten notes.

4. Drag and drop notes from Keep into Google Docs.

5. Use the Chrome Extension.

Create notes while you browse the web by downloading the Chrome Extension. One cool thing is that when you create a note using the extension, it saves the site URL with it. So if you browse back to that same URL, the extension will show your note in context.

6. Send notes from Keep to other apps you use.

7. Color-code or label your notes to find them quicker.

8. Set reminders for yourself.

https://zapier.com/apps/google-tasks/tutorials/google-tasks-guide
https://zapier.com/

Google Contacts
Sync Google Contacts in iPhone

• Add your Gmail account at Settings -> Passwords & Accounts
• Change to your Gmail account at Settings -> Contacts -> Default Account
• So now every time you create an account at Phone app in iPhone, it will be synced to Google Contacts.

Blogger
https://deepfieldinc.com/meta-tag-generator/
https://makingamark.blogspot.com/2012/05/how-to-create-search-description-for.html

Write a description that would both inform and interest users if they saw your description meta tag as a snippet in a search result.

Avoid:

• writing a description meta tag that has no relation to the content on the page
• using generic descriptions like "This is a web page" or "Page about baseball cards"
• filling the description with only keywords
• copying and pasting the entire content of the document into the description meta tag

and

Use unique descriptions for each page

Having a different description meta tag for each page helps both users and Google, especially in searches where users may bring up multiple pages on your domain (e.g. searches using the site:operator). If your site has thousands or even millions of pages, hand-crafting description meta tags probably isn't feasible. In this case, you could automatically generate description meta tags based on

Avoid:using a single description meta tag across all of your site's pages or a large group of pages

https://deepfieldinc.com/meta-tag-generator/

https://www.techprevue.com/post-search-description-blogger-seo/
It appears when you share a post on Facebook, Twitter, Linkedin or anywhere
2. The blog will get more organic search traffic.

3. More organic traffic means more readers.

4. More readers mean higher earnings from AdSense or other monetization programs.
https://www.techprevue.com/custom-permalink-for-better-seo-blogger/

https://www.stramaxon.com/2012/03/show-more-than-200-comments-on-blogger.html
The problem in this is that comments location is set to Embedded by default and Embedded comments only shows 200 comments
http://waltz.blogspot.com/2007/07/search-multiple-labels-in-blogger-feeds.html

To find posts labeled with all of multiple labels:

http://<subdomain>.blogspot.com/feeds/posts/default/-/<label1>/<label2>/.../<labelN>

In other words, the blogspot feed URL, followed by /- to indicate the beginning of a list of labels, followed by the slash-separated label names.

For example, to find all posts on this blog labeled both dance and fashion:

http://waltz.blogspot.com/feeds/posts/default/-/dance/fashion


Negations (exclude a label) and disjunctions (or) are not supported yet:

http://waltz.blogspot.com/feeds/posts/default/-/dance/-fashion

http://waltz.blogspot.com/feeds/posts/default/-/dance%7Cfashion

https://www.peggyktc.com/2018/03/blogger-custom-page-not-found.html

How create a custom "Page Not Found" message for your own Blogger blog.

1. Create your error page as a Draft Blogger Blog post
2. Copy the post HTML 
3. Open your blog's Settings > Search Preferences
4. Under "Errors and Redirections" click the option to "Edit" Custom Page Not Found 
5. Paste in the HTML of your Custom Error Page and click "Save Changes"

Remove the automatically added code: Dir=“Ltr” Trbidi=“On”

• Disable this by turn off “Enable transliteration” in Language and formatting settings
• If “Enable transliteration” is on, blogger will automatically add this code whenever it detects it’s not there
• if you have custom javascript code in your post, this behavior will break the js code.

• Make a Sticky Post in Blogger
  • Publish it first then change date to a future date
• Multiple Labels Search
  • and query: search/label/LABEL1+LABEL2
  • Or Query: search/?q=label:LABEL1|label:LABEL2
• Generate Sitemap html page including all labels
• Generate Sitemap html page including (multiple) specific labels
• Add more Labels to an Existing Post
• Bulk remove labels
  • Just click the label to be removed at the label action

https://feed.mikle.com/support/google-blogger-rss/

Full site feed:

• Atom 1.0: https://blogname.blogspot.com/feeds/posts/default
• RSS 2.0: https://blogname.blogspot.com/feeds/posts/default?alt=rss

Comments-only feed:

• Atom 1.0: https://blogname.blogspot.com/feeds/comments/default
• RSS 2.0: https://blogname.blogspot.com/feeds/comments/default?alt=rss

Label-specific site feed:

To get a feed for a specific label, change [label].

• Atom 1.0: https://blogname.blogspot.com/feeds/posts/default/-/[label]
• RSS 2.0: https://blogname.blogspot.com/feeds/posts/default/-/[label]?alt=rss

https://www.exeideas.com/2016/02/parameters-in-blogspot-feed.html

1. Using your blogger’s address : http://yourblogname.blogspot.com/feed/post/default
2. Using your blogid : http://www.blogger.com/feeds/blogID/posts/default

Meaning – Category query filter
Specifies categories (also known as labels) to filter the feed results.
http://googleblog.blogspot.com/feeds/posts/default/-/ads
http://www.blogger.com/feeds/10861780/posts/default/-/ads

orderby

Meaning – The order in which to return entries, such as lastmodified (the default), starttime, or updated.
http://googleblog.blogspot.com/feeds/posts/default?orderby=updated
http://www.blogger.com/feeds/10861780/posts/default?orderby=updated
I do not know how to using lastmodified and starttime!

orderBy

string

The sort order applied to the search results. Acceptable values are: "published": Order by the date the post was published "updated": Order by the date the post was last updated

Sitemap
https://www.howbloggerz.com/2017/03/generate-html-sitemap-page-blogger.html
Note: This is Label Based Sitemap Page and you can only apply label based themes on it.

https://blogging.nitecruzr.net/2014/11/blogs-to-have-automatically-generated.html

https://mybloggeraide.blogspot.com/2017/07/create-html-sitemap-for-specific-label.html
Create HTML Sitemap for Specific Label in Blogger

http://9jaonpointed.blogspot.com/2014/11/how-to-create-sitemap-page-for.html

https://www.wikihow.com/Make-a-Sticky-Post-in-Blogger
Changing the Date

Enter a future date. Your post will be on sticky till this date.

Using the Gadgets

https://www.labnol.org/internet/submit-blogger-sitemap-to-google/10149/
Generate a Complete Sitemap for your Blogger Blog

https://axlmulat.com/blogger/how-to-create-sitemap-page-in-blogger-blog-step-by-step/

https://www.lifewire.com/sync-calendar-to-google-assistant-4159478

• "Hey Google, what's my agenda for today?"

• "Hey Google, add a doctor appointment to my calendar."
• "Okay Google, schedule a concert for me on Friday at 7 pm."
• "Okay Google, add an event called Jenny's surprise party."

When is my doctor appointment?
Chrome
https://support.google.com/chrome/forum/AAAAP1KN0B0s3v4yG1uAcY/?hl=en&gpf=%23!msg%2Fchrome%2Fs3v4yG1uAcY%2FJ7UPRWHfAgAJ&msgid=J7UPRWHfAgAJ

This ultimately worked for me: simply white-list (or add/allow) "accounts.google.com" as a cookie through settings, chrome://settings/content/cookies

GBoard
Glide typing
https://www.hongkiat.com/blog/new-gboard-features/

• Simply tap on the "?123" button and swipe upward. The page will immediately change to the symbols page.
• Now leave the swipe over the symbol that you want to use and it will be entered, and you will be automatically switched back to the main keyboard.

https://www.reddit.com/r/gboard/comments/8m1kz9/swipe_to_delete_on_ios/

https://mashtips.com/best-google-keyboard-iphone-features/
Keyboard as a Trackpad and One-handed Mode
The features of Googlr Key Board is not just limited to search for related benefits. GBoard has aplenty of touch-based features too. You can add period and space to the previous sentence by simply double tapping the space bar.

https://www.iphoneincanada.ca/app-store/gboard-ios-one-handed-keyboard-ios-11/
How to toggle on the one-handed keyboard in iOS 11? Just tap and hold on the ‘globe’ icon, then tap the left or right keyboard indentation, and the keyboard should condense to the side you choose:

https://ios.gadgethacks.com/how-to/20-tips-help-you-master-gboard-for-iphone-0182301/
Use the Spacebar as a Trackpad
To get the cursor where you want, slide your finger back and forth on the spacebar. You can also use 3D Touch on the keyboard itself just like you can with Apple's keyboard, SwiftKey, and a few other options, but I like the spacebar option when only moving a little bit. With 3D Touch, I find that the cursor can move too fast or jump, so I save that for when I need to make big moves.

Search by Emoji Name

Say What You Want to Type

Say What You Want to Type

Google doesn't seem to like Apple's built-in dictation tool, but you can still use Google's own dictation tool, which opens another interface that dictates what you say, then pastes that text in where you want it. Just long-press on the spacebar to automatically activate the audio recording option in Gboard or Google, say what you want, then it will appear typed out back where you were.

Access Punctuation Quickly

So, there is a little-known secret in Apple's keyboard that lets you quickly add a period, question mark, number, or another non-letter character, but Gboard's implementation is better and easier to get used to.

Just tap and hold the period key (oh yeah, there's a freakin' period key!), then slide to pick your punctuation mark or character. For numbers, you'd do the same thing you do in Apple's version: tap the number key, then slide your finger over to the number you want without letting go of the screen.

Capitalize Words Faster
Along the same lines as the above tip, if you tap on the caps key then slide over to the letter you want to capitalize without letting go of the screen, then that letter will automatically be capitalized when you let go. This is useful when typing out acronyms and non-standard proper nouns that autocorrect can't figure out on its own. Of course, if you're typing out acronyms, you might just want to double-tap the caps key to enable caps lock.

Create Keyboard Shortcuts
There is no feature in Gboard that lets you actually create keyboard shortcuts, but all of the shortcut combinations you create in Settings –> General –> Keyboard –> Text Replacement will still work in Gboard.

It's not as smooth as using it in Apple's QuickType keyboard, but it's better than nothing. Sometimes when you hit the spacebar or punctuation after typing your shortcut combo out Gboard will automatically expand the text where you want it, but other times it just adds a suggestion for the expanded text in the top bar.


https://developers.google.com/blogger/docs/3.0/reference/posts/update

https://gist.github.com/LindaLawton/cff75182aac5fa42930a09f58b63a309
https://www.daimto.com/google-authentication-with-curl/
https://www.daimto.com/google-3-legged-oauth2-flow/

Now replace the values needed in the following link and put it in a web browser

https://accounts.google.com/o/oauth2/auth?client_id=[Application Client Id]&redirect_uri=urn:ietf:wg:oauth:2.0:oob&scope=[Scopes]&response_type=code

Exchanging Authentication code

You should get the standard request for authentication.   Once you have accepted authentication copy the Authentication code.   Take the following code replace the values as needed.

curl \
–request POST \
–data “code=[Authentcation code from authorization link]&client_id=[Application Client Id]&client_secret=[Application Client Secret]&redirect_uri=urn:ietf:wg:oauth:2.0:oob&grant_type=authorization_code” \
https://accounts.google.com/o/oauth2/token

https://adssettings.google.com/authenticated

https://blog.cometdocs.com/how-to-set-up-adobe-pdf-reader-to-open-at-last-page-viewed

Then choose the Document tab and check the option: “Restore last view setting when reopening documents.”

Simple Allow Copy
https://chrome.google.com/webstore/detail/simple-allow-copy/aefehdhdciieocakfobpaaolhipkcpgc/related?hl=en


https://superuser.com/questions/111675/google-chrome-auto-close-download-bar/325787

Easier keyboard shortcuts (Windows): Ctrl+J, Ctrl+W (open "Downloads" tab and close it).

For Mac, use ⌘ (Cmd)+Shift+J.

https://www.pcworld.com/article/3160936/browsers/10-frustrating-google-chrome-irritations-and-how-to-fix-them.html
https://www.makeuseof.com/tag/fullsearch-search-open-tabs/
FullSearch is a handy Chrome extension that makes managing your tab load a lot easier. All you need to do is set a keyboard shortcut to open the pop-up, and type in what you are looking for. It will scan the data contained in all of your open tabs to find what you are looking for.
https://superuser.com/questions/1035042/is-there-any-way-to-view-chrome-browser-notifications-history

If you have a Mac, there is a way! 😄

Here's how the notifications list would look like:

https://stackoverflow.com/questions/4819060/allow-google-chrome-to-use-xmlhttprequest-to-load-a-url-from-a-local-file

Using --disable-web-security switch is quite dangerous! Why disable security at all while you can just allow XMLHttpRequest to access files from other files using --allow-file-access-from-files switch?

Before using these commands be sure to end all running instances of Chrome.

On Windows:

chrome.exe --allow-file-access-from-files

On Mac:

open /Applications/Google\ Chrome.app/ --args --allow-file-access-from-files

You can also use Safari on a Mac. It allows AJAX to local files by default when the request is made from a local file.

And mention, this does exactly what you expect, it disables the web security, so be careful with it.

startup chrome with --disable-web-security

On Windows:

chrome.exe --disable-web-security

On Mac:

open /Applications/Google\ Chrome.app/ --args --disable-web-security

This will allow for cross-domain requests.

https://productforums.google.com/forum/#!topic/blogger/KDSHjgR7cvQ
Your HTML cannot be accepted: Tag is Broken: 3 HELP???

Try taking out the <3 from your title. That is a HTML special character

You can use code: &lt;3 to show heart like that <3 in title

http://blogging.nitecruzr.net/2013/09/confusion-from-message-you-have-logged.html
Confusion From The Message "You have logged out from another location."

Advice from Blogger Support staff, as occasionally provided, is reasonably simple.

If you are still receiving this error please clear cache and cookies, refreshing the page and log-in again.

https://business.tutsplus.com/tutorials/how-to-convert-powerpoint-to-google-slides--cms-28971
Just click open with Google Slides

https://exde601e.blogspot.com/2012/12/search-operators-for-Blogger-labels.html
While playing around on the blog I discovered an easier way to do this: modify the normal label URL in Blogger by adding a + sign followed by the name of the second label: <blog-URL>search/label/LABEL1+LABEL2. It’s easier to remember and to type, but it’s also case sensitive. It actually works with more than two labels – I only tested it with three, so I’m not sure if there is a limit to the number of labels you can add in the URL. I don’t use any labels with multiple words so I couldn’t test this case, but it’s safe to assume you need to escape spaces by replacing them with %20 just like Blogger does with the regular label pages.

But what about the other case, when you want to find posts with any of those labels – the OR operator instead of AND?
Unfortunately the second option doesn’t seem to support OR, but the first one does:
just replace the + in the search query with a vertical bar like this:
<blog-URL>/search/?q=label:LABEL1|label:LABEL2. 

http://www.davidsottimano.com/how-to-download-all-attachments-from-a-gmail-thread/
Step 1: Open the email thread with attachments
Step 2: Click on the top menu and select “Forward All” and forward it to yourself
Step 3: Open the forwarded email and at the bottom, you should have an option to Download all

https://www.slideshare.net/guest66ae43/how-to-add-power-point-presentations-to-blogger