Wednesday, October 28, 2015

[CareerCup] 10.5 Web Crawler 网络爬虫 - Grandyang - 博客园



[CareerCup] 10.5 Web Crawler 网络爬虫 - Grandyang - 博客园
10.5 If you were designing a web crawler, how would you avoid getting into infinite loops?

这道题问如果让我们设计一个网络爬虫,怎么样才能避免进入无限循环。那么何谓无限循环呢,如果我们将网络看做一个图Graph,无限循环就是当存在环Circle时可能发生的情况。当我们用BFS来进行搜索时,每当我们访问过一个网站,我们将其标记为已访问过,下次再遇到直接跳过。那么如何定义访问过呢,是根据其内容还是根据其URL链接呢,根据URL链接更能会有多个链接指向同一个网站的情况,根据内容可能某个网站会有随机生成内容的模块,所以一个比较好的解决方案是根据相似度来确定,即既包括内容又包括URL链接,下面我们来看具体如何实现:
1. 打开网页并根据特定的子模块和URL链接生成一个页面签名
2. 访问数据库看这个页面签名是否最近被访问过
3. 如果最近被访问过,将这个网页添加到数据库中低优先级的位置
4. 如果没有,则访问此网站并将连接加入数据库
如果是对于一个小型系统,比如局域网,我们可以对每个页面设一个让页面接受访问的最小优先级。

Read full article from [CareerCup] 10.5 Web Crawler 网络爬虫 - Grandyang - 博客园

Labels

Review (572) System Design (334) System Design - Review (198) Java (189) Coding (75) Interview-System Design (65) Interview (63) Book Notes (59) Coding - Review (59) to-do (45) Linux (43) Knowledge (39) Interview-Java (35) Knowledge - Review (32) Database (31) Design Patterns (31) Big Data (29) Product Architecture (28) MultiThread (27) Soft Skills (27) Concurrency (26) Cracking Code Interview (26) Miscs (25) Distributed (24) OOD Design (24) Google (23) Career (22) Interview - Review (21) Java - Code (21) Operating System (21) Interview Q&A (20) System Design - Practice (20) Tips (19) Algorithm (17) Company - Facebook (17) Security (17) How to Ace Interview (16) Brain Teaser (14) Linux - Shell (14) Redis (14) Testing (14) Tools (14) Code Quality (13) Search (13) Spark (13) Spring (13) Company - LinkedIn (12) How to (12) Interview-Database (12) Interview-Operating System (12) Solr (12) Architecture Principles (11) Resource (10) Amazon (9) Cache (9) Git (9) Interview - MultiThread (9) Scalability (9) Trouble Shooting (9) Web Dev (9) Architecture Model (8) Better Programmer (8) Cassandra (8) Company - Uber (8) Java67 (8) Math (8) OO Design principles (8) SOLID (8) Design (7) Interview Corner (7) JVM (7) Java Basics (7) Kafka (7) Mac (7) Machine Learning (7) NoSQL (7) C++ (6) Chrome (6) File System (6) Highscalability (6) How to Better (6) Network (6) Restful (6) CareerCup (5) Code Review (5) Hash (5) How to Interview (5) JDK Source Code (5) JavaScript (5) Leetcode (5) Must Known (5) Python (5)

Popular Posts