Monday, July 10, 2017

GFS Client



https://github.com/zxqiu/leetcode-lintcode/blob/master/system%20design/GFS_Client.java
Implement a simple client for GFS (Google File System, a distributed file system), it provides the following methods:
    read(filename). Read the file with given filename from GFS.
    write(filename, content). Write a file with given filename & content to GFS.
There are two private methods that already implemented in the base class:
    readChunk(filename, chunkIndex). Read a chunk from GFS.
    writeChunk(filename, chunkIndex, chunkData). Write a chunk to GFS.
To simplify this question, we can assume that the chunk size is chunkSize bytes. (In a real world system, it is 64M). The GFS Client's job is splitting a file into multiple chunks (if need) and save to the remote GFS server. chunkSize will be given in the constructor. You need to call these two private methods to implement read & write methods.
 Example
GFSClient(5)
read("a.txt")
>> null
write("a.txt", "World")
>> You don't need to return anything, but you need to call writeChunk("a.txt", 0, "World") to write a 5 bytes chunk to GFS.
read("a.txt")
>> "World"
write("b.txt", "111112222233")
>> You need to save "11111" at chink 0, "22222" at chunk 1, "33" at chunk 2.
write("b.txt", "aaaaabbbbb")
read("b.txt")
>> "aaaaabbbbb"
解:
由于没有提供Master Server,需要Client自己记录每个文件名对应的所有Chunk Server。
分析Example,发现Chunk Server Index只需要将文件按照chunkSize切块,然后Index从0到块数减1就可以了。
如此,只需要HashMap只需要记录切了几块,然后从第0块写到最后一块,读的时候也从0块读到最后一块即可。
/* Definition of BaseGFSClient
 * class BaseGFSClient {
 *     private Map<String, String> chunk_list;
 *     public BaseGFSClient() {}
 *     public String readChunk(String filename, int chunkIndex) {
 *         // Read a chunk from GFS
 *     }
 *     public void writeChunk(String filename, int chunkIndex,
 *                            String content) {
 *         // Write a chunk to GFS
 *     }
 * }
 */
public class GFSClient extends BaseGFSClient {
    Map<String, Integer> name2Index;
    private int chunkSize;

    public GFSClient(int chunkSize) {
        this.chunkSize = chunkSize;
        name2Index = new HashMap<>();
    }
   
    // @param filename a file name
    // @return conetent of the file given from GFS
    public String read(String filename) {
        if (!name2Index.containsKey(filename)) {
            return null;
        }
       
        String ret = "";
        for (int i = 0; i < name2Index.get(filename); i++) {
            ret += readChunk(filename, i);
        }
       
        return ret;
       
    }

    // @param filename a file name
    // @param content a string
    // @return void
    public void write(String filename, String content) {
        int chunkNum = content.length() / chunkSize;
        chunkNum += (content.length() % chunkSize == 0) ? 0 : 1;
       
        name2Index.put(filename, chunkNum);
       
        for (int i = 0; i < chunkNum; i++) {
            int start = i * chunkSize;
            int end = Math.min(content.length(), (i + 1) * chunkSize);
            writeChunk(filename, i, content.substring(start, end));
        }
    }

Labels

Review (572) System Design (334) System Design - Review (198) Java (189) Coding (75) Interview-System Design (65) Interview (63) Book Notes (59) Coding - Review (59) to-do (45) Linux (43) Knowledge (39) Interview-Java (35) Knowledge - Review (32) Database (31) Design Patterns (31) Big Data (29) Product Architecture (28) MultiThread (27) Soft Skills (27) Concurrency (26) Cracking Code Interview (26) Miscs (25) Distributed (24) OOD Design (24) Google (23) Career (22) Interview - Review (21) Java - Code (21) Operating System (21) Interview Q&A (20) System Design - Practice (20) Tips (19) Algorithm (17) Company - Facebook (17) Security (17) How to Ace Interview (16) Brain Teaser (14) Linux - Shell (14) Redis (14) Testing (14) Tools (14) Code Quality (13) Search (13) Spark (13) Spring (13) Company - LinkedIn (12) How to (12) Interview-Database (12) Interview-Operating System (12) Solr (12) Architecture Principles (11) Resource (10) Amazon (9) Cache (9) Git (9) Interview - MultiThread (9) Scalability (9) Trouble Shooting (9) Web Dev (9) Architecture Model (8) Better Programmer (8) Cassandra (8) Company - Uber (8) Java67 (8) Math (8) OO Design principles (8) SOLID (8) Design (7) Interview Corner (7) JVM (7) Java Basics (7) Kafka (7) Mac (7) Machine Learning (7) NoSQL (7) C++ (6) Chrome (6) File System (6) Highscalability (6) How to Better (6) Network (6) Restful (6) CareerCup (5) Code Review (5) Hash (5) How to Interview (5) JDK Source Code (5) JavaScript (5) Leetcode (5) Must Known (5) Python (5)

Popular Posts