1.项目背景

1.1 云存储概述

云存储是一种网上在线存储(Cloud storage)，即把数据存放在通常由第三方托管的多台虚拟服务器，而非专属的服务器上。客户便可自行使用此存储资源池来存放文件或对象。

实际上，这些资源可能被分布在众多的服务器主机上。

云存储的类型

  1. 个人云存储

  2. 私有云存储

  3. 公有云存储

  4. 混合云存储

他们都在使用云存储

为什么要使用云存储

  1. 数据转移方便
  2. 高可靠性
  3. 高安全性
  4. 可扩展性

云存储已经成为未来存储发展的一种趋势。

随着5G网络推广、大数据时代下用户体验的大幅提升、云存储市场前景广阔。

1.2 项目基础架构

1.3 项目预备环境

  1. JDK、Java IDE
  2. MySQL&MySQL Client
  3. Tomcat
  4. Linux

1.4 项目收获

通过本次项目，你将会有如下收获

  1. 了解到云存储相关概念
  2. 掌握企业中文件的常规操作
  3. 掌握企业中文件的秒传方法
  4. 掌握企业中文件的分块上传与断点续传
  5. 构建企业级分布式文件系统
  6. 了解微服务相关概念
  7. 巩固Java基础

2.项目实战

2.1 简单文件上传服务

接口列表

接口描述 接口URL
文件上传接口 POST /file/upload
文件下载接口 GET /file/download
文件查询接口 GET /file/query
文件删除接口 POST /file/delete

文件表设计

1
2
3
4
5
6
7
8
9
10
11
12
13
CREATE TABLE `t_file` (
  `id` INT NOT NULL AUTO_INCREMENT,
  `file_name` VARCHAR(256) NOT NULL COMMENT '文件名',
  `file_sha1` VARCHAR(40) NOT NULL COMMENT '文件hash',
  `file_size` INT NOT NULL COMMENT '文件大小',
  `file_addr` VARCHAR(1024) NOT NULL COMMENT '文件存储位置',
  `create_date` DATETIME NOT NULL DEFAULT now() COMMENT '文件创建时间',
  `update_date` DATETIME NULL COMMENT '文件更新时间',
  `status` INT NOT NULL DEFAULT 0 COMMENT '文件状态(可用/禁用/已删除等状态)',
  `ext1` INT NULL COMMENT '备用字段1',
  `ext2` VARCHAR(45) NULL COMMENT '备用字段2',
  PRIMARY KEY (`id`),
  UNIQUE INDEX `index2` (`file_sha1` ASC));

2.2 用户管理服务

接口列表

接口描述 接口URL
用户查询接口 /user/query
用户创建接口 /user/create
用户编辑接口 /user/update
用户删除接口 /user/${id}/delete
登录验证接口 /user/login
密码重置接口 /user/pwd/reset

用户表设计

1
2
3
4
5
6
7
8
9
10
11
12
CREATE TABLE `baidu_cloud`.`t_user` (
  `id` INT NOT NULL AUTO_INCREMENT,
  `user_account` VARCHAR(45) NOT NULL,
  `user_password` VARCHAR(32) NOT NULL,
  `user_email` VARCHAR(45) NULL DEFAULT 'none',
  `user_phone` VARCHAR(45) NULL DEFAULT 'none',
  `status` INT NOT NULL DEFAULT 0,
  `create_date` DATETIME NOT NULL DEFAULT now(),
  `update_date` DATETIME NULL,
  `ext1` INT NULL DEFAULT 0,
  `ext2` VARCHAR(45) NULL DEFAULT 'none',
  PRIMARY KEY (`id`));

2.3 用户文件关联

用户文件表设计

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CREATE TABLE `baidu_cloud`.`t_user_file` (
  `user_id` INT NOT NULL,
  `file_id` INT NOT NULL,
  INDEX `fk_user_id_idx` (`user_id` ASC),
  INDEX `fk_file_id_idx` (`file_id` ASC),
  CONSTRAINT `fk_user_id`
    FOREIGN KEY (`user_id`)
    REFERENCES `baidu_cloud`.`t_user` (`id`)
    ON DELETE NO ACTION
    ON UPDATE NO ACTION,
  CONSTRAINT `fk_file_id`
    FOREIGN KEY (`file_id`)
    REFERENCES `baidu_cloud`.`t_file` (`id`)
    ON DELETE NO ACTION
    ON UPDATE NO ACTION);

用户与文件关系设计概要

2.4 文件秒传

原理概述

未命名文件

2.5 文件分片上传

原理概述

未命名文件

2.6 断点续传

Http请求头Range

http请求头中的Range属性用于请求资源的部分内容，单位是byte，从0开始

如果服务器能够正常响应的话，服务器会返回 206 Partial Content 的状态码及说明

如果不能处理这种Range的话，就会返回整个资源以及响应状态码为 200 OK

例如

请求

1
2
3
4
GET  /test.rar  HTTP/1.1 
Connection:  close 
Host:  116.1.219.219 
Range:  bytes=0-499

表示请求test.rar的头500个字节：bytes=0-499

响应

1
2
3
4
5
6
HTTP/1.1 206 OK 
Content-Length: 801      
Content-Type: application/octet-stream;charset=UTF-8
Content-Range: bytes 0-100/2350
Last-Modified: Mon, 16 Feb 2009 16:10:12 GMT
Accept-Ranges: bytes

响应头说明

1
Content-Range: bytes 0-10/3103

表示，服务器响应了前(0-10)个字节的数据，该资源一共有(3103)个字节大小

1
Content-Length: 801

表示这次服务器响应了11个字节的数据（0-10）

简单实现代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
File file = new File("C:\\Users\\MAX\\Desktop\\CentOS-7-x86_64-Minimal-1511.iso");
        FileInputStream fis = new FileInputStream(file);
        BufferedInputStream bis = new BufferedInputStream(fis);
        long pos = 0;
        long fileSize = file.length();

        if(request.getHeader("Range") != null) {
            response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
            System.out.println(request.getHeader("Range"));
            pos =  Long.parseLong(request.getHeader("Range")
                        .replaceAll("bytes=", "")
                        .split("-")[0]);
        }

        response.setContentType("application/octet-stream;charset=UTF-8");
        response.setHeader("Accept-Ranges", "bytes");
        response.setHeader("Content-Length", fileSize + "");
        response.setHeader("Content-Disposition", "attachment;filename=" + file.getName());
        response.setHeader("Content-Range", "bytes  " + pos + "-" + (file.length() - 1) + "/" + file.length());

        ServletOutputStream out = response.getOutputStream();
        int len = 0;
        byte[] buffer = new byte[4096];
        bis.skip(pos);
        while((len = bis.read(buffer, 0, buffer.length)) != -1) {
            out.write(buffer, 0, len);
        }
        out.close();

redis分块上传实现

前端html代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<body>
	<center>
    	<p style="text-align: right">欢迎您,<a href="#">${loginUser.userName}</a></p>
    	<div>
            <table id="file-grid">
                <tr>
                    <th>编号</th>
                    <th>文件名</th>
                    <th>文件大小</th>
                    <th>操作</th>
                </tr>
            </table>
            <p>
                <input type="file" id="file"/><br>
                <input type="button" value="文件上传" id="fileUpload"/>
            </p>
            <p id="result"></p>
    	</div>
	</center>
</body>

前端js代码

1
2
3
<script src="${pageContext.request.contextPath}/js/jquery.min.js"></script>
<script src="${pageContext.request.contextPath}/js/core.js"></script>
<script src="${pageContext.request.contextPath}/js/sha1.js"></script>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
<script>
    var fileList = function () {
        $.getJSON("${pageContext.request.contextPath}/file/query",
            function (files) {
                //表格追加
                for (var i = 0; i < files.length; i++) {
                    var content = "<tr><td>" + files[i]["id"] + "</td>" +
                        "<td>" + files[i]["fileName"] + "</td>" +
                        "<td>" + (parseInt(files[i]["fileSize"]) / 1024).toFixed(2) + "kb</td>" +
                        "<td><a href='#'>更新</a>|<a href='#'>删除</a></td></tr>";
                    $("#file-grid").append(content);
                }
            }
        );
    }

    function sendFileSha1(file) {
        var reader = new FileReader();
        reader.onload = function (e) {
            var binary = e.target.result;
            var fileHash = CryptoJS.SHA1(CryptoJS.enc.Latin1.parse(binary)).toString();
            $.ajax({
                type: "post",
                url: "${pageContext.request.contextPath}/file/vhash",
                dataType: "json",
                data: {"fileHash": fileHash, "fileName": file.name, "fileSize": file.size},
                success: function (data) {
                    if(data["status"] == 0) {
                        //已存在相同文件
                        alert("上传完成");
                    }else if(data["status"] == 1) {
                        //从未上传
                        patchUpload(file, fileHash);
                    }else if(data["status"] == 2) {
                        //断点续传
                        var fileChunkNum = data["fileChunkNum"];
                        patchFileUpload(file, fileHash, 1024 * 1024 * 5, fileChunkNum);

                    }
                }
            });
        }

        reader.readAsBinaryString(file);
    }

    var patchFileUpload = function (file, hash, blockSize, fileChunks) {
        var i = parseInt(fileChunks.pop());
        //下一个文件块起始位置
        var nextSize = Math.min((i + 1) * blockSize, file.size);
        //截取文件块
        var fileData = file.slice(i * blockSize, nextSize);
        //将文件块存入FormData
        var formData = new FormData();
        //存储文件hash值
        formData.append("fileHash", hash);
        //存储文件名
        formData.append("fileName", file.name);
        //存储文件块编号
        formData.append("chunkNum", i);
        //存储文件块
        formData.append("fileData", fileData);
        //异步发送
        $.ajax({
            type: "post",
            url: "${pageContext.request.contextPath}/file/patchUpload",
            processData: false,
            contentType: false,
            data: formData,
            success: function (msg) {
                $("#result").text("已上传第" + (i + 1) + "个文件块");
                if (msg == "false" || nextSize >= file.size || fileChunks.length <= 0) {
                    return;
                }

                patchFileUpload(file, hash, blockSize, fileChunks);
            }
        });

    }

    //分片上传
    function patchUpload(file, hash) {
        //分片大小
        var blockSize = 1024 * 1024 * 5;
        //计算文件分片数
        var fileChunks = file.size % blockSize == 0 ? file.size / blockSize : parseInt(file.size / blockSize) + 1;
        //创建分片编号
        var fileChunkNum = Array();
        for (var i = 0; i < fileChunks; i++) {
            fileChunkNum.push(i);
        }
        //上传文件数据结构化
        var fileInfo = {
            "fileHash": hash,
            "fileName": file.name,
            "fileSize": file.size,
            "fileChunks": fileChunks,
            "fileChunkNum": fileChunkNum.toString()
        }
        //先保存文件整体信息，再进行分片上传
        $.ajax({
            type: "post",
            url: "${pageContext.request.contextPath}/file/info",
            data: fileInfo,
            success: function (msg) {
                if (msg = "true") {
                    //开始进行文件分片上传
                    patchFileUpload(file, hash, blockSize, fileChunkNum);
                } else {
                    alert("系统异常，请联系管理员");
                }
            }
        });
    }

    $(function () {
        $("#fileUpload").click(function () {
            var file = $("#file")[0].files[0];
            sendFileSha1(file);
        });
        fileList();

    });

</script>

后台java实现

file/info接口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@WebServlet("/file/info")
public class FileInfoUploadServlet extends HttpServlet {

    private FileService fileService = new FileService();

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        //TODO 待实现
    }

    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        request.setCharacterEncoding("utf-8");
        response.setCharacterEncoding("utf-8");

        //文件名
        String fileName = request.getParameter("fileName");
        //文件大小
        String fileSize = request.getParameter("fileSize");
        //文件hash值
        String fileHash = request.getParameter("fileHash");
        //文件分片总数
        String fileChunks = request.getParameter("fileChunks");
        //文件分片编号
        String fileChunkNum = request.getParameter("fileChunkNum");
        //存储文件信息到redis
        boolean result = fileService.saveFileInfoToRedis(fileName, fileSize, fileHash, fileChunks,fileChunkNum.split(","));
        //返回操作结果
        PrintWriter out = response.getWriter();
        out.write(result + "");
        out.close();
    }
}

file/info service层

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
public boolean saveFileInfoToRedis(String fileName,
                                       String fileSize,
                                       String fileHash,
                                       String fileChunks,
                                       String[] fileChunkNum) {
        Map<String, String> fileInfo = new HashMap<>();
        fileInfo.put("fileName", fileName);
        fileInfo.put("fileHash", fileHash);
        fileInfo.put("fileChunks", fileChunks);
        fileInfo.put("fileSize", fileSize);

        try {
            Jedis jedis = JedisUtil.getInstance();
            //存储文件信息到redis
            jedis.hset(fileHash, fileInfo);
            //存储文件分片信息到redis
            jedis.lpush(fileHash + "_chunkNum", fileChunkNum);

            return true;
        } catch (Exception e) {
            e.printStackTrace();
            return false;
        }
    }

file/patchUpload接口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
@MultipartConfig
@WebServlet("/file/patchUpload")
public class FilePatchUploadServlet extends HttpServlet {

    private FileService fileService = new FileService();

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        //TODO 待实现
    }

    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        request.setCharacterEncoding("utf-8");
        response.setCharacterEncoding("utf-8");

        Part part = request.getPart("fileData");
        String fileHash = request.getParameter("fileHash");
        String fileName = request.getParameter("fileName");
        String chunkNum = request.getParameter("chunkNum");

        boolean result = fileService.savePatchFileBlock(part,
                            request.getServletContext().getRealPath("/"),
                            fileName,
                            fileHash,
                            chunkNum);

        PrintWriter out = response.getWriter();
        out.write(result + "");
        out.close();
    }
}

file/patchUpload service层

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
public boolean savePatchFileBlock(Part part,String savePath, String fileName, String fileHash, String chunkNum) {
        //设置文件块存储路径
        try {
            String floder = savePath + fileHash;
            File outFile = new File(floder);
            if(!outFile.exists()) {
                outFile.mkdirs();
            }

            part.write( floder + "/" + fileName + "_temp" + chunkNum);
            //更新文件块记录
            Jedis jedis = JedisUtil.getInstance();
            jedis.lrem(fileHash + "_chunkNum", 0, chunkNum);
            if(!jedis.exists(fileHash + "_chunkNum")) {
                boolean result = FileUtil.fileCombine(fileHash,fileName,floder);
                if(result) {
                    //计算合成后的hash值(此处省略)
                    //信息入库
                    Map<String, String> file = jedis.hgetAll(fileHash);
                    saveFileInfo(new FileInfo(file.get("fileName"),
                            new Long(file.get("fileSize")),
                            file.get("fileHash"),
                            floder + "/" + fileName));

                    //删除分片文件
                    int fileChunks = Integer.parseInt(jedis.hget(fileHash, "fileChunks"));
                    FileUtil.deleteFiles(floder, fileName, fileChunks);


                }
            }
            return true;
        } catch (IOException e) {
            e.printStackTrace();
            return false;
        }
    }

FileUtil工具类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
public class FileUtil {

    public static boolean fileCombine(String fileHash, String fileName, String outPath) {
        try {
            FileOutputStream out = new FileOutputStream(outPath + "/" + fileName);
            //获取最大文件分片数
            Jedis jedis = JedisUtil.getInstance();
            String fileChunks = jedis.hget(fileHash, "fileChunks");
            for (int i = 0; i < Integer.parseInt(fileChunks); i++) {
                FileInputStream in = new FileInputStream(outPath + "/" + fileName + "_temp" + i);
                byte[] buffer = new byte[1024 * 1024 * 10];
                int len = 0;
                try {
                    while((len = in.read(buffer, 0, buffer.length)) > 0) {
                        out.write(buffer, 0, len);
                    }

                } catch (IOException e) {
                    e.printStackTrace();
                    return false;
                } finally {
                    try {
                        in.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                        return false;
                    }
                }
            }
            return true;
        } catch (FileNotFoundException e) {
            e.printStackTrace();
            return false;
        }
    }

    public static void deleteFiles(String floder, String fileName, int fileChunks) {
        for(int i = 0; i < fileChunks; i++) {
            File file = new File(floder + "/" + fileName + "_temp" + i);
            file.delete();
        }

    }
}