350 lines
12 KiB
Markdown
350 lines
12 KiB
Markdown
---
|
||
layout : post
|
||
title : "rClone助力大一统云存储"
|
||
subtitle : "The Swiss army knife of cloud storage"
|
||
date : 2022-11-20 14:29:09
|
||
author : "Manford Fan"
|
||
catalog : false
|
||
header-img : "img/post-bg-universe.jpg"
|
||
tags :
|
||
- rClone
|
||
- webDAV
|
||
- Cloud Storage
|
||
---
|
||
|
||
rClone是一个命令行程序，用于管理云存储上的文件。他打包了各大供应商云存储的接口提供统一的访问界面。rClone支持40多种云存储产品，包括对象存储、企业和消费者文件存储、服务以及标准传输协议。rClone符合unix POSIX规范，支持于常见的shell工具，比如rsync、cp、 mv、mount、ls、ncdu、tree、rm 和cat等交互，rClone使用我们耳闻能熟的语法，支持shell管道，支持---dry-run保护，同时为了保护数据的可靠性，rClone可保留时间戳和始终进行文件验证校验和。Rclone多种文件传输协议，支持SFTP，HTTP，WebDAV，FTP和DLNA。Rclone是一个成熟的开源软件，最初受rsync的启发并采用Golang编写。其文档和社区也都非常好，提供广泛和友好的使用用例。
|
||
|
||
## 一、应用场景
|
||
|
||
rClone的功能非常强大，被称为*"The Swiss army knife of cloud storage"*，以及*"Technology indistinguishable from magic"*。日常我们使用基本就是映射磁盘，同步数据或者多云端存储间进行数据迁移备份等。根据官方介绍，rClone可以帮助我们做如下事情：
|
||
|
||
- 加密备份文件到云端
|
||
- 从云端解密恢复文件
|
||
- 镜像云端的数据到另一个云存储或者本地
|
||
- 本地数据迁移到云端或者两个云端数据之间的迁移
|
||
- 将多个加密，并附带缓存的云端存储作为磁盘挂载到本地
|
||
- 使用内置命令lsf/ljson/size/ncdu查看并分析云存储的数据
|
||
- 聚合文件系统，使多个本地或者云端汇聚成一个文件系统
|
||
|
||
|
||
> **这里说的聚合文件系统指的是，多个云端存储能在同一个系统下共存，而不是真的将他们融合。**
|
||
|
||
|
||
|
||
## 二、支持特性
|
||
|
||
rClone支持的特性也都是直击用户痛点的，该应用是一款云存储管理工具，所以很自然传输是重中之重。
|
||
|
||
- 所有时间段MD5/SHA1校验保证文件完整性
|
||
- 文件的时间戳被保留
|
||
- 任何时间段都可以操作
|
||
- 可以是云存储提供商，也可以是云存储的用户
|
||
- 支持多线程下载
|
||
- 复制新的或者变化了的文件到云端
|
||
- 单向同步一个文件夹目录
|
||
- 将文件移动到云存储，待校验成功后删除本地文件
|
||
- 对于缺失的文件或者多余的文件做哈希校验
|
||
- 像挂载云盘一样挂载你的云存储到本地
|
||
- 把本地或者远程文件通过HTTP/WebDav/FTP/SFTP/DLNA等协议做成共享服务
|
||
- 正在开发的实验性的web页面功能
|
||
|
||
|
||
## 三、系统部署
|
||
|
||
rClone几乎支持所有主流系统下部署，Github上获得35K的Star足以证明其受欢迎的程度！根据需要，我在Windows和Linux下部署了该服务，用于云端存储的磁盘映射，更方便的进行数据的同步和备份。
|
||
|
||
#### 1. Windows部署
|
||
|
||
Windows下部署rClone的步骤如下，详细部署动作可以参考命令行以及输出：
|
||
|
||
1. 下载二进制exe[文件](https://rclone.org/downloads/)并安装
|
||
2. 【Optional】设置系统环境变量，基础操作不再赘述
|
||
3. 执行`rclone config`命令设置远程存储类型
|
||
4. 将设置好的远程挂载到本地
|
||
5. 设定开机自动挂载
|
||
|
||
```powershell
|
||
C:\Users\Manford Fan>rclone config
|
||
No remotes found, make a new one?
|
||
n) New remote
|
||
s) Set configuration password
|
||
q) Quit config
|
||
n/s/q> n
|
||
|
||
Enter name for new remote.
|
||
name> ALIST
|
||
|
||
Option Storage.
|
||
Type of storage to configure.
|
||
Choose a number from below, or type in your own value.
|
||
1 / 1Fichier
|
||
\ (fichier)
|
||
2 / Akamai NetStorage
|
||
\ (netstorage)
|
||
3 / Alias for an existing remote
|
||
\ (alias)
|
||
4 / Amazon Drive
|
||
\ (amazon cloud drive)
|
||
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
|
||
\ (s3)
|
||
6 / Backblaze B2
|
||
\ (b2)
|
||
7 / Better checksums for other remotes
|
||
\ (hasher)
|
||
8 / Box
|
||
\ (box)
|
||
9 / Cache a remote
|
||
\ (cache)
|
||
10 / Citrix Sharefile
|
||
\ (sharefile)
|
||
11 / Combine several remotes into one
|
||
\ (combine)
|
||
12 / Compress a remote
|
||
\ (compress)
|
||
13 / Dropbox
|
||
\ (dropbox)
|
||
14 / Encrypt/Decrypt a remote
|
||
\ (crypt)
|
||
15 / Enterprise File Fabric
|
||
\ (filefabric)
|
||
16 / FTP
|
||
\ (ftp)
|
||
17 / Google Cloud Storage (this is not Google Drive)
|
||
\ (google cloud storage)
|
||
18 / Google Drive
|
||
\ (drive)
|
||
19 / Google Photos
|
||
\ (google photos)
|
||
20 / HTTP
|
||
\ (http)
|
||
21 / Hadoop distributed file system
|
||
\ (hdfs)
|
||
22 / HiDrive
|
||
\ (hidrive)
|
||
23 / In memory object storage system.
|
||
\ (memory)
|
||
24 / Internet Archive
|
||
\ (internetarchive)
|
||
25 / Jottacloud
|
||
\ (jottacloud)
|
||
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
|
||
\ (koofr)
|
||
27 / Local Disk
|
||
\ (local)
|
||
28 / Mail.ru Cloud
|
||
\ (mailru)
|
||
29 / Mega
|
||
\ (mega)
|
||
30 / Microsoft Azure Blob Storage
|
||
\ (azureblob)
|
||
31 / Microsoft OneDrive
|
||
\ (onedrive)
|
||
32 / OpenDrive
|
||
\ (opendrive)
|
||
33 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
||
\ (swift)
|
||
34 / Oracle Cloud Infrastructure Object Storage
|
||
\ (oracleobjectstorage)
|
||
35 / Pcloud
|
||
\ (pcloud)
|
||
36 / Put.io
|
||
\ (putio)
|
||
37 / QingCloud Object Storage
|
||
\ (qingstor)
|
||
38 / SMB / CIFS
|
||
\ (smb)
|
||
39 / SSH/SFTP
|
||
\ (sftp)
|
||
40 / Sia Decentralized Cloud
|
||
\ (sia)
|
||
41 / Storj Decentralized Cloud Storage
|
||
\ (storj)
|
||
42 / Sugarsync
|
||
\ (sugarsync)
|
||
43 / Transparently chunk/split large files
|
||
\ (chunker)
|
||
44 / Union merges the contents of several upstream fs
|
||
\ (union)
|
||
45 / Uptobox
|
||
\ (uptobox)
|
||
46 / WebDAV
|
||
\ (webdav)
|
||
47 / Yandex Disk
|
||
\ (yandex)
|
||
48 / Zoho
|
||
\ (zoho)
|
||
49 / premiumize.me
|
||
\ (premiumizeme)
|
||
50 / seafile
|
||
\ (seafile)
|
||
Storage> 46
|
||
|
||
Option url.
|
||
URL of http host to connect to.
|
||
E.g. https://example.com.
|
||
Enter a value.
|
||
url> https://alist.rustle.cc/dav/
|
||
|
||
Option vendor.
|
||
Name of the WebDAV site/service/software you are using.
|
||
Choose a number from below, or type in your own value.
|
||
Press Enter to leave empty.
|
||
1 / Nextcloud
|
||
\ (nextcloud)
|
||
2 / Owncloud
|
||
\ (owncloud)
|
||
3 / Sharepoint Online, authenticated by Microsoft account
|
||
\ (sharepoint)
|
||
4 / Sharepoint with NTLM authentication, usually self-hosted or on-premises
|
||
\ (sharepoint-ntlm)
|
||
5 / Other site/service or software
|
||
\ (other)
|
||
vendor> 5
|
||
|
||
Option user.
|
||
User name.
|
||
In case NTLM authentication is used, the username should be in the format 'Domain\User'.
|
||
Enter a value. Press Enter to leave empty.
|
||
user> admin
|
||
|
||
Option pass.
|
||
Password.
|
||
Choose an alternative below. Press Enter for the default (n).
|
||
y) Yes, type in my own password
|
||
g) Generate random password
|
||
n) No, leave this optional password blank (default)
|
||
y/g/n> y
|
||
Enter the password:
|
||
password:
|
||
Confirm the password:
|
||
password:
|
||
|
||
Option bearer_token.
|
||
Bearer token instead of user/pass (e.g. a Macaroon).
|
||
Enter a value. Press Enter to leave empty.
|
||
bearer_token>
|
||
|
||
Edit advanced config?
|
||
y) Yes
|
||
n) No (default)
|
||
y/n>
|
||
|
||
Configuration complete.
|
||
Options:
|
||
- type: webdav
|
||
- url: https://alist.rustle.cc/dav/
|
||
- vendor: other
|
||
- user: admin
|
||
- pass: *** ENCRYPTED ***
|
||
Keep this "ALIST" remote?
|
||
y) Yes this is OK (default)
|
||
e) Edit this remote
|
||
d) Delete this remote
|
||
y/e/d> y
|
||
|
||
Current remotes:
|
||
|
||
Name Type
|
||
==== ====
|
||
ALIST webdav
|
||
|
||
e) Edit existing remote
|
||
n) New remote
|
||
d) Delete remote
|
||
r) Rename remote
|
||
c) Copy remote
|
||
s) Set configuration password
|
||
q) Quit config
|
||
e/n/d/r/c/s/q> e
|
||
|
||
Select remote.
|
||
Choose a number from below, or type in an existing value.
|
||
1 > ALIST
|
||
remote> 1
|
||
|
||
Editing existing "ALIST" remote with options:
|
||
- type: webdav
|
||
- url: https://alist.rustle.cc/dav/
|
||
- vendor: other
|
||
- user: admin
|
||
- pass: *** ENCRYPTED ***
|
||
|
||
Option url.
|
||
URL of http host to connect to.
|
||
E.g. https://example.com.
|
||
Enter a string value. Press Enter for the default (https://alist.rustle.cc/dav/).
|
||
```
|
||
|
||
配置好远程存储之后，就需要把配置好的远端挂载到本地：
|
||
|
||
```powershell
|
||
rclone mount ALIST:/ Z: --vfs-cache-mode full --vfs-cache-max-size 10G --vfs-read-ahead 100M --vfs-cache-max-age 4h --cache-dir D:\TMP\vfs-cache --bwlimit-file 20M --bwlimit 100M --log-level NOTICE --vfs-read-chunk-size-limit 10m --buffer-size 50M --attr-timeout 5m --transfers=6 --multi-thread-streams=6 --log-file D:\TMP\rclone\rclone.log
|
||
```
|
||
|
||
如果想要开机启动可以将如下保存至rclone.vbs文件，并将该文件放置在Windows开机启动目录，以我的为例：`C:\Users\Manford Fan\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup`，如下程序以及如上开机启动目录需要**根据自己的情况**修改path路径：
|
||
|
||
```vbs
|
||
Option Explicit
|
||
Dim WMIService, Process, Processes, Flag, WS
|
||
Set WMIService = GetObject("winmgmts:{impersonationlevel=impersonate}!\\.\root\cimv2")
|
||
Set Processes = WMIService.ExecQuery("select * from win32_process")
|
||
Flag = true
|
||
for each Process in Processes
|
||
if strcomp(Process.name, "rclone.exe") = 0 then
|
||
Flag = false
|
||
exit for
|
||
end if
|
||
next
|
||
Set WMIService = nothing
|
||
if Flag then
|
||
Set WS = Wscript.CreateObject("Wscript.Shell")
|
||
WS.Run "rclone mount ALIST:/ Z: --vfs-cache-mode full ^
|
||
--vfs-cache-max-size 10G ^
|
||
--vfs-read-ahead 100M ^
|
||
--vfs-cache-max-age 4h ^
|
||
--cache-dir D:\TMP\vfs-cache ^
|
||
--bwlimit-file 20M ^
|
||
--bwlimit 100M --log-level NOTICE ^
|
||
--vfs-read-chunk-size-limit 10m ^
|
||
--buffer-size 50M --attr-timeout 5m ^
|
||
--transfers=6 --multi-thread-streams=6 ^
|
||
--log-file D:\TMP\rclone\rclone.log", 0
|
||
end if
|
||
```
|
||
|
||
#### 2. Linux部署
|
||
|
||
Linux下部署和Windows差不多，只不过可配置的东西更多也更方便一点，比如开机自启动，掉盘后的自动挂载等。
|
||
|
||
1. 下载二进制deb包[文件](https://rclone.org/downloads/)并安装
|
||
2. 执行`rclone config`命令设置远程存储类型
|
||
3. 将设置好的远程挂载到本地
|
||
4. 设定开机自动挂载
|
||
5. 定时任务每分钟检测服务是否正常，如有异常则自动重启，或者手动强制重启
|
||
|
||
```bash
|
||
$ apt install rclone
|
||
$ rclone config
|
||
......
|
||
$ /usr/bin/rclone mount ALIST:/ /opt/webdav --allow-other --vfs-cache-mode full --vfs-cache-max-size 10G --vfs-read-ahead 100M --vfs-cache-max-age 4h --cache-dir /tmp/vfs-cache --bwlimit-file 20M --bwlimit 100M --log-file /opt/logs/rclone.log --log-level NOTICE --vfs-read-chunk-size-limit 10m --buffer-size 50M --attr-timeout 5m --transfers=6 --multi-thread-streams=6 --allow-non-empty
|
||
# 此时可以在/opt/webdav下看到远程磁盘挂载到了该目录下
|
||
```
|
||
|
||
需要注意的是，虽然通过`rclone`可以挂载各种云盘到本地，但是一般情况下，不建议直接从本地复制特别大的文件，或者数量特别多的文件到远程，除非明确的知道这件事情的后果，包括网络层会做什么，对你的本地机器会有什么影响等，建议还是直接从原生云盘上传文件（可以有秒传的功能），然后在挂载上的云盘上使用。
|
||
|
||
## 四、常用命令
|
||
|
||
rClone支持的命令非常多，可以参考[Rclone Commands](https://rclone.org/commands/)页面有选择的使用，对于不熟悉的命令，可以使用`rclone command --help`来查看用法。
|
||
|
||
|
||
## 五、配置选项
|
||
|
||
[Global Flags](https://rclone.org/flags/)提供了rClone支持的全部配置选项，如果想要看某一个命令具体支持哪些，可以通过如上提到的命令列表中，跳转到特定命令页查看。
|
||
|
||
|
||
## 六、参考文档
|
||
|
||
- [rClone官方主页](https://rclone.org/)
|
||
- [RCLONE.VBS 挂载脚本](https://xiwaer.com/686.html)
|
||
|