Muwahhid Wiki

Rclone

Общая информация.

Rclone.


Установка и подключение облачных хранилищ.

Установка:

rclone config 		
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n 			
name> mega 			
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Compress a remote
   \ "compress"
10 / Dropbox
   \ "dropbox"
11 / Encrypt/Decrypt a remote
   \ "crypt"
12 / Enterprise File Fabric
   \ "filefabric"
13 / FTP Connection
   \ "ftp"
14 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
15 / Google Drive
   \ "drive"
16 / Google Photos
   \ "google photos"
17 / Hadoop distributed file system
   \ "hdfs"
18 / Hubic
   \ "hubic"
19 / In memory object storage system.
   \ "memory"
20 / Jottacloud
   \ "jottacloud"
21 / Koofr
   \ "koofr"
22 / Local Disk
   \ "local"
23 / Mail.ru Cloud
   \ "mailru"
24 / Mega
   \ "mega"
25 / Microsoft Azure Blob Storage
   \ "azureblob"
26 / Microsoft OneDrive
   \ "onedrive"
27 / OpenDrive
   \ "opendrive"
28 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
29 / Pcloud
   \ "pcloud"
30 / Put.io
   \ "putio"
31 / QingCloud Object Storage
   \ "qingstor"
32 / SSH/SFTP Connection
   \ "sftp"
33 / Sugarsync
   \ "sugarsync"
34 / Tardigrade Decentralized Cloud Storage
   \ "tardigrade"
35 / Transparently chunk/split large files
   \ "chunker"
36 / Union merges the contents of several upstream fs
   \ "union"
37 / Webdav
   \ "webdav"
38 / Yandex Disk
   \ "yandex"
39 / Zoho
   \ "zoho"
40 / http Connection
   \ "http"
41 / premiumize.me
   \ "premiumizeme"
42 / seafile
   \ "seafile"

Storage> 24			 
** See help for mega backend at: https://rclone.org/mega/ **
User name
Enter a string value. Press Enter for the default ("").
user> myAccount@mail.ru  	
Password.
y) Yes type in my own password
g) Generate random password
y/g> y			
Enter the password:
password: 		
Confirm the password:
password:		
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n		    
Remote config
--------------------
[mega_1]
type = mega
user = mail@muwahhid.ru
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
mega_1               mega

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> 
q) e/n/d/r/c/s/q> q

Проверяем, сохранились ли настройки.

Монтируем к серверу, подключенное облачное хранилище.

mkdir -p /mnt/disks/mega
rclone mount --max-read-ahead 1024k --vfs-cache-mode full --allow-other mega: /mnt/disks/mega &
fusermount -u /mnt/disks/mega

Добавляем маунт в User scripts, чтоб с перезагрузкой, подключение не пропало.


Добавляем Google Drive.

Создаем подключение.

rclone config 		
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n 			
name> google_drive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Compress a remote
   \ "compress"
10 / Dropbox
   \ "dropbox"
11 / Encrypt/Decrypt a remote
   \ "crypt"
12 / Enterprise File Fabric
   \ "filefabric"
13 / FTP Connection
   \ "ftp"
14 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
15 / Google Drive
   \ "drive"
16 / Google Photos
   \ "google photos"
17 / Hadoop distributed file system
   \ "hdfs"
18 / Hubic
   \ "hubic"
19 / In memory object storage system.
   \ "memory"
20 / Jottacloud
   \ "jottacloud"
21 / Koofr
   \ "koofr"
22 / Local Disk
   \ "local"
23 / Mail.ru Cloud
   \ "mailru"
24 / Mega
   \ "mega"
25 / Microsoft Azure Blob Storage
   \ "azureblob"
26 / Microsoft OneDrive
   \ "onedrive"
27 / OpenDrive
   \ "opendrive"
28 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
29 / Pcloud
   \ "pcloud"
30 / Put.io
   \ "putio"
31 / QingCloud Object Storage
   \ "qingstor"
32 / SSH/SFTP Connection
   \ "sftp"
33 / Sugarsync
   \ "sugarsync"
34 / Tardigrade Decentralized Cloud Storage
   \ "tardigrade"
35 / Transparently chunk/split large files
   \ "chunker"
36 / Union merges the contents of several upstream fs
   \ "union"
37 / Webdav
   \ "webdav"
38 / Yandex Disk
   \ "yandex"
39 / Zoho
   \ "zoho"
40 / http Connection
   \ "http"
41 / premiumize.me
   \ "premiumizeme"
42 / seafile
   \ "seafile"
Storage> 15
** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id> 
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret> 
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and wri
scope> 1
ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Enter a string value. Press Enter for the default ("").
root_folder_id> 
Service Account Credentials JSON file path 
Leave blank normally.
Needed only if you want use SA instead of interactive login.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
service_account_file> 
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=DADtzDKfasfaT4dc1dwG6w
Log in and authorize rclone for access
Waiting for code...
ssh -L 53682:127.0.0.1:53682 -C -N -l root 192.168.1.7
y/n> n
--------------------
[google_drive]
type = drive
scope = drive
token = {"access_token":"ya45.A0dsaddasdUrJRg7_I4Z8vxvasdafQR1u7Zry_qPurIC0VWDDr_7V1bCQQBgX2heMSDUgfwMNlO2lrotzSDSRgdasKEhCQ_E3d2fOaS8Wd_nr_UHsVIIvjgda0dasdYNA3Xe341C","token_type":"Bearer","refresh_token":"1//0cnE9vG0adasddasAGAwSNwF-L9IrEwdasdPOJUDr_c4oHMa8ZZzyyZtGlWdasd5pXxWUQvQ","expiry":"2021-02-21T18:06:09.293151575+03:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
google_drive      drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
mkdir -p /mnt/disks/google_drive
rclone mount --max-read-ahead 1024k --vfs-cache-mode full --allow-other google_drive: /mnt/disks/google_drive &

RClone - Общие советы и информация.


Настроек у RClone очень много, персонально по Операционным системам, по Облачным хранилищам, тут лишь базовые вещи, которые использую сам. Полная информация в документации.

Google Drive - Создание собственного client_id

Когда вы используете rclone с Google диском в его конфигурации по умолчанию, вы используете rclone client_id. Он распределяется между всеми пользователями rclone. Существует глобальный лимит скорости на количество запросов в секунду, которые может выполнять каждый client_id, установленный Google.
Поэтому, настоятельно рекомендуется использовать свой собственный идентификатор клиента, так как по умолчанию установлен идентификатор rclone.

Регистрируем:

  1. 1.png

  1. 2.png

  1. 3.png

  1. 4.png

  1. 5.png

  1. 6.png

  1. 8.png

  1. 9.png

  1. 10.png

  1. 12-1.png

  1. 13.png

  1. 14.png

  1. 15.png

  1. 16.png

  1. 17.png

  1. 18.png

  1. 20.png

Google Drive Устанавливаем chunk значение 256Mb.

Увеличение этого параметра улучшит производительность, но обратите внимание, что каждый фрагмент буферизуется в памяти по одному на передачу.
Уменьшение этого параметра уменьшит использование памяти, но снизит производительность.
По умолчанию 8Mb - этот вариант середина между производительностью и использованию памяти.

Смотрим какой размер у нас chunk.

rclone backend get my_google_drive: -o chunk_size
{
        "chunk_size": "8M"
}

Устанавливается в байтах, поэтому значение 268435456 это 256Mb.

rclone backend set my_google_drive: -o chunk_size=268435456
{
        "chunk_size": {
                "current": "256M",
                "previous": "256M"
        }
}

Google Drive Очищаем корзину.

 rclone cleanup my_google_drive:
2021/02/22 21:47:52 NOTICE: Google drive root '': Note that emptying the trash happens in the background and can take some time.

Смотрим общую информацию по облачному диску.

rclone about my_google_drive:
Total:   15G
Used:    11.166G
Free:    3.814G
Trashed: 0
Other:   19.641M

Смотрим файлы в облаке из терминала lsd:

rclone lsd my_google_drive:
 16757874602 1970-01-01 03:00:00        -1 atadppa
           0 1970-01-01 03:00:00        -1 c1
   312838896 1970-01-01 03:00:00        -1 flashscripts
  1163717445 1970-01-01 03:00:00        -1 lqs