Burp: Difference between revisions

From DcSharedWiki
(result added (1.5Go added))
m (39 revisions imported)
 
(31 intermediate revisions by one other user not shown)
Line 6: Line 6:
Version: 1.3.48 (debian package 1.3.48-4~bpo70+1)
Version: 1.3.48 (debian package 1.3.48-4~bpo70+1)


Hardware configuration:
=== Hardware configuration ===


{| border="1" cellpadding="2" cellspacing="0"
{| border="1" cellpadding="2" cellspacing="0"
| Type
| Type
| CPU
| CPU
| SATA link
| Disk
| Disk
| RAM
| RAM
|-
|-
| Server
| Server
| Marvell Armada 370 (ReadyNAS 104)
| Marvell Armada 370 (ReadyNAS 102)
| `SATA link up 6.0 Gbps`
| 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm)
| 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm)
| 512Mo
| 512MB
|-
|-
| Client1
| Client1
| VIA Nano L2007 1600MHz
| VIA Nano L2007 1600MHz
| `SATA link up 1.5 Gbps`
| 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm)
| 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm)
| 4Go
| 4GB
|-
|-
| Client2
| Client2
| VIA Nano L2007 1600MHz
| VIA Nano L2007 1600MHz
| `SATA link up 1.5 Gbps`
| 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm)
| 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm)
| 4Go
| 4GB
|-
| Client3
| Intel Core i7-3770
| `SATA link up 3.0 Gbps`
| WDC WD1003FBYX-01Y7B1
| 4GB
|}
|}


Line 36: Line 46:
* server: `Marvell 88E1318 Gigabit Ethernet PHY` (controller), `pegasus` (module)
* server: `Marvell 88E1318 Gigabit Ethernet PHY` (controller), `pegasus` (module)


=== Performances ===
Copying 212Go from client to backup server took 8h 1m 48s (7.50Mo/s). `iperf` measures 890 Mbits/s between server and client.

==== Network ====

* `iperf` measures 890 Mbits/s between server and client.
* Copying 244.31GB grom client to backup server using rsync took 36h 21m (`sent 189.41Gbytes - received 25.48Mbytes - 1.45Mbytes/sec`).
* Copying 212GB from client1 (VIA [[PadLock]] usable by OpenSSL) to backup server using `scp` (cipher specification/MAC algorithm: `aes128-ctr`/`hmac-md5`) took 8h 1m 48s (7.50MB/s).
* Copying 216GB from client3 to backup server using `scp` (cipher specification/MAC algorithm: `aes128-ctr`/`umac-64@openssh.com`) took 4h 45m 04s (12.93MB/s).

==== Encryption ====

* SCP transfers of 698MB from the client1 (VIA [[PadLock]] usable by OpenSSL) to the server using different cipher and MAC algorithm (`aes128-ctr` and `hmac-md5` are the default settings on `Debian Wheezy`):
|| cipher specification || MAC algorithm || Throughput || Duration ||
|| aes128-ctr || hmac-md5 || 5.9MB/s || 01:59 ||
|| aes128-ctr || hmac-sha2-256 || 10.7MB/s || 01:05 ||
|| aes128-ctr || hmac-sha2-256-96 || 10.7MB/s || 01:05 ||
|| aes128-ctr || hmac-sha1 || 11.3MB/s || 01:02 ||
|| aes128-ctr || hmac-sha1-96 || 11.3MB/s || 01:02 ||
|| aes128-ctr || hmac-sha2-512 || 8.3MB/s || 01:24 ||
|| aes192-ctr || hmac-sha2-256 || 9.4MB/s || 01:14 ||
|| aes256-ctr || hmac-sha2-256 || 8.4MB/s || 01:23 ||
|| aes128-cbc || hmac-sha1 || 14.0MB/s || 00:50 ||
|| aes128-cbc || hmac-sha2-256 || 11.8MB/s || 00:59 ||
|| aes192-cbc || hmac-sha2-256 || 11.5MB/s || 01:01 ||


== Test using client1 ==
== Test using client1 ==
Line 42: Line 75:
{| border="1" cellpadding="2" cellspacing="0"
{| border="1" cellpadding="2" cellspacing="0"
| Description
| Description
| Compression (`compression`) <<[[FootNote]](Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>>
| Compression
| SSL compression
| SSL compression (`ssl_compression`)
| Duration on client (phase 1/phase 2)
| Duration on client
| phase 1
| phase 2
| Data sent
| Throughput (phase 2)
| Throughput (phase 2)
| Duration on server
| phase 3
|-
|-
| backup: 1.5Go added
| full backup: 209.17GB
| gzip9 (default value)
| zlib5 (default value)
|align="right" | 34h 17m 34s
|align="right" | 07m 18s
|align="right" | 34h 10m 18s
|align="right" | 167.4GB
|align="right" | 1.4MB/s
|align="right" | 06m 22s
|align="right" | 02m 22s
|-
| full backup: 209.17GB
| None (`compression=gzip0`)
| None (`compression=gzip0`)
| zlib5 (default value)
| (`ssl_compression=zlib0`)
|align="right" | 14h 49m 29s
| 11m 30s <span style="font-size: smaller">(4m 36s/6m 54s)</span>
|align="right" | 3.7Mo/s
|align="right" | 03m 57s
|align="right" | 14h 45m 32s
|align="right" | 209.17GB
|align="right" | 4.0MB/s
|align="right" | 07m 04s
|align="right" | 01m 09s
|-
| full backup: 214.27GB
| None (`compression=gzip0`)
| zlib5 (default value)
|align="right" | 30h 05m 44s
|align="right" | 7m 45s
|align="right" | 29h 57m 59s
|align="right" | 214.27GB
|align="right" | 2.0MB/s
|align="right" | 06m 56s
|align="right" | 01m 01s
|-
| full backup: 212.47GB
| None (`compression=gzip0`)
| None (`ssl_compression=zlib0`)
|align="right" | 24h 19m 53s
|align="right" | 04m 36s
|align="right" | 24h 15m 17s
|align="right" | 212.47GB
|align="right" | 2.5MB/s
|align="right" | 06m 57s
|align="right" | 54s
|-
| full backup: 213.99GB
| None (`compression=gzip0`)
| None (`ssl_compression=zlib0`)
|align="right" | 30h 00m 45s
|align="right" | 05m 47s
|align="right" | 29h 54m 58s
|align="right" | 213.99GB
|align="right" | 2.0MB/s
|align="right" | 06m 59s
|align="right" | 01m 05s
|-
| incremental backup: 1.5GB added
| None (`compression=gzip0`)
| None (`ssl_compression=zlib0`)
|align="right" | 11m 30s
|align="right" | 04m 36s
|align="right" | 06m 54s
|align="right" | 1.5GB
|align="right" | 3.7MB/s
|align="right" | 16m 28s
|align="right" | 01m 01s
|}
|}


== Test using client2: full backups ==
=== Old tests (there was a network configuration problem: `r8169` module was used instead of `r8168` on the client) ===


{| border="1" cellpadding="2" cellspacing="0"
==== Test using client1 ====
| Description
| librsync
| SSL algo
| `compression` <<[[FootNote]](Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>>
| `ssl_compression`
| Duration on client
| phase 1
| phase 2
| Data sent
| Throughput (phase 2)
| Duration on server
| phase 3
|-
| 227.44GB
| Enabled<<[[FootNote]](Note: default value for librsync)>>
| AES128-SHA
| gzip0
| zlib5 (default value)
|align="right" | 28h 17m 41s
|align="right" | 08m 51s
|align="right" | 28h 08m 50s
|align="right" | 227.44GB
|align="right" | 2.29MB/s
|align="right" | 09m 37s
|align="right" | 01m 21s
|-
| 227.45GB
| Enabled
| DHE-RSA-AES256-GCM-SHA384
| gzip9 (default value)
| zlib5 (default value)
|align="right" | 45h 02m 55s
|align="right" | 06m 28s
|align="right" | 44h 56m 27s
|align="right" | 176.91GB
|align="right" | 1.11MB/s
|align="right" | 12m 02s
|align="right" | 03m 08s
|-
| 227.48GB
| Enabled
| DHE-RSA-AES256-GCM-SHA384
| gzip9 (default value)
| zlib0
|align="right" | 44h 57m 35s
|align="right" | 06m 26s
|align="right" | 44h 51m 09s
|align="right" | 176.94GB
|align="right" | 1.12MB/s
|align="right" | 11m 32s
|align="right" | 03m 06s
|-
| 227.49GB
| Enabled
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| zlib5 (default value)
|align="right" | 28h 19m 06s
|align="right" | 09m 39s
|align="right" | 28h 09m 37s
|align="right" | 227.49GB
|align="right" | 2.29MB/s
|align="right" | 09m 37s
|align="right" | 01m 24s
|-
| 227.52GB
| Disabled<<[[FootNote]](Note: librsync=0)>>
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| zlib5 (default value)
|align="right" | 28h 40m 12s
|align="right" | 11m 30s
|align="right" | 28h 28m 42s
|align="right" | 227.52GB
|align="right" | 2.27MB/s
|align="right" | 09m 43s
|align="right" | 01m 23s
|-
| 227.63GB
| Disabled<<[[FootNote]](Note: librsync=0)>>
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| zlib5 (default value)
|align="right" | 28h 10m 46s
|align="right" | 10m 17s
|align="right" | 28h 00m 29s
|align="right" | 227.63GB
|align="right" | 2.31MB/s
|align="right" | 09m 58s
|align="right" | 01m 26s
|-
| 227.65GB
| Disabled<<[[FootNote]](Note: librsync=0)>>
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| zlib5 (default value)
|align="right" | 17h 56m 29s
|align="right" | 10m 38s
|align="right" | 17h 45m 51s
|align="right" | 227.65GB
|align="right" | 3.64MB/s
|align="right" | 09m 58s
|align="right" | 01m 25s
|-
| 227.65GB
| Disabled<<[[FootNote]](Note: librsync=0)>>
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| zlib5 (default value)
|align="right" | 17h 24m 56s
|align="right" | 03m 50s
|align="right" | 17h 21m 06s
|align="right" | 227.65GB
|align="right" | 3.73MB/s
|align="right" | 09m 51s
|align="right" | 01m 27s
|}

== Test using client3: full backups ==

{| border="1" cellpadding="2" cellspacing="0"
| Description
| librsync
| SSL algo
| `compression` <<[[FootNote]](Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>>
| `ssl_compression`
| Duration on client
| phase 1
| phase 2
| Data sent
| Throughput (phase 2)
| Duration on server
| phase 3
|-
| 229.63GB
| Enabled<<[[FootNote]](Note: default value for librsync)>>
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| zlib5 (default value)
|align="right" | 06h 44m 39s
|align="right" | 00m 15s
|align="right" | 06h 44m 24s
|align="right" | 229.63GB
|align="right" | 9.68MB/s
|align="right" | 00m 26s
|align="right" | 00m 04s
|-
| 229.67GB
| Enabled<<[[FootNote]](Note: default value for librsync)>>
| DHE-RSA-AES256-GCM-SHA384
| gzip9
| zlib5 (default value)
|align="right" | 06h 59m 43s
|align="right" | 00m 14s
|align="right" | 06h 59m 29s
|align="right" | 227.37GB
|align="right" | 9.25MB/s
|align="right" | 00m 36s
|align="right" | 00m 11s
|}

== Test using client2: restore one file ==


<pre><nowiki>
/usr/sbin/burp -c /tmp/burp-client.conf -a r -b 1 -r '^/mnt/data/Documents/file.pdf$' -d /tmp/ -s 4 -b 30
</nowiki></pre>


Size of backups: 249GB

{| border="1" cellpadding="2" cellspacing="0"
|-
|-
|-
|-
|}

== Test using client2: incremental backup ==

{| border="1" cellpadding="2" cellspacing="0"
| Time taken
|-
| 31m 48s
|-
| 39m 59s
|-
| 37m 39s
|-
| 39m 00s
|-
| 39m 13s
|-
| 38m 35s
|-
| 38m 37s
|-
| 37m 37s
|-
| 37m 52s
|-
| 42m 15s
|-
| 37m 50s
|-
| 41m 40s
|-
| 43m 46s
|-
| 42m 56s
|-
| 42m 16s
|-
| 42m 27s
|-
| 43m 02s
|-
| 42m 18s
|-
| 44m 41s
|-
| 41m 15s
|-
| 38m 30s
|-
| 40m 16s
|-
| 39m 40s
|-
| 41m 17s
|-
| 39m 15s
|-
| 40m 42s
|}

=== Logs ===

'''Client1 full backup: 214.27GB'''

<pre><nowiki>
2014-07-19 18:32:40: burp[9606] Client version: 1.3.48
2014-07-19 18:32:40: burp[9606] Begin phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] End phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] Begin phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] End phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] Begin phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] End phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] Backup ending - disconnect from client.
2014-07-21 00:39:25: burp[9606] Begin phase4 (shuffle files)
2014-07-21 00:39:25: burp[9606] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-19 18:32:39
End time: 2014-07-21 00:45:20
Time taken: 01:06:12:41
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 998898 0 0 0 998898 | 998898
Directories: 74226 0 0 0 74226 | 74226
Soft links: 4531 0 0 0 4531 | 4531
Hard links: 16 0 0 0 16 | 16
Grand total: 1077671 0 0 0 1077671 | 1077671
------------------------------------------------------------

Warnings: 0

Bytes estimated: 230074235176 (214.27 GB)
Bytes in backup: 230074235177 (214.27 GB)
Bytes received: 230074235177 (214.27 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-07-21 00:45:20: burp[9606] Backup completed.
2014-07-21 00:45:20: burp[9606] End phase4 (shuffle files)
</nowiki></pre>


'''Client 1 full backup 212.47GB'''

<pre><nowiki>
2014-07-14 23:49:34: burp[5369] Client version: 1.3.48
2014-07-14 23:49:34: burp[5369] Begin phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] End phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] Begin phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] End phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] Begin phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] End phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] Backup ending - disconnect from client.
2014-07-16 01:10:31: burp[5369] Begin phase4 (shuffle files)
2014-07-16 01:10:31: burp[5369] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-14 23:49:26
End time: 2014-07-16 01:16:34
Time taken: 01:01:27:08
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 998898 0 0 0 998898 | 998898
Directories: 74226 0 0 0 74226 | 74226
Soft links: 4531 0 0 0 4531 | 4531
Hard links: 16 0 0 0 16 | 16
Grand total: 1077671 0 0 0 1077671 | 1077671
------------------------------------------------------------

Warnings: 0

Bytes estimated: 228124139118 (212.46 GB)
Bytes in backup: 228135345774 (212.47 GB)
Bytes received: 228135345774 (212.47 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-07-16 01:16:34: burp[5369] Backup completed.
2014-07-16 01:16:34: burp[5369] End phase4 (shuffle files)
</nowiki></pre>


'''Client 1 full backup 213.99GB'''

<pre><nowiki>
2014-07-17 01:47:37: burp[4654] Client version: 1.3.48
2014-07-17 01:47:37: burp[4654] Begin phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] End phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] Begin phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] End phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] Begin phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] End phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] Backup ending - disconnect from client.
2014-07-18 07:49:27: burp[4654] Begin phase4 (shuffle files)
2014-07-18 07:49:27: burp[4654] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-17 01:47:36
End time: 2014-07-18 07:55:21
Time taken: 01:06:07:45
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 998898 0 0 0 998898 | 998898
Directories: 74226 0 0 0 74226 | 74226
Soft links: 4531 0 0 0 4531 | 4531
Hard links: 16 0 0 0 16 | 16
Grand total: 1077671 0 0 0 1077671 | 1077671
------------------------------------------------------------

Warnings: 0

Bytes estimated: 228149960334 (212.48 GB)
Bytes in backup: 229772435086 (213.99 GB)
Bytes received: 229772435086 (213.99 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-07-18 07:55:21: burp[4654] Backup completed.
2014-07-18 07:55:21: burp[4654] End phase4 (shuffle files)
</nowiki></pre>


'''Client 2 full backup 227.44GB'''

<pre><nowiki>
2014-08-23 15:18:35: burp[4691] Client version: 1.3.48
2014-08-23 15:18:35: burp[4691] Begin phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] End phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] Begin phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] End phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] Begin phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] End phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] Backup ending - disconnect from client.
2014-08-24 19:37:37: burp[4691] Begin phase4 (shuffle files)
2014-08-24 19:37:37: burp[4691] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-23 15:18:34
End time: 2014-08-24 19:45:53
Time taken: 01:04:27:19
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1318031 0 0 0 1318031 | 1318286
Directories: 104066 0 0 0 104066 | 104066
Soft links: 4586 0 0 0 4586 | 4586
Hard links: 99 0 0 0 99 | 99
Grand total: 1426782 0 0 0 1426782 | 1427037
------------------------------------------------------------

Warnings: 255

Bytes estimated: 244227872076 (227.45 GB)
Bytes in backup: 244212565354 (227.44 GB)
Bytes received: 244212565354 (227.44 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-08-24 19:45:53: burp[4691] Backup completed.
2014-08-24 19:45:53: burp[4691] End phase4 (shuffle files)
</nowiki></pre>


'''Client 2 full backup 227.45GB'''

<pre><nowiki>
2014-08-25 01:23:42: burp[9427] Client version: 1.3.48
2014-08-25 01:23:42: burp[9427] Begin phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] End phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] Begin phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] End phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] Begin phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] End phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] Backup ending - disconnect from client.
2014-08-26 22:29:45: burp[9427] Begin phase4 (shuffle files)
2014-08-26 22:29:45: burp[9427] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-25 01:23:42
End time: 2014-08-26 22:38:09
Time taken: 01:21:14:27
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1318211 0 0 0 1318211 | 1318211
Directories: 104070 0 0 0 104070 | 104070
Soft links: 4586 0 0 0 4586 | 4586
Hard links: 99 0 0 0 99 | 99
Grand total: 1426966 0 0 0 1426966 | 1426966
------------------------------------------------------------

Warnings: 0

Bytes estimated: 244219700992 (227.45 GB)
Bytes in backup: 244219700992 (227.45 GB)
Bytes received: 189963848136 (176.92 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-08-26 22:38:09: burp[9427] Backup completed.
2014-08-26 22:38:09: burp[9427] End phase4 (shuffle files)
</nowiki></pre>


'''Client 2 full backup 227.49GB'''

<pre><nowiki>
2014-08-30 17:26:31: burp[31130] Client version: 1.3.48
2014-08-30 17:26:31: burp[31130] Begin phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] End phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] Begin phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] End phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] Begin phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] End phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] Backup ending - disconnect from client.
2014-08-31 21:48:11: burp[31130] Begin phase4 (shuffle files)
2014-08-31 21:48:11: burp[31130] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-30 17:26:30
End time: 2014-08-31 21:56:32
Time taken: 01:04:30:02
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1320475 0 0 0 1320475 | 1320484
Directories: 104340 0 0 0 104340 | 104340
Soft links: 4588 0 0 0 4588 | 4588
Hard links: 99 0 0 0 99 | 99
Grand total: 1429502 0 0 0 1429502 | 1429511
------------------------------------------------------------

Warnings: 9

Bytes estimated: 244267585965 (227.49 GB)
Bytes in backup: 244267822787 (227.49 GB)
Bytes received: 244267822787 (227.49 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-08-31 21:56:32: burp[31130] Backup completed.
2014-08-31 21:56:32: burp[31130] End phase4 (shuffle files)
</nowiki></pre>


'''Client 2 full backup 227.52GB'''

<pre><nowiki>
2014-09-07 00:42:33: burp[22795] Client version: 1.3.48
2014-09-07 00:42:33: burp[22795] Begin phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] End phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] Begin phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] End phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] Begin phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] End phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] Backup ending - disconnect from client.
2014-09-08 05:24:08: burp[22795] Begin phase4 (shuffle files)
2014-09-08 05:24:08: burp[22795] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-07 00:42:32
End time: 2014-09-08 05:32:25
Time taken: 01:04:49:53
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1322008 0 0 0 1322008 | 1322008
Directories: 104595 0 0 0 104595 | 104595
Soft links: 4588 0 0 0 4588 | 4588
Hard links: 99 0 0 0 99 | 99
Grand total: 1431290 0 0 0 1431290 | 1431290
------------------------------------------------------------

Warnings: 0

Bytes estimated: 244299698290 (227.52 GB)
Bytes in backup: 244299767377 (227.52 GB)
Bytes received: 244299767377 (227.52 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-09-08 05:32:26: burp[22795] Backup completed.
2014-09-08 05:32:26: burp[22795] End phase4 (shuffle files)

</nowiki></pre>


'''Client 2 full backup 227.63GB'''

<pre><nowiki>
2014-09-16 01:07:22: burp[3507] Client version: 1.3.48
2014-09-16 01:07:22: burp[3507] Begin phase1 (file system scan)
2014-09-16 01:17:39: burp[3507] End phase1 (file system scan)
2014-09-16 01:17:39: burp[3507] Begin phase2 (receive file data)
2014-09-17 05:18:08: burp[3507] End phase2 (receive file data)
2014-09-17 05:18:08: burp[3507] Begin phase3 (merge manifests)
2014-09-17 05:19:34: burp[3507] End phase3 (merge manifests)
2014-09-17 05:19:34: burp[3507] Backup ending - disconnect from client.
2014-09-17 05:19:34: burp[3507] Begin phase4 (shuffle files)
2014-09-17 05:19:34: burp[3507] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-16 01:07:20
End time: 2014-09-17 05:28:06
Time taken: 01:04:20:46
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1334116 0 0 0 1334116 | 1334117
Directories: 110080 0 0 0 110080 | 110080
Soft links: 5461 0 0 0 5461 | 5461
Hard links: 99 0 0 0 99 | 99
Grand total: 1449756 0 0 0 1449756 | 1449757
------------------------------------------------------------

Warnings: 1

Bytes estimated: 244419793362 (227.63 GB)
Bytes in backup: 244419785986 (227.63 GB)
Bytes received: 244419785986 (227.63 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-09-17 05:28:06: burp[3507] Backup completed.
2014-09-17 05:28:06: burp[3507] End phase4 (shuffle files)
</nowiki></pre>


'''Client 2 full backup 227.65GB'''

<pre><nowiki>

2014-10-15 03:17:06: burp[31867] Client version: 1.3.48
2014-10-15 03:17:06: burp[31867] Begin phase1 (file system scan)
2014-10-15 03:27:44: burp[31867] End phase1 (file system scan)
2014-10-15 03:27:44: burp[31867] Begin phase2 (receive file data)
2014-10-15 21:13:35: burp[31867] End phase2 (receive file data)
2014-10-15 21:13:35: burp[31867] Begin phase3 (merge manifests)
2014-10-15 21:15:00: burp[31867] End phase3 (merge manifests)
2014-10-15 21:15:00: burp[31867] Backup ending - disconnect from client.
2014-10-15 21:15:00: burp[31867] Begin phase4 (shuffle files)
2014-10-15 21:15:00: burp[31867] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-10-15 03:17:05
End time: 2014-10-15 21:23:32
Time taken: 18:06:27
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1334339 0 0 0 1334339 | 1334339
Directories: 110116 0 0 0 110116 | 110116
Soft links: 5461 0 0 0 5461 | 5461
Hard links: 99 0 0 0 99 | 99
Grand total: 1450015 0 0 0 1450015 | 1450015
------------------------------------------------------------

Warnings: 0

Bytes estimated: 244439612263 (227.65 GB)
Bytes in backup: 244439612178 (227.65 GB)
Bytes received: 244439612178 (227.65 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-10-15 21:23:33: burp[31867] Backup completed.
2014-10-15 21:23:33: burp[31867] End phase4 (shuffle files)

</nowiki></pre>


'''Client 2 full backup 227.65GB'''

<pre><nowiki>
2014-10-16 12:32:02: burp[3210] Client version: 1.3.48
2014-10-16 12:32:02: burp[3210] Begin phase1 (file system scan)
2014-10-16 12:35:52: burp[3210] End phase1 (file system scan)
2014-10-16 12:35:52: burp[3210] Begin phase2 (receive file data)
2014-10-17 05:57:58: burp[3210] End phase2 (receive file data)
2014-10-17 05:57:58: burp[3210] Begin phase3 (merge manifests)
2014-10-17 05:59:25: burp[3210] End phase3 (merge manifests)
2014-10-17 05:59:25: burp[3210] Backup ending - disconnect from client.
2014-10-17 05:59:25: burp[3210] Begin phase4 (shuffle files)
2014-10-17 05:59:25: burp[3210] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-10-16 12:32:00
End time: 2014-10-17 06:07:50
Time taken: 17:35:50
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1334340 0 0 0 1334340 | 1334340
Directories: 110116 0 0 0 110116 | 110116
Soft links: 5461 0 0 0 5461 | 5461
Hard links: 99 0 0 0 99 | 99
Grand total: 1450016 0 0 0 1450016 | 1450016
------------------------------------------------------------

Warnings: 0

Bytes estimated: 244439619502 (227.65 GB)
Bytes in backup: 244439631290 (227.65 GB)
Bytes received: 244439631290 (227.65 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-10-17 06:07:51: burp[3210] Backup completed.
2014-10-17 06:07:51: burp[3210] End phase4 (shuffle files)
</nowiki></pre>


'''Client 3 full backup 229.63GB'''

<pre><nowiki>
2014-11-18 03:58:03: burp[4847] Client version: 1.3.48
2014-11-18 03:58:03: burp[4847] Begin phase1 (file system scan)
2014-11-18 03:58:18: burp[4847] End phase1 (file system scan)
2014-11-18 03:58:18: burp[4847] Begin phase2 (receive file data)
2014-11-18 10:42:42: burp[4847] End phase2 (receive file data)
2014-11-18 10:42:42: burp[4847] Begin phase3 (merge manifests)
2014-11-18 10:42:46: burp[4847] End phase3 (merge manifests)
2014-11-18 10:42:46: burp[4847] Backup ending - disconnect from client.
2014-11-18 10:42:46: burp[4847] Begin phase4 (shuffle files)
2014-11-18 10:42:46: burp[4847] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-11-18 03:58:03
End time: 2014-11-18 10:43:08
Time taken: 06:45:05
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 69307 0 0 0 69307 | 69307
Directories: 4273 0 0 0 4273 | 4273
Soft links: 38 0 0 0 38 | 38
Special files: 19 0 0 0 19 | 19
Grand total: 73637 0 0 0 73637 | 73637
------------------------------------------------------------

Warnings: 0

Bytes estimated: 246568150349 (229.63 GB)
Bytes in backup: 246568152212 (229.63 GB)
Bytes received: 246568152212 (229.63 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-11-18 10:43:08: burp[4847] Backup completed.
2014-11-18 10:43:08: burp[4847] End phase4 (shuffle files)
</nowiki></pre>


'''Client 3 full backup 229.67GB'''

<pre><nowiki>
2014-11-18 23:32:58: burp[8331] Client version: 1.3.48
2014-11-18 23:32:58: burp[8331] Begin phase1 (file system scan)
2014-11-18 23:33:12: burp[8331] End phase1 (file system scan)
2014-11-18 23:33:12: burp[8331] Begin phase2 (receive file data)
2014-11-19 06:32:41: burp[8331] End phase2 (receive file data)
2014-11-19 06:32:41: burp[8331] Begin phase3 (merge manifests)
2014-11-19 06:32:52: burp[8331] End phase3 (merge manifests)
2014-11-19 06:32:52: burp[8331] Backup ending - disconnect from client.
2014-11-19 06:32:52: burp[8331] Begin phase4 (shuffle files)
2014-11-19 06:32:52: burp[8331] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-11-18 23:32:57
End time: 2014-11-19 06:33:17
Time taken: 07:00:20
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 69554 0 0 0 69554 | 69554
Directories: 4505 0 0 0 4505 | 4505
Soft links: 38 0 0 0 38 | 38
Special files: 19 0 0 0 19 | 19
Grand total: 74116 0 0 0 74116 | 74116
------------------------------------------------------------

Warnings: 0

Bytes estimated: 246605869782 (229.67 GB)
Bytes in backup: 246605902574 (229.67 GB)
Bytes received: 244132071693 (227.37 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-11-19 06:33:17: burp[8331] Backup completed.
2014-11-19 06:33:17: burp[8331] End phase4 (shuffle files)
</nowiki></pre>


== Old tests ==

There was a network configuration problem: `r8169` module was used instead of `r8168` on the client.

=== Test using client1 ===


{| border="1" cellpadding="2" cellspacing="0"
{| border="1" cellpadding="2" cellspacing="0"
Line 65: Line 865:
| Throughput (phase 2)
| Throughput (phase 2)
|-
|-
| First backup: 210Go
| First backup: 210GB
| None (`compression=gzip0`)
| None (`compression=gzip0`)
| (default parameter `ssl_compression` not set)
| (default parameter `ssl_compression` not set)
| 37h 09m 30s <span style="font-size: smaller">(18m 56s/36h 50m 34s)</span>
| 37h 09m 30s <span style="font-size: smaller">(18m 56s/36h 50m 34s)</span>
|align="right" | 1.6 Mo/s
|align="right" | 1.6 MB/s
|-
|-
| Second backup: 34.33Mo added
| Second backup: 34.33MB added
| None (`compression=gzip0`)
| None (`compression=gzip0`)
| (default parameter `ssl_compression` not set)
| (default parameter `ssl_compression` not set)
| 00h 18m 10s <span style="font-size: smaller">(16m 45s/1m 25s)</span>
| 00h 18m 10s <span style="font-size: smaller">(16m 45s/1m 25s)</span>
|align="right" | 0.4 Mo/s
|align="right" | 0.4 MB/s
|-
|-
| Third backup: 4.14Go (mp4 files) added
| Third backup: 4.14GB (mp4 files) added
| None (`compression=gzip0`)
| None (`compression=gzip0`)
| (default parameter `ssl_compression` not set)
| (default parameter `ssl_compression` not set)
| 01h 19m 54s <span style="font-size: smaller">(11m 56s/1h 07m 58s)</span>
| 01h 19m 54s <span style="font-size: smaller">(11m 56s/1h 07m 58s)</span>
|align="right" | 1.03 Mo/s
|align="right" | 1.03 MB/s
|-
|-
| Fourth backup: 4.24Go (mp4 files) added
| Fourth backup: 4.24GB (mp4 files) added
| None (`compression=gzip0`)
| None (`compression=gzip0`)
| `ssl_compression=gzip0`
| `ssl_compression=gzip0`
| 00h 58m 16s <span style="font-size: smaller">(16m 39s/41m 37s)</span>
| 00h 58m 16s <span style="font-size: smaller">(16m 39s/41m 37s)</span>
|align="right" | 1.73 Mo/s
|align="right" | 1.73 MB/s
|}
|}


==== Test using client2 ====
=== Test using client2 ===


{| border="1" cellpadding="2" cellspacing="0"
{| border="1" cellpadding="2" cellspacing="0"
Line 103: Line 903:
| (default parameter `ssl_compression` not set)
| (default parameter `ssl_compression` not set)
| 37h 06m 53s <span style="font-size: smaller">(15m 40s/36h 51m 13s)</span>
| 37h 06m 53s <span style="font-size: smaller">(15m 40s/36h 51m 13s)</span>
|align="right" |
|align="right" |
|}

----

Latest revision as of 05:22, 22 April 2019

Burp tests

Description

Version: 1.3.48 (debian package 1.3.48-4~bpo70+1)

Hardware configuration

Type CPU SATA link Disk RAM
Server Marvell Armada 370 (ReadyNAS 102) `SATA link up 6.0 Gbps` 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) 512MB
Client1 VIA Nano L2007 1600MHz `SATA link up 1.5 Gbps` 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) 4GB
Client2 VIA Nano L2007 1600MHz `SATA link up 1.5 Gbps` 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) 4GB
Client3 Intel Core i7-3770 `SATA link up 3.0 Gbps` WDC WD1003FBYX-01Y7B1 4GB

All hosts have gigabit NICs with properly cabled CAT 6 cables and all are connected to a Gigabit switch (Netgear GS108E).

Ethernet controllers and modules:

  • client: `Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)` (controller), `r8168` (module)
  • server: `Marvell 88E1318 Gigabit Ethernet PHY` (controller), `pegasus` (module)

Performances

Network

  • `iperf` measures 890 Mbits/s between server and client.
  • Copying 244.31GB grom client to backup server using rsync took 36h 21m (`sent 189.41Gbytes - received 25.48Mbytes - 1.45Mbytes/sec`).
  • Copying 212GB from client1 (VIA PadLock usable by OpenSSL) to backup server using `scp` (cipher specification/MAC algorithm: `aes128-ctr`/`hmac-md5`) took 8h 1m 48s (7.50MB/s).
  • Copying 216GB from client3 to backup server using `scp` (cipher specification/MAC algorithm: `aes128-ctr`/`umac-64@openssh.com`) took 4h 45m 04s (12.93MB/s).

Encryption

  • SCP transfers of 698MB from the client1 (VIA PadLock usable by OpenSSL) to the server using different cipher and MAC algorithm (`aes128-ctr` and `hmac-md5` are the default settings on `Debian Wheezy`):
|| cipher specification || MAC algorithm || Throughput || Duration ||
|| aes128-ctr || hmac-md5         ||  5.9MB/s || 01:59 ||
|| aes128-ctr || hmac-sha2-256    || 10.7MB/s || 01:05 ||
|| aes128-ctr || hmac-sha2-256-96 || 10.7MB/s || 01:05 ||
|| aes128-ctr || hmac-sha1        || 11.3MB/s || 01:02 ||
|| aes128-ctr || hmac-sha1-96     || 11.3MB/s || 01:02 ||
|| aes128-ctr || hmac-sha2-512    ||  8.3MB/s || 01:24 ||
|| aes192-ctr || hmac-sha2-256    ||  9.4MB/s || 01:14 ||
|| aes256-ctr || hmac-sha2-256    ||  8.4MB/s || 01:23 ||
|| aes128-cbc || hmac-sha1        || 14.0MB/s || 00:50 ||
|| aes128-cbc || hmac-sha2-256    || 11.8MB/s || 00:59 ||
|| aes192-cbc || hmac-sha2-256    || 11.5MB/s || 01:01 ||

Test using client1

Description Compression (`compression`) <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> SSL compression (`ssl_compression`) Duration on client phase 1 phase 2 Data sent Throughput (phase 2) Duration on server phase 3
full backup: 209.17GB gzip9 (default value) zlib5 (default value) 34h 17m 34s 07m 18s 34h 10m 18s 167.4GB 1.4MB/s 06m 22s 02m 22s
full backup: 209.17GB None (`compression=gzip0`) zlib5 (default value) 14h 49m 29s 03m 57s 14h 45m 32s 209.17GB 4.0MB/s 07m 04s 01m 09s
full backup: 214.27GB None (`compression=gzip0`) zlib5 (default value) 30h 05m 44s 7m 45s 29h 57m 59s 214.27GB 2.0MB/s 06m 56s 01m 01s
full backup: 212.47GB None (`compression=gzip0`) None (`ssl_compression=zlib0`) 24h 19m 53s 04m 36s 24h 15m 17s 212.47GB 2.5MB/s 06m 57s 54s
full backup: 213.99GB None (`compression=gzip0`) None (`ssl_compression=zlib0`) 30h 00m 45s 05m 47s 29h 54m 58s 213.99GB 2.0MB/s 06m 59s 01m 05s
incremental backup: 1.5GB added None (`compression=gzip0`) None (`ssl_compression=zlib0`) 11m 30s 04m 36s 06m 54s 1.5GB 3.7MB/s 16m 28s 01m 01s

Test using client2: full backups

Description librsync SSL algo `compression` <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> `ssl_compression` Duration on client phase 1 phase 2 Data sent Throughput (phase 2) Duration on server phase 3
227.44GB Enabled<<FootNote(Note: default value for librsync)>> AES128-SHA gzip0 zlib5 (default value) 28h 17m 41s 08m 51s 28h 08m 50s 227.44GB 2.29MB/s 09m 37s 01m 21s
227.45GB Enabled DHE-RSA-AES256-GCM-SHA384 gzip9 (default value) zlib5 (default value) 45h 02m 55s 06m 28s 44h 56m 27s 176.91GB 1.11MB/s 12m 02s 03m 08s
227.48GB Enabled DHE-RSA-AES256-GCM-SHA384 gzip9 (default value) zlib0 44h 57m 35s 06m 26s 44h 51m 09s 176.94GB 1.12MB/s 11m 32s 03m 06s
227.49GB Enabled DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 28h 19m 06s 09m 39s 28h 09m 37s 227.49GB 2.29MB/s 09m 37s 01m 24s
227.52GB Disabled<<FootNote(Note: librsync=0)>> DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 28h 40m 12s 11m 30s 28h 28m 42s 227.52GB 2.27MB/s 09m 43s 01m 23s
227.63GB Disabled<<FootNote(Note: librsync=0)>> DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 28h 10m 46s 10m 17s 28h 00m 29s 227.63GB 2.31MB/s 09m 58s 01m 26s
227.65GB Disabled<<FootNote(Note: librsync=0)>> DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 17h 56m 29s 10m 38s 17h 45m 51s 227.65GB 3.64MB/s 09m 58s 01m 25s
227.65GB Disabled<<FootNote(Note: librsync=0)>> DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 17h 24m 56s 03m 50s 17h 21m 06s 227.65GB 3.73MB/s 09m 51s 01m 27s

Test using client3: full backups

Description librsync SSL algo `compression` <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> `ssl_compression` Duration on client phase 1 phase 2 Data sent Throughput (phase 2) Duration on server phase 3
229.63GB Enabled<<FootNote(Note: default value for librsync)>> DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 06h 44m 39s 00m 15s 06h 44m 24s 229.63GB 9.68MB/s 00m 26s 00m 04s
229.67GB Enabled<<FootNote(Note: default value for librsync)>> DHE-RSA-AES256-GCM-SHA384 gzip9 zlib5 (default value) 06h 59m 43s 00m 14s 06h 59m 29s 227.37GB 9.25MB/s 00m 36s 00m 11s

Test using client2: restore one file

/usr/sbin/burp -c /tmp/burp-client.conf -a r -b 1 -r '^/mnt/data/Documents/file.pdf$' -d /tmp/ -s 4 -b 30


Size of backups: 249GB

Test using client2: incremental backup

Time taken
31m 48s
39m 59s
37m 39s
39m 00s
39m 13s
38m 35s
38m 37s
37m 37s
37m 52s
42m 15s
37m 50s
41m 40s
43m 46s
42m 56s
42m 16s
42m 27s
43m 02s
42m 18s
44m 41s
41m 15s
38m 30s
40m 16s
39m 40s
41m 17s
39m 15s
40m 42s

Logs

Client1 full backup: 214.27GB 

2014-07-19 18:32:40: burp[9606] Client version: 1.3.48
2014-07-19 18:32:40: burp[9606] Begin phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] End phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] Begin phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] End phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] Begin phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] End phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] Backup ending - disconnect from client.
2014-07-21 00:39:25: burp[9606] Begin phase4 (shuffle files)
2014-07-21 00:39:25: burp[9606] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-19 18:32:39
  End time: 2014-07-21 00:45:20
Time taken: 01:06:12:41
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:    998898         0         0         0    998898 |   998898
      Directories:     74226         0         0         0     74226 |    74226
       Soft links:      4531         0         0         0      4531 |     4531
       Hard links:        16         0         0         0        16 |       16
      Grand total:   1077671         0         0         0   1077671 |  1077671
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   230074235176 (214.27 GB)
      Bytes in backup:   230074235177 (214.27 GB)
       Bytes received:   230074235177 (214.27 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-07-21 00:45:20: burp[9606] Backup completed.
2014-07-21 00:45:20: burp[9606] End phase4 (shuffle files)


Client 1 full backup 212.47GB 

2014-07-14 23:49:34: burp[5369] Client version: 1.3.48
2014-07-14 23:49:34: burp[5369] Begin phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] End phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] Begin phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] End phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] Begin phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] End phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] Backup ending - disconnect from client.
2014-07-16 01:10:31: burp[5369] Begin phase4 (shuffle files)
2014-07-16 01:10:31: burp[5369] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-14 23:49:26
  End time: 2014-07-16 01:16:34
Time taken: 01:01:27:08
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:    998898         0         0         0    998898 |   998898
      Directories:     74226         0         0         0     74226 |    74226
       Soft links:      4531         0         0         0      4531 |     4531
       Hard links:        16         0         0         0        16 |       16
      Grand total:   1077671         0         0         0   1077671 |  1077671
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   228124139118 (212.46 GB)
      Bytes in backup:   228135345774 (212.47 GB)
       Bytes received:   228135345774 (212.47 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-07-16 01:16:34: burp[5369] Backup completed.
2014-07-16 01:16:34: burp[5369] End phase4 (shuffle files)


Client 1 full backup 213.99GB 

2014-07-17 01:47:37: burp[4654] Client version: 1.3.48
2014-07-17 01:47:37: burp[4654] Begin phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] End phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] Begin phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] End phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] Begin phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] End phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] Backup ending - disconnect from client.
2014-07-18 07:49:27: burp[4654] Begin phase4 (shuffle files)
2014-07-18 07:49:27: burp[4654] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-17 01:47:36
  End time: 2014-07-18 07:55:21
Time taken: 01:06:07:45
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:    998898         0         0         0    998898 |   998898
      Directories:     74226         0         0         0     74226 |    74226
       Soft links:      4531         0         0         0      4531 |     4531
       Hard links:        16         0         0         0        16 |       16
      Grand total:   1077671         0         0         0   1077671 |  1077671
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   228149960334 (212.48 GB)
      Bytes in backup:   229772435086 (213.99 GB)
       Bytes received:   229772435086 (213.99 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-07-18 07:55:21: burp[4654] Backup completed.
2014-07-18 07:55:21: burp[4654] End phase4 (shuffle files)


Client 2 full backup 227.44GB 

2014-08-23 15:18:35: burp[4691] Client version: 1.3.48
2014-08-23 15:18:35: burp[4691] Begin phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] End phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] Begin phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] End phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] Begin phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] End phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] Backup ending - disconnect from client.
2014-08-24 19:37:37: burp[4691] Begin phase4 (shuffle files)
2014-08-24 19:37:37: burp[4691] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-23 15:18:34
  End time: 2014-08-24 19:45:53
Time taken: 01:04:27:19
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1318031         0         0         0   1318031 |  1318286
      Directories:    104066         0         0         0    104066 |   104066
       Soft links:      4586         0         0         0      4586 |     4586
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1426782         0         0         0   1426782 |  1427037
                   ------------------------------------------------------------

             Warnings:           255

      Bytes estimated:   244227872076 (227.45 GB)
      Bytes in backup:   244212565354 (227.44 GB)
       Bytes received:   244212565354 (227.44 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-08-24 19:45:53: burp[4691] Backup completed.
2014-08-24 19:45:53: burp[4691] End phase4 (shuffle files)


Client 2 full backup 227.45GB 

2014-08-25 01:23:42: burp[9427] Client version: 1.3.48
2014-08-25 01:23:42: burp[9427] Begin phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] End phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] Begin phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] End phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] Begin phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] End phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] Backup ending - disconnect from client.
2014-08-26 22:29:45: burp[9427] Begin phase4 (shuffle files)
2014-08-26 22:29:45: burp[9427] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-25 01:23:42
  End time: 2014-08-26 22:38:09
Time taken: 01:21:14:27
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1318211         0         0         0   1318211 |  1318211
      Directories:    104070         0         0         0    104070 |   104070
       Soft links:      4586         0         0         0      4586 |     4586
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1426966         0         0         0   1426966 |  1426966
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   244219700992 (227.45 GB)
      Bytes in backup:   244219700992 (227.45 GB)
       Bytes received:   189963848136 (176.92 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-08-26 22:38:09: burp[9427] Backup completed.
2014-08-26 22:38:09: burp[9427] End phase4 (shuffle files)


Client 2 full backup 227.49GB 

2014-08-30 17:26:31: burp[31130] Client version: 1.3.48
2014-08-30 17:26:31: burp[31130] Begin phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] End phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] Begin phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] End phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] Begin phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] End phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] Backup ending - disconnect from client.
2014-08-31 21:48:11: burp[31130] Begin phase4 (shuffle files)
2014-08-31 21:48:11: burp[31130] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-30 17:26:30
  End time: 2014-08-31 21:56:32
Time taken: 01:04:30:02
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1320475         0         0         0   1320475 |  1320484
      Directories:    104340         0         0         0    104340 |   104340
       Soft links:      4588         0         0         0      4588 |     4588
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1429502         0         0         0   1429502 |  1429511
                   ------------------------------------------------------------

             Warnings:             9

      Bytes estimated:   244267585965 (227.49 GB)
      Bytes in backup:   244267822787 (227.49 GB)
       Bytes received:   244267822787 (227.49 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-08-31 21:56:32: burp[31130] Backup completed.
2014-08-31 21:56:32: burp[31130] End phase4 (shuffle files)


Client 2 full backup 227.52GB 

2014-09-07 00:42:33: burp[22795] Client version: 1.3.48
2014-09-07 00:42:33: burp[22795] Begin phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] End phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] Begin phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] End phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] Begin phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] End phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] Backup ending - disconnect from client.
2014-09-08 05:24:08: burp[22795] Begin phase4 (shuffle files)
2014-09-08 05:24:08: burp[22795] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-07 00:42:32
  End time: 2014-09-08 05:32:25
Time taken: 01:04:49:53
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1322008         0         0         0   1322008 |  1322008
      Directories:    104595         0         0         0    104595 |   104595
       Soft links:      4588         0         0         0      4588 |     4588
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1431290         0         0         0   1431290 |  1431290
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   244299698290 (227.52 GB)
      Bytes in backup:   244299767377 (227.52 GB)
       Bytes received:   244299767377 (227.52 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-09-08 05:32:26: burp[22795] Backup completed.
2014-09-08 05:32:26: burp[22795] End phase4 (shuffle files)


Client 2 full backup 227.63GB 

2014-09-16 01:07:22: burp[3507] Client version: 1.3.48
2014-09-16 01:07:22: burp[3507] Begin phase1 (file system scan)
2014-09-16 01:17:39: burp[3507] End phase1 (file system scan)
2014-09-16 01:17:39: burp[3507] Begin phase2 (receive file data)
2014-09-17 05:18:08: burp[3507] End phase2 (receive file data)
2014-09-17 05:18:08: burp[3507] Begin phase3 (merge manifests)
2014-09-17 05:19:34: burp[3507] End phase3 (merge manifests)
2014-09-17 05:19:34: burp[3507] Backup ending - disconnect from client.
2014-09-17 05:19:34: burp[3507] Begin phase4 (shuffle files)
2014-09-17 05:19:34: burp[3507] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-16 01:07:20
  End time: 2014-09-17 05:28:06
Time taken: 01:04:20:46
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1334116         0         0         0   1334116 |  1334117
      Directories:    110080         0         0         0    110080 |   110080
       Soft links:      5461         0         0         0      5461 |     5461
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1449756         0         0         0   1449756 |  1449757
                   ------------------------------------------------------------

             Warnings:             1

      Bytes estimated:   244419793362 (227.63 GB)
      Bytes in backup:   244419785986 (227.63 GB)
       Bytes received:   244419785986 (227.63 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-09-17 05:28:06: burp[3507] Backup completed.
2014-09-17 05:28:06: burp[3507] End phase4 (shuffle files)


Client 2 full backup 227.65GB 


2014-10-15 03:17:06: burp[31867] Client version: 1.3.48
2014-10-15 03:17:06: burp[31867] Begin phase1 (file system scan)
2014-10-15 03:27:44: burp[31867] End phase1 (file system scan)
2014-10-15 03:27:44: burp[31867] Begin phase2 (receive file data)
2014-10-15 21:13:35: burp[31867] End phase2 (receive file data)
2014-10-15 21:13:35: burp[31867] Begin phase3 (merge manifests)
2014-10-15 21:15:00: burp[31867] End phase3 (merge manifests)
2014-10-15 21:15:00: burp[31867] Backup ending - disconnect from client.
2014-10-15 21:15:00: burp[31867] Begin phase4 (shuffle files)
2014-10-15 21:15:00: burp[31867] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-10-15 03:17:05
  End time: 2014-10-15 21:23:32
Time taken: 18:06:27
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1334339         0         0         0   1334339 |  1334339
      Directories:    110116         0         0         0    110116 |   110116
       Soft links:      5461         0         0         0      5461 |     5461
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1450015         0         0         0   1450015 |  1450015
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   244439612263 (227.65 GB)
      Bytes in backup:   244439612178 (227.65 GB)
       Bytes received:   244439612178 (227.65 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-10-15 21:23:33: burp[31867] Backup completed.
2014-10-15 21:23:33: burp[31867] End phase4 (shuffle files)


Client 2 full backup 227.65GB 

2014-10-16 12:32:02: burp[3210] Client version: 1.3.48
2014-10-16 12:32:02: burp[3210] Begin phase1 (file system scan)
2014-10-16 12:35:52: burp[3210] End phase1 (file system scan)
2014-10-16 12:35:52: burp[3210] Begin phase2 (receive file data)
2014-10-17 05:57:58: burp[3210] End phase2 (receive file data)
2014-10-17 05:57:58: burp[3210] Begin phase3 (merge manifests)
2014-10-17 05:59:25: burp[3210] End phase3 (merge manifests)
2014-10-17 05:59:25: burp[3210] Backup ending - disconnect from client.
2014-10-17 05:59:25: burp[3210] Begin phase4 (shuffle files)
2014-10-17 05:59:25: burp[3210] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-10-16 12:32:00
  End time: 2014-10-17 06:07:50
Time taken: 17:35:50
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1334340         0         0         0   1334340 |  1334340
      Directories:    110116         0         0         0    110116 |   110116
       Soft links:      5461         0         0         0      5461 |     5461
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1450016         0         0         0   1450016 |  1450016
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   244439619502 (227.65 GB)
      Bytes in backup:   244439631290 (227.65 GB)
       Bytes received:   244439631290 (227.65 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-10-17 06:07:51: burp[3210] Backup completed.
2014-10-17 06:07:51: burp[3210] End phase4 (shuffle files)


Client 3 full backup 229.63GB 

2014-11-18 03:58:03: burp[4847] Client version: 1.3.48
2014-11-18 03:58:03: burp[4847] Begin phase1 (file system scan)
2014-11-18 03:58:18: burp[4847] End phase1 (file system scan)
2014-11-18 03:58:18: burp[4847] Begin phase2 (receive file data)
2014-11-18 10:42:42: burp[4847] End phase2 (receive file data)
2014-11-18 10:42:42: burp[4847] Begin phase3 (merge manifests)
2014-11-18 10:42:46: burp[4847] End phase3 (merge manifests)
2014-11-18 10:42:46: burp[4847] Backup ending - disconnect from client.
2014-11-18 10:42:46: burp[4847] Begin phase4 (shuffle files)
2014-11-18 10:42:46: burp[4847] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-11-18 03:58:03
  End time: 2014-11-18 10:43:08
Time taken: 06:45:05
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:     69307         0         0         0     69307 |    69307
      Directories:      4273         0         0         0      4273 |     4273
       Soft links:        38         0         0         0        38 |       38
    Special files:        19         0         0         0        19 |       19
      Grand total:     73637         0         0         0     73637 |    73637
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   246568150349 (229.63 GB)
      Bytes in backup:   246568152212 (229.63 GB)
       Bytes received:   246568152212 (229.63 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-11-18 10:43:08: burp[4847] Backup completed.
2014-11-18 10:43:08: burp[4847] End phase4 (shuffle files)


Client 3 full backup 229.67GB 

2014-11-18 23:32:58: burp[8331] Client version: 1.3.48
2014-11-18 23:32:58: burp[8331] Begin phase1 (file system scan)
2014-11-18 23:33:12: burp[8331] End phase1 (file system scan)
2014-11-18 23:33:12: burp[8331] Begin phase2 (receive file data)
2014-11-19 06:32:41: burp[8331] End phase2 (receive file data)
2014-11-19 06:32:41: burp[8331] Begin phase3 (merge manifests)
2014-11-19 06:32:52: burp[8331] End phase3 (merge manifests)
2014-11-19 06:32:52: burp[8331] Backup ending - disconnect from client.
2014-11-19 06:32:52: burp[8331] Begin phase4 (shuffle files)
2014-11-19 06:32:52: burp[8331] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-11-18 23:32:57
  End time: 2014-11-19 06:33:17
Time taken: 07:00:20
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:     69554         0         0         0     69554 |    69554
      Directories:      4505         0         0         0      4505 |     4505
       Soft links:        38         0         0         0        38 |       38
    Special files:        19         0         0         0        19 |       19
      Grand total:     74116         0         0         0     74116 |    74116
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   246605869782 (229.67 GB)
      Bytes in backup:   246605902574 (229.67 GB)
       Bytes received:   244132071693 (227.37 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-11-19 06:33:17: burp[8331] Backup completed.
2014-11-19 06:33:17: burp[8331] End phase4 (shuffle files)


Old tests

There was a network configuration problem: `r8169` module was used instead of `r8168` on the client.

Test using client1

Description Compression SSL compression Duration on client (phase 1/phase 2) Throughput (phase 2)
First backup: 210GB None (`compression=gzip0`) (default parameter `ssl_compression` not set) 37h 09m 30s (18m 56s/36h 50m 34s) 1.6 MB/s
Second backup: 34.33MB added None (`compression=gzip0`) (default parameter `ssl_compression` not set) 00h 18m 10s (16m 45s/1m 25s) 0.4 MB/s
Third backup: 4.14GB (mp4 files) added None (`compression=gzip0`) (default parameter `ssl_compression` not set) 01h 19m 54s (11m 56s/1h 07m 58s) 1.03 MB/s
Fourth backup: 4.24GB (mp4 files) added None (`compression=gzip0`) `ssl_compression=gzip0` 00h 58m 16s (16m 39s/41m 37s) 1.73 MB/s

Test using client2

Description Compression SSL compression Duration on client (phase 1/phase 2) Throughput (phase 2)
First backup: None (`compression=gzip0`) (default parameter `ssl_compression` not set) 37h 06m 53s (15m 40s/36h 51m 13s)
Retrieved from "https://wiki.duckcorp.org//index.php?title=Burp&oldid=237"