Burp
From DcSharedWiki
Burp tests
Description
Version: 1.3.48 (debian package 1.3.48-4~bpo70+1)
Hardware configuration
| Type | CPU | Disk | RAM |
| Server | Marvell Armada 370 (ReadyNAS 104) | 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) | 512Mo |
| Client1 | VIA Nano L2007 1600MHz | 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) | 4Go |
| Client2 | VIA Nano L2007 1600MHz | 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) | 4Go |
All hosts have gigabit NICs with properly cabled CAT 6 cables and all are connected to a Gigabit switch (Netgear GS108E).
Ethernet controllers and modules:
- client: `Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)` (controller), `r8168` (module)
- server: `Marvell 88E1318 Gigabit Ethernet PHY` (controller), `pegasus` (module)
Performances
Network
- `iperf` measures 890 Mbits/s between server and client.
- Copying 244.31Go grom client to backup server using rsync took 36h 21m (`sent 189.41G bytes - received 25.48Mbytes - 1.45Mbytes/sec`).
- Copying 212Go from client1 (VIA PadLock usable by OpenSSL) to backup server using `scp` (default cipher specification and MAC algorithm) took 8h 1m 48s (7.50Mo/s).
Encryption
- SCP transfers of 698Mo from the client1 (VIA PadLock usable by OpenSSL) to the server using different cipher and MAC algorithm (`aes128-ctr` and `hmac-md5` are the default settings on `Debian Wheezy`):
|| cipher specification || MAC algorithm || Throughput || Duration || || aes128-ctr || hmac-md5 || 5.9MB/s || 01:59 || || aes128-ctr || hmac-sha2-256 || 10.7MB/s || 01:05 || || aes128-ctr || hmac-sha2-256-96 || 10.7MB/s || 01:05 || || aes128-ctr || hmac-sha1 || 11.3MB/s || 01:02 || || aes128-ctr || hmac-sha1-96 || 11.3MB/s || 01:02 || || aes128-ctr || hmac-sha2-512 || 8.3MB/s || 01:24 || || aes192-ctr || hmac-sha2-256 || 9.4MB/s || 01:14 || || aes256-ctr || hmac-sha2-256 || 8.4MB/s || 01:23 || || aes128-cbc || hmac-sha1 || 14.0MB/s || 00:50 || || aes128-cbc || hmac-sha2-256 || 11.8MB/s || 00:59 || || aes192-cbc || hmac-sha2-256 || 11.5MB/s || 01:01 ||
Test using client1
| Description | Compression (`compression`) <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> | SSL compression (`ssl_compression`) | Duration on client | phase 1 | phase 2 | Data send | Throughput (phase 2) | Duration on server | phase 3 |
| full backup: 209.17Go | gzip9 (default value) | zlib5 (default value) | 34h 17m 34s | 07m 18s | 34h 10m 18s | 167.4Go | 1.4Mo/s | 06m 22s | 02m 22s |
| full backup: 209.17Go | None (`compression=gzip0`) | zlib5 (default value) | 14h 49m 29s | 03m 57s | 14h 45m 32s | 209.17Go | 4.0Mo/s | 07m 04s | 01m 09s |
| full backup: 214.27Go | None (`compression=gzip0`) | zlib5 (default value) | 30h 05m 44s | 7m 45s | 29h 57m 59s | 214.27Go | 2.0Mo/s | 06m 56s | 01m 01s |
| full backup: 212.47Go | None (`compression=gzip0`) | None (`ssl_compression=zlib0`) | 24h 19m 53s | 04m 36s | 24h 15m 17s | 212.47Go | 2.5Mo/s | 06m 57s | 54s |
| full backup: 213.99Go | None (`compression=gzip0`) | None (`ssl_compression=zlib0`) | 30h 00m 45s | 05m 47s | 29h 54m 58s | 213.99Go | 2.0Mo/s | 06m 59s | 01m 05s |
| incremental backup: 1.5Go added | None (`compression=gzip0`) | None (`ssl_compression=zlib0`) | 11m 30s | 04m 36s | 06m 54s | 1.5Go | 3.7Mo/s | 16m 28s | 01m 01s |
Test using client2: full backups
| Description | librsync | SSL algo | `compression` <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> | `ssl_compression` | Duration on client | phase 1 | phase 2 | Data send | Throughput (phase 2) | Duration on server | phase 3 |
| 227.44Go | Enabled<<FootNote(Note: default value for librsync)>> | AES128-SHA | gzip0 | zlib5 (default value) | 28h 17m 41s | 08m 51s | 28h 08m 50s | 227.44Go | 2.29Mo/s | 09m 37s | 01m 21s |
| 227.45Go | Enabled | DHE-RSA-AES256-GCM-SHA384 | gzip9 (default value) | zlib5 (default value) | 45h 02m 55s | 06m 28s | 44h 56m 27s | 176.91Go | 1.11Mo/s | 12m 02s | 03m 08s |
| 227.48Go | Enabled | DHE-RSA-AES256-GCM-SHA384 | gzip9 (default value) | zlib0 | 44h 57m 35s | 06m 26s | 44h 51m 09s | 176.94Go | 1.12Mo/s | 11m 32s | 03m 06s |
| 227.49Go | Enabled | DHE-RSA-AES256-GCM-SHA384 | gzip0 | zlib5 (default value) | 28h 19m 06s | 09m 39s | 28h 09m 37s | 227.49Go | 2.29Mo/s | 09m 37s | 01m 24s |
| 227.52Go | Disabled<<FootNote(Note: librsync=0)>> | DHE-RSA-AES256-GCM-SHA384 | gzip0 | zlib5 (default value) | 28h 40m 12s | 11m 30s | 28h 28m 42s | 227.52Go | 2.27Mo/s | 09m 43s | 01m 23s |
| 227.63Go | Disabled<<FootNote(Note: librsync=0)>> | DHE-RSA-AES256-GCM-SHA384 | gzip0 | zlib5 (default value) | 28h 10m 46s | 10m 17s | 28h 00m 29s | 227.63Go | 2.31Mo/s | 09m 58s | 01m 26s |
| 227.65Go | Disabled<<FootNote(Note: librsync=0)>> | DHE-RSA-AES256-GCM-SHA384 | gzip0 | zlib5 (default value) | 17h 56m 29s | 10m 38s | 17h 45m 51s | 227.65Go | 3.64Mo/s | 09m 58s | 01m 25s |
Logs
Client1 full backup: 214.27Go
2014-07-19 18:32:40: burp[9606] Client version: 1.3.48
2014-07-19 18:32:40: burp[9606] Begin phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] End phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] Begin phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] End phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] Begin phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] End phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] Backup ending - disconnect from client.
2014-07-21 00:39:25: burp[9606] Begin phase4 (shuffle files)
2014-07-21 00:39:25: burp[9606] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-19 18:32:39
End time: 2014-07-21 00:45:20
Time taken: 01:06:12:41
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 998898 0 0 0 998898 | 998898
Directories: 74226 0 0 0 74226 | 74226
Soft links: 4531 0 0 0 4531 | 4531
Hard links: 16 0 0 0 16 | 16
Grand total: 1077671 0 0 0 1077671 | 1077671
------------------------------------------------------------
Warnings: 0
Bytes estimated: 230074235176 (214.27 GB)
Bytes in backup: 230074235177 (214.27 GB)
Bytes received: 230074235177 (214.27 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-07-21 00:45:20: burp[9606] Backup completed.
2014-07-21 00:45:20: burp[9606] End phase4 (shuffle files)
Client 1 full backup 212.47Go
2014-07-14 23:49:34: burp[5369] Client version: 1.3.48
2014-07-14 23:49:34: burp[5369] Begin phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] End phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] Begin phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] End phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] Begin phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] End phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] Backup ending - disconnect from client.
2014-07-16 01:10:31: burp[5369] Begin phase4 (shuffle files)
2014-07-16 01:10:31: burp[5369] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-14 23:49:26
End time: 2014-07-16 01:16:34
Time taken: 01:01:27:08
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 998898 0 0 0 998898 | 998898
Directories: 74226 0 0 0 74226 | 74226
Soft links: 4531 0 0 0 4531 | 4531
Hard links: 16 0 0 0 16 | 16
Grand total: 1077671 0 0 0 1077671 | 1077671
------------------------------------------------------------
Warnings: 0
Bytes estimated: 228124139118 (212.46 GB)
Bytes in backup: 228135345774 (212.47 GB)
Bytes received: 228135345774 (212.47 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-07-16 01:16:34: burp[5369] Backup completed.
2014-07-16 01:16:34: burp[5369] End phase4 (shuffle files)
Client 1 full backup 213.99Go
2014-07-17 01:47:37: burp[4654] Client version: 1.3.48
2014-07-17 01:47:37: burp[4654] Begin phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] End phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] Begin phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] End phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] Begin phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] End phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] Backup ending - disconnect from client.
2014-07-18 07:49:27: burp[4654] Begin phase4 (shuffle files)
2014-07-18 07:49:27: burp[4654] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-17 01:47:36
End time: 2014-07-18 07:55:21
Time taken: 01:06:07:45
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 998898 0 0 0 998898 | 998898
Directories: 74226 0 0 0 74226 | 74226
Soft links: 4531 0 0 0 4531 | 4531
Hard links: 16 0 0 0 16 | 16
Grand total: 1077671 0 0 0 1077671 | 1077671
------------------------------------------------------------
Warnings: 0
Bytes estimated: 228149960334 (212.48 GB)
Bytes in backup: 229772435086 (213.99 GB)
Bytes received: 229772435086 (213.99 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-07-18 07:55:21: burp[4654] Backup completed.
2014-07-18 07:55:21: burp[4654] End phase4 (shuffle files)
Client 2 full backup 227.44Go
2014-08-23 15:18:35: burp[4691] Client version: 1.3.48
2014-08-23 15:18:35: burp[4691] Begin phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] End phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] Begin phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] End phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] Begin phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] End phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] Backup ending - disconnect from client.
2014-08-24 19:37:37: burp[4691] Begin phase4 (shuffle files)
2014-08-24 19:37:37: burp[4691] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-23 15:18:34
End time: 2014-08-24 19:45:53
Time taken: 01:04:27:19
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1318031 0 0 0 1318031 | 1318286
Directories: 104066 0 0 0 104066 | 104066
Soft links: 4586 0 0 0 4586 | 4586
Hard links: 99 0 0 0 99 | 99
Grand total: 1426782 0 0 0 1426782 | 1427037
------------------------------------------------------------
Warnings: 255
Bytes estimated: 244227872076 (227.45 GB)
Bytes in backup: 244212565354 (227.44 GB)
Bytes received: 244212565354 (227.44 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-08-24 19:45:53: burp[4691] Backup completed.
2014-08-24 19:45:53: burp[4691] End phase4 (shuffle files)
Client 2 full backup 227.45Go
2014-08-25 01:23:42: burp[9427] Client version: 1.3.48
2014-08-25 01:23:42: burp[9427] Begin phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] End phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] Begin phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] End phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] Begin phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] End phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] Backup ending - disconnect from client.
2014-08-26 22:29:45: burp[9427] Begin phase4 (shuffle files)
2014-08-26 22:29:45: burp[9427] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-25 01:23:42
End time: 2014-08-26 22:38:09
Time taken: 01:21:14:27
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1318211 0 0 0 1318211 | 1318211
Directories: 104070 0 0 0 104070 | 104070
Soft links: 4586 0 0 0 4586 | 4586
Hard links: 99 0 0 0 99 | 99
Grand total: 1426966 0 0 0 1426966 | 1426966
------------------------------------------------------------
Warnings: 0
Bytes estimated: 244219700992 (227.45 GB)
Bytes in backup: 244219700992 (227.45 GB)
Bytes received: 189963848136 (176.92 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-08-26 22:38:09: burp[9427] Backup completed.
2014-08-26 22:38:09: burp[9427] End phase4 (shuffle files)
Client 2 full backup 227.49Go
2014-08-30 17:26:31: burp[31130] Client version: 1.3.48
2014-08-30 17:26:31: burp[31130] Begin phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] End phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] Begin phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] End phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] Begin phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] End phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] Backup ending - disconnect from client.
2014-08-31 21:48:11: burp[31130] Begin phase4 (shuffle files)
2014-08-31 21:48:11: burp[31130] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-30 17:26:30
End time: 2014-08-31 21:56:32
Time taken: 01:04:30:02
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1320475 0 0 0 1320475 | 1320484
Directories: 104340 0 0 0 104340 | 104340
Soft links: 4588 0 0 0 4588 | 4588
Hard links: 99 0 0 0 99 | 99
Grand total: 1429502 0 0 0 1429502 | 1429511
------------------------------------------------------------
Warnings: 9
Bytes estimated: 244267585965 (227.49 GB)
Bytes in backup: 244267822787 (227.49 GB)
Bytes received: 244267822787 (227.49 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-08-31 21:56:32: burp[31130] Backup completed.
2014-08-31 21:56:32: burp[31130] End phase4 (shuffle files)
Client 2 full backup 227.52Go
2014-09-07 00:42:33: burp[22795] Client version: 1.3.48
2014-09-07 00:42:33: burp[22795] Begin phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] End phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] Begin phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] End phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] Begin phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] End phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] Backup ending - disconnect from client.
2014-09-08 05:24:08: burp[22795] Begin phase4 (shuffle files)
2014-09-08 05:24:08: burp[22795] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-07 00:42:32
End time: 2014-09-08 05:32:25
Time taken: 01:04:49:53
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1322008 0 0 0 1322008 | 1322008
Directories: 104595 0 0 0 104595 | 104595
Soft links: 4588 0 0 0 4588 | 4588
Hard links: 99 0 0 0 99 | 99
Grand total: 1431290 0 0 0 1431290 | 1431290
------------------------------------------------------------
Warnings: 0
Bytes estimated: 244299698290 (227.52 GB)
Bytes in backup: 244299767377 (227.52 GB)
Bytes received: 244299767377 (227.52 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-09-08 05:32:26: burp[22795] Backup completed.
2014-09-08 05:32:26: burp[22795] End phase4 (shuffle files)
Client 2 full backup 227.63Go
2014-09-16 01:07:22: burp[3507] Client version: 1.3.48
2014-09-16 01:07:22: burp[3507] Begin phase1 (file system scan)
2014-09-16 01:17:39: burp[3507] End phase1 (file system scan)
2014-09-16 01:17:39: burp[3507] Begin phase2 (receive file data)
2014-09-17 05:18:08: burp[3507] End phase2 (receive file data)
2014-09-17 05:18:08: burp[3507] Begin phase3 (merge manifests)
2014-09-17 05:19:34: burp[3507] End phase3 (merge manifests)
2014-09-17 05:19:34: burp[3507] Backup ending - disconnect from client.
2014-09-17 05:19:34: burp[3507] Begin phase4 (shuffle files)
2014-09-17 05:19:34: burp[3507] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-16 01:07:20
End time: 2014-09-17 05:28:06
Time taken: 01:04:20:46
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1334116 0 0 0 1334116 | 1334117
Directories: 110080 0 0 0 110080 | 110080
Soft links: 5461 0 0 0 5461 | 5461
Hard links: 99 0 0 0 99 | 99
Grand total: 1449756 0 0 0 1449756 | 1449757
------------------------------------------------------------
Warnings: 1
Bytes estimated: 244419793362 (227.63 GB)
Bytes in backup: 244419785986 (227.63 GB)
Bytes received: 244419785986 (227.63 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-09-17 05:28:06: burp[3507] Backup completed.
2014-09-17 05:28:06: burp[3507] End phase4 (shuffle files)
Client 2 full backup 227.65Go
2014-10-15 03:17:06: burp[31867] Client version: 1.3.48
2014-10-15 03:17:06: burp[31867] Begin phase1 (file system scan)
2014-10-15 03:27:44: burp[31867] End phase1 (file system scan)
2014-10-15 03:27:44: burp[31867] Begin phase2 (receive file data)
2014-10-15 21:13:35: burp[31867] End phase2 (receive file data)
2014-10-15 21:13:35: burp[31867] Begin phase3 (merge manifests)
2014-10-15 21:15:00: burp[31867] End phase3 (merge manifests)
2014-10-15 21:15:00: burp[31867] Backup ending - disconnect from client.
2014-10-15 21:15:00: burp[31867] Begin phase4 (shuffle files)
2014-10-15 21:15:00: burp[31867] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-10-15 03:17:05
End time: 2014-10-15 21:23:32
Time taken: 18:06:27
New Changed Unchanged Deleted Total | Scanned
------------------------------------------------------------
Files: 1334339 0 0 0 1334339 | 1334339
Directories: 110116 0 0 0 110116 | 110116
Soft links: 5461 0 0 0 5461 | 5461
Hard links: 99 0 0 0 99 | 99
Grand total: 1450015 0 0 0 1450015 | 1450015
------------------------------------------------------------
Warnings: 0
Bytes estimated: 244439612263 (227.65 GB)
Bytes in backup: 244439612178 (227.65 GB)
Bytes received: 244439612178 (227.65 GB)
Bytes sent: 0
--------------------------------------------------------------------------------
2014-10-15 21:23:33: burp[31867] Backup completed.
2014-10-15 21:23:33: burp[31867] End phase4 (shuffle files)
Old tests
There was a network configuration problem: `r8169` module was used instead of `r8168` on the client.
Test using client1
| Description | Compression | SSL compression | Duration on client (phase 1/phase 2) | Throughput (phase 2) |
| First backup: 210Go | None (`compression=gzip0`) | (default parameter `ssl_compression` not set) | 37h 09m 30s (18m 56s/36h 50m 34s) | 1.6 Mo/s |
| Second backup: 34.33Mo added | None (`compression=gzip0`) | (default parameter `ssl_compression` not set) | 00h 18m 10s (16m 45s/1m 25s) | 0.4 Mo/s |
| Third backup: 4.14Go (mp4 files) added | None (`compression=gzip0`) | (default parameter `ssl_compression` not set) | 01h 19m 54s (11m 56s/1h 07m 58s) | 1.03 Mo/s |
| Fourth backup: 4.24Go (mp4 files) added | None (`compression=gzip0`) | `ssl_compression=gzip0` | 00h 58m 16s (16m 39s/41m 37s) | 1.73 Mo/s |
Test using client2
| Description | Compression | SSL compression | Duration on client (phase 1/phase 2) | Throughput (phase 2) |
| First backup: | None (`compression=gzip0`) | (default parameter `ssl_compression` not set) | 37h 06m 53s (15m 40s/36h 51m 13s) |
