Burp: Difference between revisions

From DcSharedWiki
(Add new test)
(Add librsync parameter)
Line 141: Line 141:
|}
|}


== Test using client2 ==
== Test using client2: full backups ==


{| border="1" cellpadding="2" cellspacing="0"
{| border="1" cellpadding="2" cellspacing="0"
| Description
| Description
| librsync
| SSL algo
| SSL algo
| `compression` <<[[FootNote]](Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>>
| `compression` <<[[FootNote]](Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>>
Line 156: Line 157:
| phase 3
| phase 3
|-
|-
| full backup: 227.44Go
| 227.44Go
| Enabled<<[[FootNote]](Note: default value for librsync)>>
| AES128-SHA
| AES128-SHA
| gzip0
| gzip0
Line 168: Line 170:
|align="right" | 01m 21s
|align="right" | 01m 21s
|-
|-
| full backup: 227.45Go
| 227.45Go
| Enabled
| DHE-RSA-AES256-GCM-SHA384
| DHE-RSA-AES256-GCM-SHA384
| gzip9 (default value)
| gzip9 (default value)
Line 180: Line 183:
|align="right" | 03m 08s
|align="right" | 03m 08s
|-
|-
| full backup: 227.48Go
| 227.48Go
| Enabled
| DHE-RSA-AES256-GCM-SHA384
| DHE-RSA-AES256-GCM-SHA384
| gzip9 (default value)
| gzip9 (default value)
Line 192: Line 196:
|align="right" | 03m 06s
|align="right" | 03m 06s
|-
|-
| full backup: 227.49Go
| 227.49Go
| Enabled
| DHE-RSA-AES256-GCM-SHA384
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| gzip0
Line 204: Line 209:
|align="right" | 01m 24s
|align="right" | 01m 24s
|-
|-
| full backup: 227.52Go
| 227.52Go
| Disabled<<[[FootNote]](Note: librsync=0)>>
| DHE-RSA-AES256-GCM-SHA384
| DHE-RSA-AES256-GCM-SHA384
| gzip0
| gzip0

Revision as of 23:52, 15 September 2014

Burp tests

Description

Version: 1.3.48 (debian package 1.3.48-4~bpo70+1)

Hardware configuration

Type CPU Disk RAM
Server Marvell Armada 370 (ReadyNAS 104) 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) 512Mo
Client1 VIA Nano L2007 1600MHz 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) 4Go
Client2 VIA Nano L2007 1600MHz 2*WDC WD20EFRX-68EUZN0 (soft raid1 using mdadm) 4Go

All hosts have gigabit NICs with properly cabled CAT 6 cables and all are connected to a Gigabit switch (Netgear GS108E).

Ethernet controllers and modules:

  • client: `Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)` (controller), `r8168` (module)
  • server: `Marvell 88E1318 Gigabit Ethernet PHY` (controller), `pegasus` (module)

Performances

Network

  • `iperf` measures 890 Mbits/s between server and client.
  • Copying 244.31Go grom client to backup server using rsync took 36h 21m (`sent 189.41G bytes - received 25.48Mbytes - 1.45Mbytes/sec`).
  • Copying 212Go from client1 (VIA PadLock usable by OpenSSL) to backup server using `scp` (default cipher specification and MAC algorithm) took 8h 1m 48s (7.50Mo/s).

Encryption

  • SCP transfers of 698Mo from the client1 (VIA PadLock usable by OpenSSL) to the server using different cipher and MAC algorithm (`aes128-ctr` and `hmac-md5` are the default settings on `Debian Wheezy`):
|| cipher specification || MAC algorithm || Throughput || Duration ||
|| aes128-ctr || hmac-md5         ||  5.9MB/s || 01:59 ||
|| aes128-ctr || hmac-sha2-256    || 10.7MB/s || 01:05 ||
|| aes128-ctr || hmac-sha2-256-96 || 10.7MB/s || 01:05 ||
|| aes128-ctr || hmac-sha1        || 11.3MB/s || 01:02 ||
|| aes128-ctr || hmac-sha1-96     || 11.3MB/s || 01:02 ||
|| aes128-ctr || hmac-sha2-512    ||  8.3MB/s || 01:24 ||
|| aes192-ctr || hmac-sha2-256    ||  9.4MB/s || 01:14 ||
|| aes256-ctr || hmac-sha2-256    ||  8.4MB/s || 01:23 ||
|| aes128-cbc || hmac-sha1        || 14.0MB/s || 00:50 ||
|| aes128-cbc || hmac-sha2-256    || 11.8MB/s || 00:59 ||
|| aes192-cbc || hmac-sha2-256    || 11.5MB/s || 01:01 ||

Test using client1

Description Compression (`compression`) <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> SSL compression (`ssl_compression`) Duration on client phase 1 phase 2 Data send Throughput (phase 2) Duration on server phase 3
full backup: 209.17Go gzip9 (default value) zlib5 (default value) 34h 17m 34s 07m 18s 34h 10m 18s 167.4Go 1.4Mo/s 06m 22s 02m 22s
full backup: 209.17Go None (`compression=gzip0`) zlib5 (default value) 14h 49m 29s 03m 57s 14h 45m 32s 209.17Go 4.0Mo/s 07m 04s 01m 09s
full backup: 214.27Go None (`compression=gzip0`) zlib5 (default value) 30h 05m 44s 7m 45s 29h 57m 59s 214.27Go 2.0Mo/s 06m 56s 01m 01s
full backup: 212.47Go None (`compression=gzip0`) None (`ssl_compression=zlib0`) 24h 19m 53s 04m 36s 24h 15m 17s 212.47Go 2.5Mo/s 06m 57s 54s
full backup: 213.99Go None (`compression=gzip0`) None (`ssl_compression=zlib0`) 30h 00m 45s 05m 47s 29h 54m 58s 213.99Go 2.0Mo/s 06m 59s 01m 05s
incremental backup: 1.5Go added None (`compression=gzip0`) None (`ssl_compression=zlib0`) 11m 30s 04m 36s 06m 54s 1.5Go 3.7Mo/s 16m 28s 01m 01s

Test using client2: full backups

Description librsync SSL algo `compression` <<FootNote(Note: Extensions which are not compressed: bz, bz2, zip, gz, webm, mov, png, jpg, jpeg, rar, m4v, avi, xv, jar, tgz, mp4, mkv, ogg, flv)>> `ssl_compression` Duration on client phase 1 phase 2 Data send Throughput (phase 2) Duration on server phase 3
227.44Go Enabled<<FootNote(Note: default value for librsync)>> AES128-SHA gzip0 zlib5 (default value) 28h 17m 41s 08m 51s 28h 08m 50s 227.44Go 2.29Mo/s 09m 37s 01m 21s
227.45Go Enabled DHE-RSA-AES256-GCM-SHA384 gzip9 (default value) zlib5 (default value) 45h 02m 55s 06m 28s 44h 56m 27s 176.91Go 1.11Mo/s 12m 02s 03m 08s
227.48Go Enabled DHE-RSA-AES256-GCM-SHA384 gzip9 (default value) zlib0 44h 57m 35s 06m 26s 44h 51m 09s 176.94Go 1.12Mo/s 11m 32s 03m 06s
227.49Go Enabled DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 28h 19m 06s 09m 39s 28h 09m 37s 227.49Go 2.29Mo/s 09m 37s 01m 24s
227.52Go Disabled<<FootNote(Note: librsync=0)>> DHE-RSA-AES256-GCM-SHA384 gzip0 zlib5 (default value) 28h 40m 12s 11m 30s 28h 28m 42s 227.52Go 2.27Mo/s 09m 43s 01m 23s

Logs

Client1 full backup: 214.27Go 

2014-07-19 18:32:40: burp[9606] Client version: 1.3.48
2014-07-19 18:32:40: burp[9606] Begin phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] End phase1 (file system scan)
2014-07-19 18:40:25: burp[9606] Begin phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] End phase2 (receive file data)
2014-07-21 00:38:24: burp[9606] Begin phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] End phase3 (merge manifests)
2014-07-21 00:39:25: burp[9606] Backup ending - disconnect from client.
2014-07-21 00:39:25: burp[9606] Begin phase4 (shuffle files)
2014-07-21 00:39:25: burp[9606] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-19 18:32:39
  End time: 2014-07-21 00:45:20
Time taken: 01:06:12:41
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:    998898         0         0         0    998898 |   998898
      Directories:     74226         0         0         0     74226 |    74226
       Soft links:      4531         0         0         0      4531 |     4531
       Hard links:        16         0         0         0        16 |       16
      Grand total:   1077671         0         0         0   1077671 |  1077671
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   230074235176 (214.27 GB)
      Bytes in backup:   230074235177 (214.27 GB)
       Bytes received:   230074235177 (214.27 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-07-21 00:45:20: burp[9606] Backup completed.
2014-07-21 00:45:20: burp[9606] End phase4 (shuffle files)


Client 1 full backup 212.47Go 

2014-07-14 23:49:34: burp[5369] Client version: 1.3.48
2014-07-14 23:49:34: burp[5369] Begin phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] End phase1 (file system scan)
2014-07-14 23:54:10: burp[5369] Begin phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] End phase2 (receive file data)
2014-07-16 01:09:27: burp[5369] Begin phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] End phase3 (merge manifests)
2014-07-16 01:10:31: burp[5369] Backup ending - disconnect from client.
2014-07-16 01:10:31: burp[5369] Begin phase4 (shuffle files)
2014-07-16 01:10:31: burp[5369] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-14 23:49:26
  End time: 2014-07-16 01:16:34
Time taken: 01:01:27:08
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:    998898         0         0         0    998898 |   998898
      Directories:     74226         0         0         0     74226 |    74226
       Soft links:      4531         0         0         0      4531 |     4531
       Hard links:        16         0         0         0        16 |       16
      Grand total:   1077671         0         0         0   1077671 |  1077671
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   228124139118 (212.46 GB)
      Bytes in backup:   228135345774 (212.47 GB)
       Bytes received:   228135345774 (212.47 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-07-16 01:16:34: burp[5369] Backup completed.
2014-07-16 01:16:34: burp[5369] End phase4 (shuffle files)


Client 1 full backup 213.99Go 

2014-07-17 01:47:37: burp[4654] Client version: 1.3.48
2014-07-17 01:47:37: burp[4654] Begin phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] End phase1 (file system scan)
2014-07-17 01:53:24: burp[4654] Begin phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] End phase2 (receive file data)
2014-07-18 07:48:22: burp[4654] Begin phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] End phase3 (merge manifests)
2014-07-18 07:49:27: burp[4654] Backup ending - disconnect from client.
2014-07-18 07:49:27: burp[4654] Begin phase4 (shuffle files)
2014-07-18 07:49:27: burp[4654] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-07-17 01:47:36
  End time: 2014-07-18 07:55:21
Time taken: 01:06:07:45
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:    998898         0         0         0    998898 |   998898
      Directories:     74226         0         0         0     74226 |    74226
       Soft links:      4531         0         0         0      4531 |     4531
       Hard links:        16         0         0         0        16 |       16
      Grand total:   1077671         0         0         0   1077671 |  1077671
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   228149960334 (212.48 GB)
      Bytes in backup:   229772435086 (213.99 GB)
       Bytes received:   229772435086 (213.99 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-07-18 07:55:21: burp[4654] Backup completed.
2014-07-18 07:55:21: burp[4654] End phase4 (shuffle files)


Client 2 full backup 227.44Go 

2014-08-23 15:18:35: burp[4691] Client version: 1.3.48
2014-08-23 15:18:35: burp[4691] Begin phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] End phase1 (file system scan)
2014-08-23 15:27:26: burp[4691] Begin phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] End phase2 (receive file data)
2014-08-24 19:36:16: burp[4691] Begin phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] End phase3 (merge manifests)
2014-08-24 19:37:37: burp[4691] Backup ending - disconnect from client.
2014-08-24 19:37:37: burp[4691] Begin phase4 (shuffle files)
2014-08-24 19:37:37: burp[4691] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-23 15:18:34
  End time: 2014-08-24 19:45:53
Time taken: 01:04:27:19
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1318031         0         0         0   1318031 |  1318286
      Directories:    104066         0         0         0    104066 |   104066
       Soft links:      4586         0         0         0      4586 |     4586
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1426782         0         0         0   1426782 |  1427037
                   ------------------------------------------------------------

             Warnings:           255

      Bytes estimated:   244227872076 (227.45 GB)
      Bytes in backup:   244212565354 (227.44 GB)
       Bytes received:   244212565354 (227.44 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-08-24 19:45:53: burp[4691] Backup completed.
2014-08-24 19:45:53: burp[4691] End phase4 (shuffle files)


Client 2 full backup 227.45Go 

2014-08-25 01:23:42: burp[9427] Client version: 1.3.48
2014-08-25 01:23:42: burp[9427] Begin phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] End phase1 (file system scan)
2014-08-25 01:30:10: burp[9427] Begin phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] End phase2 (receive file data)
2014-08-26 22:26:37: burp[9427] Begin phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] End phase3 (merge manifests)
2014-08-26 22:29:45: burp[9427] Backup ending - disconnect from client.
2014-08-26 22:29:45: burp[9427] Begin phase4 (shuffle files)
2014-08-26 22:29:45: burp[9427] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-25 01:23:42
  End time: 2014-08-26 22:38:09
Time taken: 01:21:14:27
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1318211         0         0         0   1318211 |  1318211
      Directories:    104070         0         0         0    104070 |   104070
       Soft links:      4586         0         0         0      4586 |     4586
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1426966         0         0         0   1426966 |  1426966
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   244219700992 (227.45 GB)
      Bytes in backup:   244219700992 (227.45 GB)
       Bytes received:   189963848136 (176.92 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-08-26 22:38:09: burp[9427] Backup completed.
2014-08-26 22:38:09: burp[9427] End phase4 (shuffle files)


Client 2 full backup 227.49Go 

2014-08-30 17:26:31: burp[31130] Client version: 1.3.48
2014-08-30 17:26:31: burp[31130] Begin phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] End phase1 (file system scan)
2014-08-30 17:36:10: burp[31130] Begin phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] End phase2 (receive file data)
2014-08-31 21:46:47: burp[31130] Begin phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] End phase3 (merge manifests)
2014-08-31 21:48:11: burp[31130] Backup ending - disconnect from client.
2014-08-31 21:48:11: burp[31130] Begin phase4 (shuffle files)
2014-08-31 21:48:11: burp[31130] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-08-30 17:26:30
  End time: 2014-08-31 21:56:32
Time taken: 01:04:30:02
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1320475         0         0         0   1320475 |  1320484
      Directories:    104340         0         0         0    104340 |   104340
       Soft links:      4588         0         0         0      4588 |     4588
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1429502         0         0         0   1429502 |  1429511
                   ------------------------------------------------------------

             Warnings:             9

      Bytes estimated:   244267585965 (227.49 GB)
      Bytes in backup:   244267822787 (227.49 GB)
       Bytes received:   244267822787 (227.49 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-08-31 21:56:32: burp[31130] Backup completed.
2014-08-31 21:56:32: burp[31130] End phase4 (shuffle files)


Client 2 full backup 227.52Go 

2014-09-07 00:42:33: burp[22795] Client version: 1.3.48
2014-09-07 00:42:33: burp[22795] Begin phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] End phase1 (file system scan)
2014-09-07 00:54:03: burp[22795] Begin phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] End phase2 (receive file data)
2014-09-08 05:22:45: burp[22795] Begin phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] End phase3 (merge manifests)
2014-09-08 05:24:08: burp[22795] Backup ending - disconnect from client.
2014-09-08 05:24:08: burp[22795] Begin phase4 (shuffle files)
2014-09-08 05:24:08: burp[22795] Doing the atomic data jiggle...
--------------------------------------------------------------------------------
Start time: 2014-09-07 00:42:32
  End time: 2014-09-08 05:32:25
Time taken: 01:04:49:53
                         New   Changed Unchanged   Deleted     Total |  Scanned
                   ------------------------------------------------------------
            Files:   1322008         0         0         0   1322008 |  1322008
      Directories:    104595         0         0         0    104595 |   104595
       Soft links:      4588         0         0         0      4588 |     4588
       Hard links:        99         0         0         0        99 |       99
      Grand total:   1431290         0         0         0   1431290 |  1431290
                   ------------------------------------------------------------

             Warnings:             0

      Bytes estimated:   244299698290 (227.52 GB)
      Bytes in backup:   244299767377 (227.52 GB)
       Bytes received:   244299767377 (227.52 GB)
           Bytes sent:             0
--------------------------------------------------------------------------------
2014-09-08 05:32:26: burp[22795] Backup completed.
2014-09-08 05:32:26: burp[22795] End phase4 (shuffle files)


Old tests

There was a network configuration problem: `r8169` module was used instead of `r8168` on the client.

Test using client1

Description Compression SSL compression Duration on client (phase 1/phase 2) Throughput (phase 2)
First backup: 210Go None (`compression=gzip0`) (default parameter `ssl_compression` not set) 37h 09m 30s (18m 56s/36h 50m 34s) 1.6 Mo/s
Second backup: 34.33Mo added None (`compression=gzip0`) (default parameter `ssl_compression` not set) 00h 18m 10s (16m 45s/1m 25s) 0.4 Mo/s
Third backup: 4.14Go (mp4 files) added None (`compression=gzip0`) (default parameter `ssl_compression` not set) 01h 19m 54s (11m 56s/1h 07m 58s) 1.03 Mo/s
Fourth backup: 4.24Go (mp4 files) added None (`compression=gzip0`) `ssl_compression=gzip0` 00h 58m 16s (16m 39s/41m 37s) 1.73 Mo/s

Test using client2

Description Compression SSL compression Duration on client (phase 1/phase 2) Throughput (phase 2)
First backup: None (`compression=gzip0`) (default parameter `ssl_compression` not set) 37h 06m 53s (15m 40s/36h 51m 13s)
Retrieved from "https://wiki.duckcorp.org//index.php?title=Burp&oldid=222"