This article is intended for administrators wishing to create a 2x node + witness High Availability cluster utilizing Proxmox Virtual Environment (https://www.proxmox.com/en/proxmox-virtual-environment/overview) and StorMagic SvSAN (https://stormagic.com/svsan/).
Note: All images are clickable for enlarging, or can be opened in a new tab
Information
- Guide Summary/shortcuts
- Create the SvSAN Storage Controller Virtual Machines via the SvSAN VSA OVA
- Configure the VSA Networking
- Configure the VSA Storage
- Add the RDM sym link/SAN Storage
- Example 1 - Multiple disk device add
- Example 2 - single RDM/Pool add
- Boot and Configure the VSA VM
- First Boot Wizard
Guide Summary/shortcuts
This multipart guide will walk through the process to deploy 2x hyperconverged Proxmox VE nodes utilizing SvSAN virtualized block storage and a lightweight witness node, such as a Raspberry Pi.
Create the SvSAN Storage Controller Virtual Machines via the SvSAN VSA OVA
Note: Repeat the below steps on both hyperconverged nodes.
Note: Ensure to create VSA2 with a different VM ID - per the previous mention of unable to join when there’s VMs
With the local storage volume selected, select import, select the previously uploaded VSA OVA and select Import
Give the VSA VM a name, changing from the default StorMagic-SvSAN and confirm it has 1vCPU and 2GB RAM (up to 32GB if utilizing SSD/Memory caching)
https://support.stormagic.com/hc/en-gb/articles/5887719016989-SvHCI-SvSAN-Caching
Ensure the imports complete successfully:
On both VSAS, with the VSA VM selected, select Options and enable start at boot:
Configure the VSA Networking
Note: Repeat the below steps on both hyperconverged nodes/VSAs.
Modify net1 to attach to a storage NIC
Modify both net0 and net1 from the default vmxnet3 to virtio NIC drivers.
Note that the SvSAN VSA includes drivers for vmxnet3, virtio and E1000E.
Technical Services have seen different network throughputs on different systems and advise validating network performance per the below KB:
add additional logical vNICs to the previously created bridges, if required
Ensuring to attach the NIC to the correct bridge/network
Configure the VSA Storage
On both VSAs, with the VSA VM selected, select Hardware, and add a Hard Disk (the VSA Journal Disk)
Create the disk as 20GB, to be the Journal for change block tracking and logs.
Modify the SCSI Controller from the Default "LSI 53C895A" over to VirtIO SCSI
Modify the VSA storage controller to virtIO
It is also possible to add a disk image, equivalent to a VMDK in VMware, as SAN storage that we would add here as scsi3.
However in this example we'll utilize an RDM equivalent symbolic link to the physical disk, to utilize as SAN storage.
Add the RDM sym link/SAN Storage
Note: Repeat the below steps on both hyperconverged nodes.
The below equivalent to vmkfstools in VMware, documented below:
https://stormagic.com/doc/svsan/6-7/en/Content/allocating-additional-storage-vs.htm
It is not possible to add an RDM link disk via Proxmox GUI, like you can in VMware or Hyper-V.
The below third party video details disk passthru also.
https://www.youtube.com/watch?v=U-UTMuhmC1U
Most SvSAN customers will typically have either
1. some form of boot storage and some form of SAN storage. e.g. RAID1 boot (such as BOSS/NS204 etc) and separate SAN (RAID10, RAID5, RAID50 etc)
SvSAN supports both hardware RAID and can also support software RAID over individual disks:
https://support.stormagic.com/hc/en-gb/articles/17597617525917-SvHCI-SvSAN-and-Software-RAID
2. or split virtual disks
In both cases above the host will see the boot as /dev/sda and the storage to be used for the SAN as /dev/sdb
If there are other disks the letters will increment through the alphabet being /dev/sdc etc
NVMe based storage, may appear differently. Just be careful here and ensure nothing else is using the disks.
Commands used:
ls -n /dev/disk/by-id/
/sbin/qm set [VM-ID] -virtio2 /dev/disk/by-id/[DISK-ID]
Example 1 - Multiple disk device add
These Edgeline systems have 2x 3.87TB NVMe drives that, in this example, we'll map to the SvSAN VSA and software RAID1 to provide internal node resiliency, and then the SvSAN synchronous mirroring across the VSAs/nodes providing cross-node resiliency.
This architecture enables 2x fault domains.
The below putty sessions are from the Edgeline systems demonstration the symbolic link creation.
This example was run on hardware with 2x NVMe's assigning to SvSAN to be used via software RAID1 for the pool storage
If mapping up multiple disks for software RAID, note the VM ID
101
and the virtio id
virtio2
example - multiple RDM/pools disk add (from HPE Edgelines with 2x Samsung NVMe for SvSAN software RAID1)
root@proxmox-1:~# ls -n /dev/disk/by-id/
total 0
lrwxrwxrwx 1 0 0 9 Oct 10 09:45 ata-IM2S33D4_2K24291DCC1U - ../../sda
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 ata-IM2S33D4_2K24291DCC1U-part1 - ../../sda1
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 ata-IM2S33D4_2K24291DCC1U-part2 - ../../sda2
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 ata-IM2S33D4_2K24291DCC1U-part3 - ../../sda3
lrwxrwxrwx 1 0 0 10 Oct 10 09:40 dm-name-pve-root - ../../dm-1
lrwxrwxrwx 1 0 0 10 Sep 28 18:27 dm-name-pve-swap - ../../dm-0
lrwxrwxrwx 1 0 0 10 Oct 10 09:40 dm-uuid-LVM-rk53mp1TcHumnHnAxU6G12FI7KjFvXMK4Rbm0N5hFIyDERfHbfFdxEcmxTz6OycW - ../../dm-1
lrwxrwxrwx 1 0 0 10 Sep 28 18:27 dm-uuid-LVM-rk53mp1TcHumnHnAxU6G12FI7KjFvXMKhnwdeMkQJemP4yiqT2lZtLmNLrGJpSFm - ../../dm-0
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 lvm-pv-uuid-VWXWt8-8cgk-hxo9-FOcX-c1yz-U8KL-zC43FB - ../../sda3
lrwxrwxrwx 1 0 0 13 Sep 28 18:27 nvme-eui.343646304e6108110025385900000001 - ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Sep 28 18:27 nvme-eui.343646304e6108360025385900000001 - ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Sep 28 18:27 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610811 - ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Sep 28 18:27 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610836 - ../../nvme1n1
lrwxrwxrwx 1 0 0 9 Oct 10 09:45 wwn-0x5707c18100925c1a - ../../sda
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 wwn-0x5707c18100925c1a-part1 - ../../sda1
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 wwn-0x5707c18100925c1a-part2 - ../../sda2
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 wwn-0x5707c18100925c1a-part3 - ../../sda3
root@proxmox-1:~#
root@proxmox-1:~# /sbin/qm set 100 -virtio2 /dev/disk/by-id/nvme-eui.343646304e6108110025385900000001
update VM 100: -virtio2 /dev/disk/by-id/nvme-eui.343646304e6108110025385900000001
root@proxmox-1:~# /sbin/qm set 100 -virtio3 /dev/disk/by-id/nvme-eui.343646304e6108360025385900000001
update VM 100: -virtio3 /dev/disk/by-id/nvme-eui.343646304e6108360025385900000001
root@proxmox-1:~#
root@proxmox-2:~# ls -n /dev/disk/by-id/
total 0
lrwxrwxrwx 1 0 0 9 Oct 10 09:45 ata-IM2S33D4_2K242L1DE2TY - ../../sda
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 ata-IM2S33D4_2K242L1DE2TY-part1 - ../../sda1
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 ata-IM2S33D4_2K242L1DE2TY-part2 - ../../sda2
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 ata-IM2S33D4_2K242L1DE2TY-part3 - ../../sda3
lrwxrwxrwx 1 0 0 10 Oct 10 09:40 dm-name-pve-root - ../../dm-1
lrwxrwxrwx 1 0 0 10 Sep 28 18:24 dm-name-pve-swap - ../../dm-0
lrwxrwxrwx 1 0 0 10 Oct 10 09:40 dm-uuid-LVM-ng5R1ctgy041DlLIagXtSqn6454eaenqF4sEOW9d2RM315ejVzxpOUlu9ayhn3YK - ../../dm-1
lrwxrwxrwx 1 0 0 10 Sep 28 18:24 dm-uuid-LVM-ng5R1ctgy041DlLIagXtSqn6454eaenquJLJNeWfSD4Hg5Q9faLlAv9PWZielnX8 - ../../dm-0
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 lvm-pv-uuid-WGMruL-XZfV-7zPW-3BVp-Qzqg-Twrd-PS8qic - ../../sda3
lrwxrwxrwx 1 0 0 13 Sep 28 18:24 nvme-eui.343646304e6102800025385900000001 - ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Sep 28 18:24 nvme-eui.343646304e6107270025385900000001 - ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Sep 28 18:24 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610280 - ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Sep 28 18:24 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610727 - ../../nvme0n1
lrwxrwxrwx 1 0 0 9 Oct 10 09:45 wwn-0x5707c18100925646 - ../../sda
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 wwn-0x5707c18100925646-part1 - ../../sda1
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 wwn-0x5707c18100925646-part2 - ../../sda2
lrwxrwxrwx 1 0 0 10 Oct 10 09:45 wwn-0x5707c18100925646-part3 - ../../sda3
root@proxmox-2:~# /sbin/qm set 101 -virtio2 /dev/disk/by-id/nvme-eui.343646304e6102800025385900000001
update VM 101: -virtio2 /dev/disk/by-id/nvme-eui.343646304e6102800025385900000001
root@proxmox-2:~# /sbin/qm set 101 -virtio3 /dev/disk/by-id/nvme-eui.343646304e6107270025385900000001
update VM 101: -virtio3 /dev/disk/by-id/nvme-eui.343646304e6107270025385900000001
root@proxmox-2:~#Once completed successfully this will display in the GUI per the below:
Example 2 - single RDM/Pool add
This example was completed on a system nested on VMware, hence the disk name.
Note: below sda and sdb are seen twice. once with a SCSI label:
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11 -> ../../sda
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 scsi-36000c29fd24a75bc12f86ebaa096dbb6 -> ../../sdband once with a wwn label:
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11 -> ../../sda
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 wwn-0x6000c29fd24a75bc12f86ebaa096dbb6 -> ../../sdbFor simplicities sake, and to keep this architecture similar to VMware, where SvSAN maps to the naa. device (e.g. scsi) it is recommended to use the SCSI option
root@proxmox1:~# ls -n /dev/disk/by-id/
total 0
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 ata-VMware_Virtual_IDE_CDROM_Drive_00000000000000000001 -> ../../sr0
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 dm-name-pve-root -> ../../dm-1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 dm-name-pve-swap -> ../../dm-0
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 dm-uuid-LVM-397ePpY9Yy0rnlBAaao02wNyCEIqmouhKzBkyr2czUFry9Raklzym2p2zynHSDnB -> ../../dm-1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 dm-uuid-LVM-397ePpY9Yy0rnlBAaao02wNyCEIqmouhnvouNGymAO9TNqIkjunt0nK3NogmNjfr -> ../../dm-0
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 lvm-pv-uuid-mEFIaM-ZrA9-Ulan-3XhS-i1DL-hbFZ-9iS3dp -> ../../sda3
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11 -> ../../sda
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-36000c296a1fedf604ddc9bfff9978a11-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 scsi-36000c29fd24a75bc12f86ebaa096dbb6 -> ../../sdb
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 scsi-SVMware_Virtual_disk_6000c296a1fedf604ddc9bfff9978a11 -> ../../sda
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-SVMware_Virtual_disk_6000c296a1fedf604ddc9bfff9978a11-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-SVMware_Virtual_disk_6000c296a1fedf604ddc9bfff9978a11-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 scsi-SVMware_Virtual_disk_6000c296a1fedf604ddc9bfff9978a11-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 scsi-SVMware_Virtual_disk_6000c29fd24a75bc12f86ebaa096dbb6 -> ../../sdb
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11 -> ../../sda
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Mar 24 18:14 wwn-0x6000c296a1fedf604ddc9bfff9978a11-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 9 Mar 24 18:14 wwn-0x6000c29fd24a75bc12f86ebaa096dbb6 -> ../../sdb
root@proxmox1:~# /sbin/qm set 100 -virtio2 /dev/disk/by-id/scsi-36000c29fd24a75bc12f86ebaa096dbb6
update VM 100: -virtio2 /dev/disk/by-id/scsi-36000c29fd24a75bc12f86ebaa096dbb6
root@proxmox1:~#Noting the disk ID ending dbb6
Boot and Configure the VSA VM
The VSA will boot up like an blank deployed OVA in VMware with DHCP on the NICs.
Per the below Console walkthrough KB:
https://support.stormagic.com/hc/en-gb/articles/18860298535709-SvHCI-SvSAN-Console-usage
Login to the VSA using the default password of ‘password’ and configure the IP addresses statically.
Management:
and Storage interface:
It is also advised to set DNS, NTP and a network hostname such that the VSA will license correctly when in the first boot wizard, being able to route out to and resolve https://licensing.stormagic.com.
Offline license files may be used for closed/air gapped environments per the below KB:
First Boot Wizard
Complete the standard default boot wizard via the WebGUI
Accept the EULA:
Proceed through the wizard:
License the VSA VM
https://support.stormagic.com/hc/en-gb/articles/5355376684445-SvHCI-SvSAN-Licensing-model
Once licensed certain features will display TRUE or if using an evaluation key display FALSE but have a evaluation time remaining:
Specify hostname and domain name, if not previously set via the console:
Set a password
Complete the wizard:
Once logged in with your own password explore the node WebGUI such as Devices:
And create a pool ready to proceed to Storage Provisioning:
See Also
Comments
0 comments
Article is closed for comments.