Deploying a 2x node + witness Proxmox and SvSAN HA cluster - Part 5 Storage Provisioning

Mark Christie
  • Updated

This article is intended for administrators wishing to create a 2x node + witness High Availability cluster utilizing Proxmox Virtual Environment (https://www.proxmox.com/en/proxmox-virtual-environment/overview) and StorMagic SvSAN (https://stormagic.com/svsan/).

Note:  All images are clickable for enlarging, or can be opened in a new tab

Information

  1. Guide Summary/shortcuts 
  2. Ensure the Proxmox node software iSCSI IQNs are Unique
  3. Create the Mirrored Target
  4. Add and Configure MPIO
  5. Login to the storage using iscsiadm
  6. Confirm the disk is visible in multipath
  7. Confirm the WWID is visible in multipath
  8. Confirm the disk is visible via disk list
  9. Create an LVM PV (LVM physical volume) on the multipath device
  10. Create an VG (Volume Group) on the multipath device
  11. Add the storage to the nodes

Guide Summary/shortcuts

This multipart guide will walk through the process to deploy 2x hyperconverged Proxmox VE nodes utilizing SvSAN virtualized block storage and a lightweight witness node, such as a Raspberry Pi.

https://support.stormagic.com/hc/en-gb/articles/17158984245789-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-0-5-Creating-nested-VMs-for-Proxmox-on-VMware

https://support.stormagic.com/hc/en-gb/articles/15088839377053-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-1-Installing-Proxmox

https://support.stormagic.com/hc/en-gb/articles/15567317393181-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-2-Preparing-the-Hosts

https://support.stormagic.com/hc/en-gb/articles/16198574072733-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-3-Creating-the-Proxmox-Cluster

https://support.stormagic.com/hc/en-gb/articles/16204170204061-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-4-VSA-Deployment

https://support.stormagic.com/hc/en-gb/articles/16343256511261-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-5-Storage-Provisioning

https://support.stormagic.com/hc/en-gb/articles/16347739183005-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-6-Create-Guests-and-Enable-HA

https://support.stormagic.com/hc/en-gb/articles/27437315756573-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-7-Enabling-healthy-login-and-cluster-restart

Ensure the Proxmox node software iSCSI IQNs are Unique

Note:  Repeat the below steps on both hyperconverged nodes.

There have been issues, working nested, in the past with IQNs being the same.

Confirm they’re different, however maybe you want to rename to something more friendly.

nano /etc/iscsi/initiatorname.iscsi

In our systems we have the below:

host1

root@edgeline01:~# cat /etc/iscsi/initiatorname.iscsi 
## DO NOT EDIT OR REMOVE THIS FILE! 
## If you remove this file, the iSCSI daemon will not start. 
## If you change the InitiatorName, existing access control lists 
## may reject this initiator. The InitiatorName must be unique 
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames. 
InitiatorName=iqn.1993-08.org.debian:01:48aa658c4691

host2

root@edgeline02:~# cat /etc/iscsi/initiatorname.iscsi 
## DO NOT EDIT OR REMOVE THIS FILE! 
## If you remove this file, the iSCSI daemon will not start. 
## If you change the InitiatorName, existing access control lists 
## may reject this initiator. The InitiatorName must be unique 
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames. 
InitiatorName=iqn.1993-08.org.debian:01:bea18b518bc6

Should they be edited restart the iSCSI daemon via systemctl

root@edgeline01:~# systemctl restart iscsid.service

These IQNs can be manually added to the VSA via the WebGUI or will be picked up automatically later with a rescan and then need to be added to the target ACL.

Create the Mirrored Target

https://stormagic.com/doc/svsan/6-7/en/Content/datastore-create-manually.htm

mceclip0.png

Add and Configure MPIO

Note:  Repeat the below steps on both hyperconverged nodes.

https://pve.proxmox.com/wiki/ISCSI_Multipath

apt-get install multipath-tools

For SvSAN with Linux initators and multipath configure the below:

https://support.stormagic.com/hc/en-gb/articles/6435122249501-SvSAN-Linux-iSCSI-Initiators

Create the below folder path:

mkdir /etc/multipath/conf.d/

Create the file per the below (on both hosts):

nano /etc/multipath/conf.d/StorMagc.conf

To contain

defaults {
    polling_interval     2
    max_polling_interval 4
    find_multipaths      "yes"
}

blacklist {
    devnode "^(ram|raw|loop|fd|md|sr|scd|st|sda)[0-9]*"
}

devices {
    device {
        user_friendly_names  "yes"
        no_path_retry        5
        detect_checker       "no"
        path_checker         "tur"
        path_grouping_policy "group_by_prio"
        detect_prio          "no"
        prio                 "alua"
        prio_args            "exclusive_pref_bit"
        hardware_handler     "1 alua"
        vendor               "StorMagc"
        product              "iSCSI Volume"
    }
}

and restart multipath

systemctl restart multipathd.service
multipath -r

Login to the storage using iscsiadm

Note:  Repeat the below steps on both hyperconverged nodes.

iscsiadm is a commandline tool similar to the iSCSI Control panel (iscsicpl) in Hyper-V and is documented : - https://linux.die.net/man/8/iscsiadm

The below section will use the below flags to log the host initiators into the Target.

-m or --mode
-T or --target
-p or --portal
-l or --login
-n node.startup -v automatic

Via the commandline on each Proxmox host run the below to discover (search out on the network) to the SvSAN VSA iSCSI IP addresses (not the Proxmox hosts IP addresses). 

This is the equivalent to adding the IPs to iscsicpl (iSCSI Control Panel) in Microsoft Hyper-V.

e.g. go out and see what targets the host is allowed to see or not.

In the above example the target has a blank Access Control List (ACL).  The method used auto populates the IQNs to the SvSAN VSAs, to save copy and paste

mceclip0.png

The below command will return iscsiadm: No portals found as expected behaviour. This is due to the Proxmox node IQN not yet being present on the SvSAN VSAs or not yet being in the Target ACL.

root@edgeline01:~# iscsiadm -m discovery -t st -p 192.168.1.5
iscsiadm: No portals found
root@edgeline01:~# iscsiadm -m discovery -t st -p 192.168.1.6
iscsiadm: No portals found

 

root@edgeline02:~# iscsiadm -m discovery -t st -p 192.168.1.5
iscsiadm: No portals found
root@edgeline02:~# iscsiadm -m discovery -t st -p 192.168.1.6
iscsiadm: No portals found

Having run the above browse to the VSA WebGUI, and select the Initiators page, noting that the IQNs will now be displayed having triggered the Proxmox nodes to scan out with the above command

mceclip0.png

Next select the Targets page and select the target.

Add the initiators, by selecting Initiators on the Target page...

mceclip0.png

and add both Proxmox node IQNs to the target Access Control list (ACL).

mceclip0.png

Run the command again and, all being correct, the available targets the Proxmox node initiator may login to will now be displayed, 

host1

root@edgeline01:~# iscsiadm -m discovery -t st -p 192.168.1.5
192.168.1.5:3260,1 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
192.168.1.6:3260,3 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
root@edgeline01:~# iscsiadm -m discovery -t st -p 192.168.1.6
192.168.1.6:3260,3 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
192.168.1.5:3260,1 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
root@edgeline01:~#

host2

root@edgeline02:~# iscsiadm -m discovery -t st -p 192.168.1.5
192.168.1.5:3260,1 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
192.168.1.6:3260,3 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
root@edgeline02:~# iscsiadm -m discovery -t st -p 192.168.1.6
192.168.1.6:3260,3 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
192.168.1.5:3260,1 iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror
root@edgeline02:~#

Login session to the hosts

Now log each host into the SvSAN presented iSCSI target, ensuring each Proxmox node has the correct number of expected sessions to each SvSAN VSA.

host1

root@edgeline01:~# iscsiadm -m node -T iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror -p 192.168.1.5 -l -n node.startup -v automatic
Login to [iface: default, target: iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror, portal: 192.168.1.5,3260] successful.

root@edgeline01:~# iscsiadm -m node -T iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror -p 192.168.1.6 -l -n node.startup -v automatic
Login to [iface: default, target: iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror, portal: 192.168.1.6,3260] successful.
root@edgeline01:~#

host2

root@edgeline02:~# iscsiadm -m node -T iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror -p 192.168.1.5 -l -n node.startup -v automatic
Login to [iface: default, target: iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror, portal: 192.168.1.5,3260] successful.

root@edgeline02:~# iscsiadm -m node -T iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror -p 192.168.1.6 -l -n node.startup -v automatic
Login to [iface: default, target: iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror, portal: 192.168.1.6,3260] successful.
root@edgeline02:~#

Check the expected number of sessions are correct in the SvSAN VSA webGUI

mceclip0.png

The below example shows how to remove/logout a session, for troubleshooting purposes.

root@edgeline02:~# iscsiadm -m node -T iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror -p 192.168.1.5 --logout
root@edgeline02:~# iscsiadm -m node -T iqn.2006-06.com.stormagic:8661170200000018.m0svsanmirror -p 192.168.1.5 --op=delete

Confirm the disk is visible in multipath

host1

root@edgeline01:~# multipath -ll
mpatha (20003398661170001) dm-2 StorMagc,iSCSI Volume
size=3.5T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 7:0:0:0 sdc 8:32 active ready running

host2

root@edgeline02:~# multipath -ll
mpatha (20003398661170001) dm-2 StorMagc,iSCSI Volume
size=3.5T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 7:0:0:0 sdc 8:32 active ready running

Confirm the WWID is visible in multipath

host1

root@edgeline01:~# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/20003398661170001/

host2

root@edgeline02:~# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/20003398661170001/

Confirm the disk is visible via disk list

When listing the disks with the following command ls -n /dev/disk/by-id/ we now see an mpath disk.

In this example dm-name-mpatha as dm-2 that is the map of

 dm-uuid-mpath-20003398661170001 -> ../../dm-2, the same WWID in the WWIDs file.

Host 1

root@edgeline01:~# ls -n /dev/disk/by-id/
total 0
lrwxrwxrwx 1 0 0  9 Jan  2 09:20 ata-SSD_512GB_AA20231025512G651778 -> ../../sda
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 ata-SSD_512GB_AA20231025512G651778-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 ata-SSD_512GB_AA20231025512G651778-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 ata-SSD_512GB_AA20231025512G651778-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 dm-name-mpatha -> ../../dm-2
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 dm-name-pve-root -> ../../dm-1
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 dm-name-pve-swap -> ../../dm-0
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 dm-uuid-LVM-H2J26vqGLDfuYNblvXYnYJNrbpX0mirTMCRZfEzBZnfhkfxcz7yhwnWvv6vfBiFu -> ../../dm-0
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 dm-uuid-LVM-H2J26vqGLDfuYNblvXYnYJNrbpX0mirTQ07jpOAHTty4U9Kv3TQa10EHtBy8V31y -> ../../dm-1
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 dm-uuid-mpath-20003398661170001 -> ../../dm-2
lrwxrwxrwx 1 0 0 10 Jan  2 09:20 lvm-pv-uuid-KsbYC9-tEH1-11TE-yB3r-kyTQ-us1d-dGjFgc -> ../../sda3
lrwxrwxrwx 1 0 0 13 Jan  2 09:20 nvme-eui.343646304e6108110025385900000001 -> ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:20 nvme-eui.343646304e6108360025385900000001 -> ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:20 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610811 -> ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:20 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610811_1 -> ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:20 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610836 -> ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:20 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610836_1 -> ../../nvme1n1
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 scsi-20003398661170001 -> ../../dm-2
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 wwn-0x0003398661170001 -> ../../dm-2
root@edgeline01:~#
mceclip0.png

Host 2

root@edgeline02:~# ls -n /dev/disk/by-id/
total 0
lrwxrwxrwx 1 0 0  9 Jan  2 09:18 ata-SSD_512GB_AA20231025512G651754 -> ../../sda
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 ata-SSD_512GB_AA20231025512G651754-part1 -> ../../sda1
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 ata-SSD_512GB_AA20231025512G651754-part2 -> ../../sda2
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 ata-SSD_512GB_AA20231025512G651754-part3 -> ../../sda3
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 dm-name-mpatha -> ../../dm-2
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 dm-name-pve-root -> ../../dm-1
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 dm-name-pve-swap -> ../../dm-0
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 dm-uuid-LVM-p3CX3QeVLAcxSjIwKVYndFvzijwzxrsfdLR25zUqtPksIaMM7eMDZEyTc6BpLAK1 -> ../../dm-1
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 dm-uuid-LVM-p3CX3QeVLAcxSjIwKVYndFvzijwzxrsfX82XHSZigh79E3IdY3FYpxqWag8k8S4x -> ../../dm-0
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 dm-uuid-mpath-20003398661170001 -> ../../dm-2
lrwxrwxrwx 1 0 0 10 Jan  2 09:18 lvm-pv-uuid-xiRydb-7Osr-SdPb-7v6c-32SD-kyz2-lizJoh -> ../../sda3
lrwxrwxrwx 1 0 0 13 Jan  2 09:18 nvme-eui.343646304e6102800025385900000001 -> ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:18 nvme-eui.343646304e6107270025385900000001 -> ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:18 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610280 -> ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:18 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610280_1 -> ../../nvme1n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:18 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610727 -> ../../nvme0n1
lrwxrwxrwx 1 0 0 13 Jan  2 09:18 nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0N610727_1 -> ../../nvme0n1
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 scsi-20003398661170001 -> ../../dm-2
lrwxrwxrwx 1 0 0 10 Jan  2 09:22 wwn-0x0003398661170001 -> ../../dm-2
mceclip0.png

It is also possible, however not necessary to create a disk alias for the disk per the below settings in the multipath.conf on each node: 

root@edgeline01:~# cat /etc/multipath/conf.d/StorMagc.conf
defaults {
    polling_interval     2
    max_polling_interval 4
    find_multipaths      "yes"
}

blacklist {
    devnode "^(ram|raw|loop|fd|md|sr|scd|st|sda)[0-9]*"
}

devices {
    device {
        user_friendly_names  "yes"
        no_path_retry        5
        detect_checker       "no"
        path_checker         "tur"
        path_grouping_policy "group_by_prio"
        detect_prio          "no"
        prio                 "alua"
        prio_args            "exclusive_pref_bit"
        hardware_handler     "1 alua"
        vendor               "StorMagc"
        product              "iSCSI Volume"
    }
}

multipaths {
  multipath {
        wwid "20003398661170001"
        alias mpatha
  }
}

Create an LVM PV (LVM physical volume) on the multipath device

From one host, noting the disk alias to select, in this instance being mpatha:

root@edgeline01:~# pvcreate /dev/mapper/mpatha
  Physical volume "/dev/mapper/mpatha" successfully created.

Create an VG (Volume Group) on the multipath device

root@edgeline01:~# vgcreate vg-svsan-storage /dev/mapper/mpatha
  Volume group "vg-svsan-storage" successfully created

Further SvSAN disks will be mpathb mpathc etc

Add the storage to the nodes

Connect to a node in the cluster, select the Datacenter object, select Storage, select Add and select LVM to add the storage to the nodes and make it usable.

mceclip0.png

Under ID: specify a name for the datastore, in this example "svsan-storage", set the base storage as the previous created volume group "vg-svsan-storage", and ensure to select "Shared".

Should shared not be selected when a guest VM is migrated between hosts, the data will also be copied rather than purely the Virtual Machine memory.

mceclip0.png

See Also

https://stormagic.com/manual/

https://support.stormagic.com/hc/en-gb/articles/15088839377053-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-1-Installing-Proxmox

https://support.stormagic.com/hc/en-gb/articles/15567317393181-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-2-Preparing-the-Hosts

https://support.stormagic.com/hc/en-gb/articles/16198574072733-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-3-Creating-the-Proxmox-Cluster

https://support.stormagic.com/hc/en-gb/articles/16204170204061-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-4-VSA-Deployment

https://support.stormagic.com/hc/en-gb/articles/16343256511261-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-5-Storage-Provisioning

https://support.stormagic.com/hc/en-gb/articles/16347739183005-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-6-Create-Guests-and-Enable-HA

https://support.stormagic.com/hc/en-gb/articles/27437315756573-Deploying-a-2x-node-witness-Proxmox-and-SvSAN-HA-cluster-Part-7-Enabling-healthy-login-and-cluster-restart

 

 

Related articles

Comments

0 comments

Article is closed for comments.