iSCSI initiator and Dell PowerVault MD3000i

[Miroslav Lachman]

please Cc: me, I am not subscribed to freebsd-scsi

Sossi Andrej wrote:
>> On 16. 12. 2009 15:57, Miroslav Lachman wrote:
>> [...]
>> I use MD300i with FreeBSD 7.0 and 7.1 with iscsi-2.2.2. It work fine.
>> But be careful to configure MD3000i. MD3000i assign by default first
>> disk to preferred controller 0, second disk to preferred controller 1,
>> third disk to preferred controller 0, and so on. First, third, fifth...
>> disks is usable from FreeBSD, but second, fourth,... disks result
unusable.
>> Work around: manually assign all disks to controller 0.
>
> When you say "unusable" do you mean you can't access it at all / it
> errors even if it's the only path (drive) used? It would be normal if
> you have for example two paths to each drive and can't mount the other
> path if one path to the drive is mounted - this is not a usable
> combination. You can use geom_multipath to get multipath failover.

I got errors even in unmounted state.
I tried iscsi-2.2.3 and got same errors. I tried second path first
(device da0) and it produces same errors, then I run iscontrol for the
first path (device da1) and everything is fine.

---- path throught second controller: ERROR ----
# diskinfo -t /dev/da0
/dev/da0
512 # sectorsize
2998998663168 # mediasize in bytes (2.7T)
5857419264 # mediasize in sectors
364607 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.

Seek times:
Full stroke: diskinfo: read error or disk too small for
test.: Invalid argument


---- path throught first controller: OK ----
# diskinfo -t /dev/da1
/dev/da1
512 # sectorsize
2998998663168 # mediasize in bytes (2.7T)
5857419264 # mediasize in sectors
364607 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.

Seek times:
Full stroke: 250 iter in 2.483517 sec = 9.934 msec
Half stroke: 250 iter in 2.575778 sec = 10.303 msec
Quarter stroke: 500 iter in 2.926170 sec = 5.852 msec
Short forward: 400 iter in 0.916901 sec = 2.292 msec
Short backward: 400 iter in 2.181790 sec = 5.454 msec
Seq outer: 2048 iter in 0.520920 sec = 0.254 msec
Seq inner: 2048 iter in 0.545300 sec = 0.266 msec
Transfer rates:
outside: 102400 kbytes in 1.414997 sec = 72368
kbytes/sec
middle: 102400 kbytes in 1.454444 sec = 70405
kbytes/sec
inside: 102400 kbytes in 1.422527 sec = 71985
kbytes/sec


Do you have experiences with iSCSI multipath? I read about geom_fox and
gmultipath...

Miroslav Lachman

[Ivan Voras]

2009/12/17 Miroslav Lachman <***@quip.cz>:
> please Cc: me, I am not subscribed to freebsd-scsi
>
> Sossi Andrej wrote:
>>> On 16. 12. 2009 15:57, Miroslav Lachman wrote:
>>> [...]
>>> I use MD300i with FreeBSD 7.0 and 7.1 with iscsi-2.2.2. It work fine.
>>> But be careful to configure MD3000i. MD3000i assign by default first
>>> disk to preferred controller 0, second disk to preferred controller 1,
>>> third disk to preferred controller 0, and so on. First, third, fifth...
>>> disks is usable from FreeBSD, but second, fourth,... disks result
>>> unusable.
>>> Work around: manually assign all disks to controller 0.
>>
>> When you say "unusable" do you mean you can't access it at all / it
>> errors even if it's the only path (drive) used? It would be normal if
>> you have for example two paths to each drive and can't mount the other
>> path if one path to the drive is mounted - this is not a usable
>> combination. You can use geom_multipath to get multipath failover.
>
> I got errors even in unmounted state.
> I tried iscsi-2.2.3 and got same errors. I tried second path first (device
> da0) and it produces same errors, then I run iscontrol for the first path
> (device da1) and everything is fine.
>
>  ---- path throught second controller: ERROR ----
> # diskinfo -t /dev/da0
> /dev/da0
>        512             # sectorsize
>        2998998663168   # mediasize in bytes (2.7T)
>        5857419264      # mediasize in sectors
>        364607          # Cylinders according to firmware.
>        255             # Heads according to firmware.
>        63              # Sectors according to firmware.
>
> Seek times:
>        Full stroke:    diskinfo: read error or disk too small for test.:
> Invalid argument
>
>
>  ---- path throught first controller: OK ----
> # diskinfo -t /dev/da1
> /dev/da1
>        512             # sectorsize
>        2998998663168   # mediasize in bytes (2.7T)
>        5857419264      # mediasize in sectors
>        364607          # Cylinders according to firmware.
>        255             # Heads according to firmware.
>        63              # Sectors according to firmware.
>
> Seek times:
>        Full stroke:      250 iter in   2.483517 sec =    9.934 msec
>        Half stroke:      250 iter in   2.575778 sec =   10.303 msec
>        Quarter stroke:   500 iter in   2.926170 sec =    5.852 msec
>        Short forward:    400 iter in   0.916901 sec =    2.292 msec
>        Short backward:   400 iter in   2.181790 sec =    5.454 msec
>        Seq outer:       2048 iter in   0.520920 sec =    0.254 msec
>        Seq inner:       2048 iter in   0.545300 sec =    0.266 msec
> Transfer rates:
>        outside:       102400 kbytes in   1.414997 sec =    72368 kbytes/sec
>        middle:        102400 kbytes in   1.454444 sec =    70405 kbytes/sec
>        inside:        102400 kbytes in   1.422527 sec =    71985 kbytes/sec

This is strange and probably indicates a bug somewhere. Can you check
your SAN configuration for example, wrong access permissions assigned
to the problematic port?

> Do you have experiences with iSCSI multipath? I read about geom_fox and
> gmultipath...

You should probably skip geom_fox and just use gmultipath. It works as
advertised, nothing fancy to report.

[Miroslav Lachman]

Daniel Braniss wrote:

[...]

Hi Daniel

> the numbers seem ok to me, concidering that the net is 1Gb.
> can you configure the target virtual disk to have luns?

Each virtual disk is automatically on its LUN (started from 0)

> in any case the errors seem to be in the md3000i, can you see/check its
> error log?

No warnings or errors in log, just notices without anything meaningfull
to me - I can send you whole log if you are interested

>> Do you have experiences with iSCSI multipath? I read about geom_fox and
>> gmultipath...
> i have no experience with it, and personaly see no benefit in it (but then
> others might disagree :-)

I tried 'ifconfig bce1 down' to simulate broken preferred path (hoping
that second path become preferred and virtual disks will be available by
the second path... but it failed) and then "all hangs".
I tried to access /dev/da0 in the meantime by diskinfo and diskinfo
became unresponsive and unkillable by kill -9. Then I sent SIGHUP to
iscontrol but again - process is hang and unkillable.

I put bce1 up again, but no change. The machine was livelocked -
responding to ping, allows new SSH login, but any commands hang. I
couldn't 'su - root', or run 'top' etc. I waited about 20 hours, but no
change. "Graceful shutdown" from remote console (Dell DRAC6) also failed.

Everything is fine after powercycle so I did it again to check if it was
coincidence... same result - unkillable processes.

So instead of iSCSI connected storage with multipath failover, I got
singlepath storage causing system freeze in case of network disconnection :(

Please let me know if you are interested in this and if you want some
more details or access to this machine.

Miroslav Lachman