TC + AOE + VFS: Opinions/theories requested!

[CentralWare]

Scenario: Imagine a machine running as an AoE server.  Enclosed is a 500GB hard drive which will be set up with a single partition (EXT4) which if this works out, is NOT going to be exported as a whole, but instead, it would be preferable to create virtual drives and have those exported to AOE.

1) Does anyone have reason to believe this would even work in the first place?
2) Are there caveats you can think of which would cause unnecessary resource usage/lags/etc.?

The thought here is diving a drive, set of drives, arrays, etc. is going to get a bit ugly after a while (such as LV usage) whereas single files would be easier to maintain, create, delete, clone/copy/backup and so forth, but that's only if it doesn't come with a huge drain on resources.

Thanks for any insight!

[curaga]

I'm not familiar with AoE, but in general using disk images on ext4 vs direct partitions has minimal overhead. Unless the disk is almost full, ext4 will avoid fragmentation, and the blocks will be in sequential order.

[CentralWare]

@curaga: Thanks for the insight.  I'm thinking (just a scenario, not sure I'll beat on it this hard) imagine a couple/few dozen "dumb terminals" launching from PXE.  Kernel loads...  image loads...  command line says "Use AOE (ATA over Ethernet) as a make-believe hard drive on this machine and then set tce=, opt= and home= to that drive."  I'm thinking to myself, if each vfs is just a file and IF EXT4 does in fact have a layered block write (sequential) then in theory, the virtual file systems should do perfectly for the client machines.  Now, I've noticed that just a dozen machines hitting a server with cifs, nfs and/or tftp...  the server has only a few gigs of memory and a single core 3.4...  but that resource drains in no time until the server's virtually useless.  (I haven't traced it out yet, but will eventually.)  I can only imagine a heavy load with AoE (direct partitions or virtual) may cause fs corruption...  we'll see!

[gerald_clark]

The wiki has an article on using AoE for a tce drive. http://wiki.tinycorelinux.net/wiki:netbooting
You will need to be running vblade on the server machine, and only one client can connect to an AoE block device.

[CentralWare]

@gerald: Yes, one client per device is a given (otherwise it's like having 2+ people sharing simultaneous read/writes on a file...  it's likely to get ugly!)  I don't imagine anyone has implemented a r/o flag which would have come in handy in some instances, but there are other protocols we could use for that if necessary.

[CentralWare]

@all: tc-config will need a mild tweaking in order to get AoE to function correctly.  (We have no tools in the repo for AoE so we have to rely strictly on kernel, which doesn't TELL us much in the event of a problem.)

In the segment following
if [ -n "$AOE" ]; then
about a dozen or so lines down is a conditional if statement
if [ $CNT -gt 0 ] ; then
Within, it creates a link to the attached AoE device in /dev/etherd/ to /dev, makes the directory for [dev] under /mnt and attempts to mount it.
A good number of flags are thrown while attempting to mount the device without a specified type.

after the link is created, it would be wise to nab the fs type

Code: [Select]

fstype=`blkid /dev/$DEV | awk -F "TYPE=\"" '{print $2}' | awk -F "\"" '{print $1}'`
* Or something similar
The mount line should be conditional such as
if [ ! "$fstype" == "" ]; then
which should relieve any headaches and preventing a failed mount when an error should be thrown by us instead.

Beyond that, I'll see about compiling and testing a set of aoe-utils once time permits.


Testing environment:
TC 4.7.7 x86 under i686/3GB/GBe
vblade extension loaded, running the console export of
vblade 1 1 eth0 diskimage.img
built by using
dd if=/dev/zero of=diskimage.img bs=1024 count=1M
Drive created
Mounted the drive and partitioned it, thus creating e1.1p1

Client: aoe=eth0:e1.1p1 tce=e1.1p1 opt=e1.1p1 running TC4 under VMWare

Errors:
mount complained the partition was not ext3 and then ext2 compliant, though it's formatted ext4
Invalid device specified (for OPT persist)

Mounting the drive manually worked.
mount -t ext4 /dev/e1.1p1 /mnt/e1.1p1
Once it was mounted manually, rebooting was successful with automation...  for some reason...  though still complains about ext3/ext2.
Otherwise it works terrifically...  now onto seeing whether or not vbladed is functional and if so...  off to stress testing!