Mods input needed on disk activity (theoretical)

[CentralWare]

Due to different options and settings for different brands of hard drives, I'm going to be building a system to handle spin-down of hard drives based on a static numeric value (in minutes) for each given drive and using /sys/block/sdX/stat and/or similar to monitor disk activity when hdparm indicates a drive isn't in standby.  For sake of this information being utilized elsewhere (such as a stats page or something) I need to save the information (expected to be placed in /tmp/drivemon/sdX) and each sdX file will contain either an incremented value (in seconds) or the word "standby" when a drive is put to sleep using hdparm -y.

JUST IN CASE I'm not looking at every aspect I should be, can anyone think of any reason a write to /tmp causing an HDD write (or sync) elsewhere by doing so, thus defeating the purpose of monitoring the drive(s) and their temperatures?  (ie: or could a call to hddtemp, which I'm understanding communicates with SMART on the drive controller...  would this show up as activity?)

The goal here is to monitor drive bandwidth (in/out) and based on a few assumptions (such as IF hddtemp does show up as activity, ignore the activity for that one particular read cycle) and in the event there is no outside calls to the drive, allow the drive to be put to sleep after a given timeout.

Cause: I have WD Green/Black drives and a few Seagate SATA 1.5TB and above drives which don't show APM support and seemingly ignore hdparm -B/-S commands and they rarely if ever go into power saving mode...  and I know there's nothing calling on them.  I imagine this applies to many different breeds of storage, thus the need to put these puppies to bed when they're not being utilized.

Thanks for your input in advance!

[Rich]

Hi centralware
Accessing the /tmp directory shouldn't show up in the HDD stats since it's RAM based. Probably won't be of much use to you but
I wrote a script for monitoring HDD activity last year:
http://forum.tinycorelinux.net/index.php/topic,16256.msg96390.html#msg96390

[CentralWare]

@Rich: Thank you for your input.
Reading /proc I didn't think would cause any activity and /tmp I figured the same (ram based) but I'm worried about hddtemp pinging the drive from a sleep mode as well as /proc/diskstats or /sys/.../sda might see the SMART request as traffic.  I'll be building the scripts in the next few days so I'll know for sure, but figured I'd inquire beforehand to hopefully prevent a little tail-chasing.

[Rich]

Hi centralware
/proc/diskstats is also RAM based. Reading it will not wake up the drives. It is periodically update by the OS, not because it gets read.

[CentralWare]

@Rich: Thanks for the input.  One last question if you don't mind...

I built a testing script which reads active ext# partitions from blkid, scans for activity (I'm using /sys/block/[drive]/stat) and loops with about a one second delay between readings.  Once the timeout is reached, a given drive is sent to sleep...  but it just dawned on me; what if there is data in memory that hasn't been written yet...  what happens to the data?  (ie: sits in cache until the drive wakes, gets left for the next breeze to send it to Oz, etc.?)

hddtemp: Thankfully, the app answered the question itself...  if the drive is asleep, the drive doesn't hit SMART for results on temperature (likely they learned this same issue I was thinking would happen - a sleeping drive would be woken to take it's temperature) so I have stats and temp both running in a display similar to your example, just not aterm formatted.

I haven't done any digging, but is there a simple (and preferably built-in) way to determine dirty write cache on a specific drive?  I'm trying to write this so that we're depending on as little as possible.  (Temp readings are only going to be utilized IF the app's installed, but will run the spin-down regardless.)

Here's what's in place thus far in case you want to format it for a window (aterm) and repost; running currently under 4.7.7 raw core:

Code: [Select]

#!/bin/sh
clear
timeout=300;                                                                                    # Idle time (approx seconds) before sleep

echo -e "DEV\tIDLE\tREAD\tWRITE\tTEMP"
echo "======================================"
echo -ne "\033[s"

while true
do

echo -ne "\033[u";
drives=`blkid | grep -v loop | grep -v ram > /tmp/drives`

while read drv
do
    device=`echo "$drv" | awk -F: '{print $1}'`
    dev=`echo "$device" | awk -F/ '{print $3}'`; dev=${dev:0:3};                                # We only want sdx from sdx#
    type=`echo "$drv" | awk -F "TYPE=\"" '{print $2}' | awk -F "\"" '{print $1}'`
    uuid=`echo "$drv" | awk -F "UUID=\"" '{print $2}' | awk -F "\"" '{print $1}'`
    labl=`echo "$drv" | awk -F "LABEL=\"" '{print $2}' | awk -F "\"" '{print $1}'`

    dcnt=0;

    # Bypass test if the drive is already suspended
    test=`hdparm -C /dev/$dev | grep standby`
    if [ ! "${test}" == "" ]; then
        echo "$dev SLEEPING                                         "
    else
        case $type in
            ext*)
                dcnt=$((dcnt+1))
                path="/tmp/diskmon"; if [ ! -d $path ]; then mkdir -p $path; fi
                file="$path/$dev"; if [ ! -f $file ]; then echo "0 0 0" > $file; fi
                path="/sys/block/$dev/stat"; data=`cat $path`;                                  # Live data from device block
                time=`cat $file | awk '{print $1}'`; time=$((time+0));                          # Elapsed (saved) Time in Seconds
                exrd=`cat $file | awk '{print $2}'`; exrd=$((exrd+0));                          # Existing READ Value
                exwt=`cat $file | awk '{print $3}'`; exwt=$((exwt+0));                          # Existing Write Value
                iord=`echo "$data" | awk '{print $1}'`; iord=$((iord+0));                       # Current READ Value
                iowt=`echo "$data" | awk '{print $5}'`; iowt=$((iowt+0));                       # Current WRITE Value

                refresh=0;
                if [ $iord -gt $exrd ]; then refresh=1; fi
                if [ $iowt -gt $exwt ]; then refresh=1; fi
                if [ $refresh -gt 0 ]; then time=0; fi                                          # If there's a change, reset timer to zero
                heat=`hddtemp -q -uF /dev/$dev | awk -F: '{print $3}' | awk '{print $1}'`;
                if [ "${heat:0:2}" == "S." ]; then heat="0"; fi

                # C to F Conversion  ( heat * 9 / 5 - 32 )
                if [ $heat -ne 0 ]; then
                    heat=$((heat*9/5+32));
                    heat="${heat}*F"
                fi

                echo -e "\033[K$dev\t$time\t$iord\t$iowt\t$heat                 "
                time=$((time+1));
                echo "$time $iord $iowt $heat" > $file                                          # Save content to temp storage
                if [ $time -gt $timeout ]; then
                    # Put drive to sleep
                    time=0; echo "$time $iord $iowt $heat" > $file                              # Save content to temp storage
                    hdparm -y $device >/dev/null 2>&1;
                fi
            ;;
        esac
    fi
done < /tmp/drives
sudo rm /tmp/drives >/dev/null 2>&1;

delay=0.75
sleep $delay
done

Thanks again for your input and efforts!

~TJ~

[Rich]

Hi centralware
If there is data still in the OS cache, the kernel will eventually decide that there are no more writes pending and flush it out to the
drive which will wake it up. As for the drives built in cache, I should hope the manufacturers are smart enough to flush that upon
receiving a sleep command. With that timeout of 300 I would open that delay to 5 or 10 seconds, no point in spinning that loop
any faster than necessary.

[CentralWare]

@Rich, the per-second-ish delay is to see live stats on my end - yes, I agree completely with a polling time of about 5-10 seconds otherwise it just wastes resources; if being done within an X GUI I might consider 2 seconds or so.  One of the main reasons it was set to 0.5~1.0 seconds was I was seeing some odd behavior on the TCE drive which I still haven't traced back.  (Reads every 1.5 seconds or so on a machine that's doing virtually nothing except the loop in question.)  I'm still digging, but there's something within persist (opt and tce) that's being read after the directories go virtual/memory and there's nothing in /opt that I can think of which would be read again after the initial boot.  (ext2 format, so journal shouldn't be the case.)

Code: [Select]

DEV     IDLE    READ    WRIT    TEMP
======================================
sda     0       8412    4       0 <-- Drive is a 128MB CF; boot of tce= and opt= point to it.  Read blocks were at 1,100 a few hours ago - it should be dormant.
sdb SLEEPING
sdc SLEEPING
sdd SLEEPING
sde     4       325     153054  87*F <-- stubborn drive (WD Green 1.5TB) which wakes for no apparent reason from time to time


[Rich]

Hi centralware

... I was seeing some odd behavior on the TCE drive which I still haven't traced back.  (Reads every 1.5 seconds or so on a machine that's doing virtually nothing except the loop in question.)

If you stop the loop, do the reads stop? Is there a disk access LED you can watch to see if the reads stop?

Do you have a swap file or partition on any of your drives?

[CentralWare]

Swap FILES: none on any of the drives
Drive indicator: Yes and no.  The TCE drive is a compact flash (via IDE reader) which has two LEDs; the RED one is like a power indicator and YELLOW shows activity.  A single block every 1.5 or so seconds doesn't make the LED so much as flicker.  When writing tiny scripts, you don't even notice anything, but that's when writing.
Loop on/off - testing something now...
Last scan: 13,450 blocks READ
...waited a bit...
New scan: 13,452 blocks READ
New scan: 13,457 blocks READ
New scan: 13,459 blocks READ
(10 seconds lapsed - sitting idle now without loop running for 30 sec)
New scan: 13,461 blocks READ

Based on the differences, I have to assume TCE drive is being hit by something within the script...  but it's a READ function and I can't imagine what that would be.  WRITE has been 4 ever since boot which was almost a day ago.  Being a flash based drive, I really can't say it's overly important as it's not something that can power down, and again, it's a READ, not a write...  which would be a problem - and is where I'm assuming the read is coming from.  (ie: a drive not supported or fully supported by SMART may actually be seen as activity if a command fails...  and it might be possible that reading /sys devices might actually require the OS to "go look" if the information isn't being fed by the device itself.)  I haven't tested the script on an HDD based system or with USB based hardware -- that's yet to come when time permits.

[Rich]

Hi centralware
Another thread on this forum may have provided an idea to help you troubleshoot this. Using inotify-tools.tcz you should be
able to identify the source of the disk accesses. See:
http://linux.die.net/man/1/inotifywatch
http://linux.die.net/man/1/inotifywait

[CentralWare]

Thank you, Rich.  You've been tremendously helpful throughout this area.

I'm setting up the white-board for the next remastering system (which is what made me think to check in with TCL) and the sda issue is now just tagged for the moment to be investigated as it's not a vital tid-bit (I'm hoping it's CF flash related due to the IDE->CF reader, but we'll see once it's on some actual, live production machines.)  For the moment, though, I'm spending a little too much time in the trenches, so, since I have the majority of the spin-down system operational, this will have to do for the time being until the core itself housing these utilities is properly tested.

One last question (mildly off-topic, but still storage related) if you don't mind my asking - an obstacle that's actually proving to be more of a thorn or maybe I'm just not thinking outside the box enough...

When tc-config is launched and the boot codes are processed...  devadm is told to fire up (add/trigger) and IF by chance there was a boot code of usbwait=LABEL=something (or UUID) the system basically loops for "x" seconds or until the device in question is found.  This is an awesome concept and works perfectly for everyone...  except what I'm in need of.

Scenario:
tce=something is detected in the cmdline...  no problem.
tcp=something is an option to override tce/opt/etc. IF DETECTED (tcp=LABEL=blahblah) it overrides tce/opt/home/local/etc. found in the cmdline.
While all of this is going on, usbwait isn't going to do much good for me as there are other detection methods I'm using that are ON the devices where UUID/LABEL information won't be known in advance.

What I need to do is either force devadm to settle or find a way to monitor the device cache from within the boot script, allowing for a hybrid usbwait.  Once USB storage is loaded, we move forward and scan the devices we find, test-mounting the devices to see if files exist on a given device which are intended to override the boot codes.  The goal here is to have a clean kernel/image boot up and if tcp, nfsmount, cifsmount and the last noted "scan" don't have any success, the persistence falls back to the boot device (if tce= is in the cmdline at all.)  If not, we fall back to the memory-only version where TCE is in the /tmp directory.

Example:
/dev/sda1 is an internal storage medium with tce=sda1 opt=sda1 (nothing else listed)
tcp= is not listed; rules are ignored
nfs/cifs mounts also not listed and are ignored
udevadm is launched to start hardware implementation (usb_storage doesn't exist, so I have to find alternatives) and once USB, FireWire and similar storage based drivers are online, we then scan any partitions we find by listing hardware from blkid.
/dev/sdb1 is now detected, a 2GB flash drive plugged into USB.  On this device we find a file /mnt/test/.persist
     This tells us to revamp the command line and have tce=, opt=, etc. point to this new location instead of the default of sda1.
     (I cheated and just bind-mounted on top of /proc/cmdline so that I didn't have to completely rewrite tc-config)

If I add a manual wait=10, this just wastes time if the drivers have already loaded, whereas on slower machines, there's no guarantee we even have the drivers running yet.  What's needed is a way where we can baby-sit the system and wait for specific drivers to load, once loaded give a second or two for them to scan by waiting for the kernel to load an unrelated driver and (in theory) we should be ready to move on and let devadm "do it's thing" with everything else in its cache unrelated to storage.

If it's not too much trouble, I'd love an extra brain-storm partner.  I'll be the first to admit that I do not know all there is to know either within Linux in general, let alone TC, but I've been tearing things apart ever since I was seven, so it's in the blood...  TC won't be any different and compared to many others, the way in which you've responded to the question on this post tells me you're likely the perfect person to ask; knowledgeable...  level-headed...  and quite considerate.

If it's not possible "as is" I'll delve into the kernel...  but I have a motto (in electronics) "If it ain't broke...  don't fix break it" so I'm hoping we can conjure up a solution outside.

Thanks again for your time and efforts; it's truly appreciated.

[coreplayer2]

While all of this is going on, usbwait isn't going to do much good for me as there are other detection methods I'm using that are ON the devices where UUID/LABEL information won't be known in advance.

The beauty of using a label is that in a remaster the file system LABEL can be specified so will always be known in advance, this is how I accomplish the same task of deployed image for various hardware



Sent from my iPhone using Tapatalk

[Rich]

Hi centralware
I'm not sure I follow exactly what  you are trying to do, but I suspect you will probably need to modify tc-config. It sounds like
you might want to place a modified version of the  waitusb  section of code and place it in a separate script that you can launch
to run in the background while you are checking for tcp, cifs, or whatever. Have the script create a file in /tmp to serve as a
flag to signal when it's done.

[CentralWare]

Hi guys!  Okay, allow me to draw up an example scenario as to what we're after.  Picture a classroom.  There's roughly 30 workstations where by themselves, they boot from a read-only version of TC (most likely SD card, CD, CF card, etc.) and if nobody is logged into the station, it loads into a basic kiosk (desktop, browser and a few low-end utilities.)

Now, let's say a student parks it in front of the same machine.  They whip out their USB device (which in this case looks like a thick business card, similar to the smart-card design, with an embedded flash of so many gigs.)  The LABEL and UUID on this device are unknown, but within the root directory of the device will be a flag file (such as /.persist or the likes) which when the system reboots, this flag file is detected and overrides the existing tce=/opt= settings from within the boot command line.  (In kiosk mode, the user plugs in the USB device, the flag is detected, the user is prompted for authentication to prove the flash drive belongs to them and if successful, the machine reboots.  Upon rebooting, it's the same flag file we need to detect before /opt and sysconfig/tcedir launched.  Within the flag file contains their login information and the date stamp for when they typed it in helping prevent flash cards from being hijacked and avoiding having to retype credentials.

The problem here is the hardware daemon is still loading at this point (boot-up) and I haven't come up with a rock-solid way to detect when USB hubs and associated devices are detected and drivers and running.  (Under RH-like operating systems I'd just probe for usb_storage, sit an extra second for sake of safety and we're off to the races.  I haven't come up with a similar mile-stone under TC's boot process.)

@Rich: Yes, tc-config needs to be modified.  In fact, I have 6 tc-config files launching from rc where most of the original/stock tc-config is tc-config-3 (numbered 0-5) as well as modified tce-load/ab/setup scripts, tc-functions, etc.   It's tc-config-0 which is dedicated to pre-boot-code manipulation.  The sysconfig/tcedir persistence doesn't bug me much as I could just kill the link and re-link elsewhere and not even fuss with tce=SRC=something...  but /opt isn't going to be something I can replace later (after hardware settles) without risking havoc.

Thanks again for everyone's input!

[Rich]

Hi centralware
As I recall, if you don't specify a  tce  directory, the tc-config script scans all drives and uses the first  tce  directory it finds. You
might want to look at that section of code to see if it lends itself to being modified to check for your flag file.