Fluff FileMgr directory filecount bug and workaround

[MikeLockmoore]

It's embarrassing that I did not catch this earlier  , but there is a bug in the version of Fluff (0.8.5) included in TC Base 3.4.  When you open the Properties window  for a directory, Fluff should count the number of files inside and estimate the used storage size.  However, Fluff uses a hard-coded path (/usr/local/bin) to a helper script to do this that does not match the actual arrangement in TC Base.

Fortunately, there is a straightforward work-around that will make the feature work properly.  You can create a symbolic link in /usr/local/bin back to the version in /usr/bin and make it persistent:

Code: [Select]

sudo ln -s /usr/bin/fluff_fc.sh /usr/local/bin/fluff_fc.sh
echo "usr/local/bin/fluff_fc.sh" >> /opt/.filetool.lst

A fix for Fluff will be prepared for later releases of TC Base.  However, this workaround should not cause any problems if left in place.

EDIT: That was quick...  TC 3.4.1 was released within hours of me posting this bug and releasing a fixed Fluff (version 0.8.6) to the Core Team.  So you may either upgrade TC or use the workaround above as you see fit.
--
Mike Lockmoore

[Rich]

Hi Mike
Nice program, but  I think I may have found a few quirks.

1. When I go to the /tmp/tcloop directory not all of the entries show up. For example, pci-utils shows
    up under mc and ls at the command line but not under fluff. Yes, "show all" is checked.
2. Under properties I think you may have reversed the Creation and Modification dates.
3. The file pane seems to repaint needlessly. If I start the Arora web broswer the pane repaints
    about a dozen times. A simple way to reproduce this behaviour is to open a terminal window
    and alternate between 2 commands such as free and uname for example.

I am currently running TCL3.4, fluff beta ver. 0.8.x Dec. 2010 according to the help file and the
above mentioned fixes implemented. If you need more information let me know.

[MikeLockmoore]

Thanks for your report, Rich.  Can you tell me what filesystem(s) you see these issues with? (Ext3, NTFS, etc.)

1) The current way Fluff gathers filecount and size info about a directory's contents is a bit of a hack... I wrote an external script that uses the recursive "find" command-line utility and the command-line disk usage utility:

Code: [Select]

#!/bin/sh
# Assess size and recursive filecount within a specified directory

diskspace=`du -shL $1 2> /dev/null`
space_incomplete=$?
filecount=`find $1 -type f -o -type l $2 2> /dev/null > /dev/null`
count_incomplete=$?
filecount=`find $1 -type f -o -type l $2 2> /dev/null | wc -l`

incomplete=0

if [ $space_incomplete -ne 0 ]; then
    incomplete=1
fi
if [ $count_incomplete -ne 0 ]; then
    incomplete=1
fi

echo "incomplete?: $incomplete; filecount: $filecount; space: $diskspace"  
I did this to avoid the work and code space to write my own tree-traversal routine to gather this info.  Unfortunately, there seems to be a few things that can lead to invalid or misleading results from the current method.  I know these stats are sometimes handy, but I don't want to bloat up Fluff much.  Can anyone offer a better method to gather these statistics in the script?  If not, maybe I will try to write a tight tree-traversal method of gathering the info, or maybe I'll drop the directory contents statistics completely.  Will this info be missed much?

2) For the Creation and Modification dates, here is the code:

Code: [Select]

   
    mdate_dsp_p->copy_label(ctime(& fi_p->status.st_mtime));
    cdate_dsp_p->copy_label(ctime(& fi_p->status.st_ctime));
    adate_dsp_p->copy_label(ctime(& fi_p->status.st_atime));
The .st_ctime and .st_mtime are data elements returned by the the posix stat() routine (file STATistics) in a data structure.  This _should_ be ok.  I'll double-check... maybe I made some other mistake, but maybe the filesystem is not reporting the dates properly?  Does anyone else see the dates backwards?

3) The repainting is triggered by "inotify" events from the Linux kernel.  If Arora is updating info in a settings file or any other file that is included in your session of Fluff, you will see the repaint.  Perhaps Fluff could be a bit lazy about repaints and only do a repaint if nothing else happens for a reasonable time period (so multiple successive repaints are consolodated).  FLTK also offers double-buffered screen refresh, so if I activate that feature, the repainting won't be so noticable.  But it could impact resource usage (speed, memory).  

For all three issues, maybe I will experiment to see what can be done with minimal impact on size and performance.  
--
ML

[Rich]

I am booting from a CD and my applications reside on a FAT32 USB memory stick. While my
programming and scripting skills are weak at best I do get the jist of what you are doing. That
said, I would not call piping the output of find through wordcount a hack, I'd call it creative. My
problem was not filecount or size though. When I go to /tmp/tcloop not all of the subdirectories
show up in the file panel. It never occurred to me that the system might be reporting the Creation
and Modification dates transposed, so I checked using ls and you are correct, the system has
it backwards, files are listed as modified before being created. On Inotify, would it not make more
sense to repaint only if the currently displayed directory changes?

[MikeLockmoore]

Rich: oh, I see.  Then #1 is more serious than I first thought.  I will try to duplicate and try to find a root cause.  Fluff uses posix system calls like readdir() to explore the file tree.  I'm not sure why or how it can fail yet.

As far as repainting, yes, it might make more sense to only repaint the list if something visible in the app changes, but there can be times that it hard to tell if the changing thing _is_ visible, or affects something visible (e.g. parent directory deleted).  However I agree that noticeable flashing from repainting is annoying, so I will try to find a solution that either repaints more needfully or in a less intrusive way.

[tinypoodle]

It never occurred to me that the system might be reporting the Creation
and Modification dates transposed, so I checked using ls and you are correct, the system has
it backwards, files are listed as modified before being created.

Expectable, particularly after a restore of TC's backup.

Here's a detailed explanation of mtime, ctime and atime which I find very enlightening:

http://www.unix.com/tips-tutorials/20526-mtime-ctime-atime.html

[Rich]

Hi Mike
Now you've got it. I only see this behaviour on /tmp/tcloop, I do not see it in any other subdirectories.
I understand, something visible on the tree in the left panel might change.

Hi Tinypoodle
Ok, so it's not creation time but change time. So in a nutshell modification time changes when the
contents of the file change and change time changes when the directory entry of the file changes.
Thanks for the link, it was enlightening, and sorry for accusing the operating system of lying.

[MikeLockmoore]

OK, I understand why Fluff is not showing many of the files in /tmp/tcloop... one of my key assumptions appears to be broken.  

As Fluff scans through directory contents, it checks the "inode" number of the file returned by the stat() routine.  If it sees the same inode number for another item in the same directory, Fluff assumes it is a file is has already added to the list of things to display, so it does not create a new entry in the list, but just updates the existing entry with the same inode number.  However, it seems that many of the directories in /tmp/tcloop have the same inode number.  To see this:

   ls -i1 /tmp/tcloop | sort

The duplication of inode numbers is really unexpected by me.    I seem to remember reading somewhere inode numbers should be unique to each file (including directories), unless more than one filename is hard-linked to the same underlying file.  Can anyone explain this?  If inode is not a truly distinct ID number, I guess Fluff will need to go back to a slower alphabetic comparison between the files' names.

UPDATE: I have switched the "do I know you?" test from an inode number comparison to a full string comparison of the file name, which seems to fix Rich's issue #1.  The source code for this new version (0.8.7) will be posted at http://retrospectiva.krautsoft.de/projects/useful-lightweight-software/browse.  This version also fixes a regression (reappearing bug) for the file "clone" feature (copy a file and paste it back into the same directory... Fluff is intended to create a new copy with the name modified with the suffix "(copy1)" for the first copy, and so on). This stopped working in version 0.8.6, but works in 0.8.7.  Compile it if you want to get it working immediately, or wait a bit for a new TC release.
--
ML

[Rich]

Hi Mike,
While I do not fully understand how the inode structures work I believe I have a partial answer.
The /tmp/tcloop is a directory of mount points. According to a couple of sites I looked at the
inode gets overlayed when a file system gets mounted to it. Try this and you will see what I mean.
Go to your /mnt directory and use  ls -i1 | sort. Then mount or umount a drive and do it again and
you will see the inode for that drive change. Hope this helps.

[maro]

I'm not overly suprised about the not unique inode numbers in '/tmp/tcloop': Inodes are unique per file system (or mount point), each extension is a seperate mount point, hense each of those can use for its root (e.g. '/tmp/tcloop/EXT1' with inode 1000) an inode number which is the same as for another extension (e.g. '/tmp/tcloop/EXT2', again with inode 1000).

[MikeLockmoore]

Rich, maro: Yes, in retrospect it should not have been a big surprise. TinyCore uses overlays and mounts a lot more than other distributions, so this impacts us more than it might if we were running on another type of Linux. 

So now Fluff won't use inode for any identification purposes.  I found another spot were Fluff 0.8.7 still referenced inodes, which I just removed.  So Fluff from 0.8.8 will be inode-free. 
--
Mike

[MikeLockmoore]

Rich:  I have added a check in the inotify event handler to determine if the event's associated filename appears anywhere in the directory tree or the file details window.  If so, the "need to synch" flag is set and the screen will soon get a full repaint request (as in the older versions of Flume).  However, if the filename does not appear in the directory tree or filelist, it will be ignored and not repainted.  Note this method is not precise, since the new code does not check full paths, so it is possible that the event is actually associated with a non-visible file, but happens to have the same name as something currently visible.  However, as a quick and compact way to eliminate most of the annoying flickering, it should work fairly well.

Note that .ash_history is a file that will be updated (and the modification time changed) each time you enter something new into the command-line.  So now, if you "show all" in the Home directory, you should see the date change for this file's list item when the command-line is used.  But if "show all" is unchecked, you should not see .ash_history in the list, and you will see any update flickers either.

This update should also be in the version 0.8.8 that I will post to retrospectiva soon ().
--
Mike
   

[Rich]

Thanks Mike, I'm looking foward to trying it out.

[MikeLockmoore]

One more tweak: the repaint avoidance was a bit too aggressive... version 0.8.9 strikes a better balance.  A new version will be passed along to the Core team.