Tiny Core Linux
General TC => Programming & Scripting - Unofficial => Topic started by: bigpcman on January 26, 2010, 05:31:30 PM
-
I noticed when I use the ( cd / ; sudo cpio -pud /tmp/package < /mnt/sda1/myfilelist.lst) command, directorys in the package directory have not maintained their original ownership. For instance:
tc@box:/tmp/package$ ls -l
drwxr-xr-x 2 tc staff 80 Jan 26 19:14 etc/
drwxr-xr-x 3 tc staff 60 Jan 26 18:25 home/
tc@box:/tmp/package$ cd /
tc@box:/$ ls -l
drwxr-xr-x 2 root root 1400 Jan 26 18:07 bin/
drwxr-xr-x 10 root root 9260 Jan 26 18:13 dev/
drwxr-xr-x 9 root root 780 Jan 26 18:38 etc/
If I execute the command as root things get even worse, /home and tc become owned by root.
root@box:/tmp# ( cd / ; cpio -pud /tmp/package < /mnt/sda1/myfilelist.lst)
3 blocks
root@box:/tmp# cd package/
root@box:/tmp/package# ls -l
drwxr-xr-x 2 root root 80 Jan 26 19:36 etc
drwxr-xr-x 3 root root 60 Jan 26 19:36 home
In the first example the files in etc have the correct ownership. In the second example the files in /home/tc seem to be correct as well..
Any ideas as to how to fix this?
-
I noticed when I use the ( cd / ; sudo cpio -pud /tmp/package < /mnt/sda1/myfilelist.lst) command, directorys in the package directory have not maintained their original ownership. For instance:
tc@box:/tmp/package$ ls -l
drwxr-xr-x 2 tc staff 80 Jan 26 19:14 etc/
drwxr-xr-x 3 tc staff 60 Jan 26 18:25 home/
tc@box:/tmp/package$ cd /
tc@box:/$ ls -l
drwxr-xr-x 2 root root 1400 Jan 26 18:07 bin/
drwxr-xr-x 10 root root 9260 Jan 26 18:13 dev/
drwxr-xr-x 9 root root 780 Jan 26 18:38 etc/
Is this really the non-sudo case? If you are executing as user tc, you cannot create files owned by root, so you would get these ownerships.
If I execute the command as root things get even worse, /home and tc become owned by root.
root@box:/tmp# ( cd / ; cpio -pud /tmp/package < /mnt/sda1/myfilelist.lst)
3 blocks
root@box:/tmp# cd package/
root@box:/tmp/package# ls -l
drwxr-xr-x 2 root root 80 Jan 26 19:36 etc
drwxr-xr-x 3 root root 60 Jan 26 19:36 home
In the first example the files in etc have the correct ownership. In the second example the files in /home/tc seem to be correct as well..
Any ideas as to how to fix this?
I am not sure to which examples you are referring. Is it the non-sudo and sudo cases or the /etc and /home cases of the second example output.
If cpio executes as a non-root user, the ownership rights are set for that user.
If cpio executes as root, the ownership rights are copied from the original files.
Create two files owned by tc and root:
echo date >test.tc
sudo sh -c "echo date >test.root"
Execute cpio as tc:
( cd .
{ echo test.tc ; echo test.root ; } | cpio -pud /tmp/test_tc
ls -l /tmp/test_tc
)
-rw-r--r-- 1 tc staff 5 Jan 27 02:28 test.root
-rw-r--r-- 1 tc staff 5 Jan 27 02:28 test.tc
File test.root is owned by user tc since that user cannot create files owned by root.
Execute cpio as root:
( cd .
{ echo test.tc ; echo test.root ; } | sudo cpio -pud /tmp/test_root
ls -l /tmp/test_root
)
-rw-r--r-- 1 root staff 5 Jan 27 02:28 test.root
-rw-r--r-- 1 tc staff 5 Jan 27 02:28 test.tc
In this case test.root is owned by root and test.tc is owned by tc since root has the power to ownerships arbitrarily.
Please try your question again if this has not helped.
-
In the example below I have a list of files that are in /etc and /home/tc. There are a mix of files owned by root and tc. I have executed cpio as root. All the file ownership is preserved correctly.
However the directory level ownership has changed from tc for /home and /home/tc to root.
edit: Sorry for the confusion - The permission on the /home directory is correct, the problem is the /home/tc directory changed from tc to root.
tc@box:~$ cd ..
tc@box:/home$ ls -l
drwxr-sr-x 6 tc staff 340 Jan 27 07:01 tc/
tc@box:/home$ sudo su
root@box:~# ( cd / ; cpio -pud /tmp/package < /mnt/sda1/myfilelist.lst)
4 blocks
root@box:~# cd /tmp/package/home
root@box:/tmp/package/home# ls -l
drwxr-xr-x 2 root root 80 Jan 27 07:02 tc
root@box:/tmp/package/home#
-
Using passthrough mode with find will preserve the target directories' permissions.
cd /tmp/package
find . -depth -not -type d -print | sudo cpio -pumd / 2>&1 > /dev/null
-
Using passthrough mode with find will preserve the target directories' permissions.
cd /tmp/package
find . -depth -not -type d -print | sudo cpio -pumd / 2>&1 > /dev/null
I used "-p" passthrough mode in my example.
I'm not sure what your command does but I'm trying to create the "package container" comprised of the files in my list with full paths intact.
-
The permission on the /home directory is correct, the problem is the /home/tc directory changed from tc to root.
Thanks for the clarification.
I tried your example with both Busybox and GNU cpio and got the same behavior, which in retrospect makes some sense to me. If cpio is executing as root with the "-d" option to create directories and the only reference to /home/tc is to a file within that directory, then cpio knows nothing about the ownership of /home/tc and creates it with its own (root) ownership.
The simplest solution is to put "home/tc" in your file list so cpio will give /home/tc the right ownership. ("home/tc" can go anywhere in the list, even after all of the files in /home/tc. This is important if the file list is being generated using "find ... -depth ..." which lists directories after the files within them.)
-
The permission on the /home directory is correct, the problem is the /home/tc directory changed from tc to root.
Thanks for the clarification.
I tried your example with both Busybox and GNU cpio and got the same behavior, which in retrospect makes some sense to me. If cpio is executing as root with the "-d" option to create directories and the only reference to /home/tc is to a file within that directory, then cpio knows nothing about the ownership of /home/tc and creates it with its own (root) ownership.
The simplest solution is to put "home/tc" in your file list so cpio will give /home/tc the right ownership. ("home/tc" can go anywhere in the list, even after all of the files in /home/tc. This is important if the file list is being generated using "find ... -depth ..." which lists directories after the files within them.)
Thanks for your solution. I worry that this is a very easy thing to screw up by simply missing one entry or forgetting to perform this step.
Perhaps my original "tar" approach works better to eliminate the possibility of error. Using "tar" to compress files from a list and then uncompressing them to "package" seems to maintain all ownership and permissions correctly. What do you think?
-
A tar pipeline would also work.
tar -C //tmp/package -cvpf - . | tar -C / -xvf -
I didn't actually test for errors but that is the basic routine.
-
A tar pipeline would also work.
tar -C //tmp/package -cvpf - . | tar -C / -xvf -
I didn't actually test for errors but that is the basic routine.
Wow, that is quite a trick Jason. However, I don't see how it could use a list file to create the compressed archive. Anyway here are the two commands I had in mind:
tar -C / -T /mnt/sda1/myfilelist.lst -zcf /tmp/backup_name.tar.gz
tar -xf /tmp/backup_name.tar.gz -C /tmp/packagePerhaps these two commands can be arranged in a pipeline?
-
I would use rsync:
sudo rsync -av --files-from=/mnt/sda1/myfilelist.lst / /tmp/package
Rsync is a very valuable tool for backup-related tasks.
-
Plain old cp -a is even simpler, and cp is in base. It would be my preference.
cp -a /tmp/package/* /
Some preserve upstream directory perms, and others preserve existing target directory perms. In extensions, we decided to go with preserving upstream extension directory perms to give the extension maker control over created directories.
But on your own system, there are many ways to move files.
-
cp -a /tmp/package/* /
Bigpcman's original question was about using a list of files to specify a subset of / to copy to /tmp/package. Your example is copying in the opposite direction.
-
Granted, I thought the goal was to copy from /tmp/package to /. But this would also work both ways if constructed for each:
tar -C / -T /mnt/sda1/myfilelist.lst -cpf - | (cd /tmp/package ; sudo tar xf -)
Cpio, rsync, tar pipe work with each having their own behavior of preserving perms. Rsync preserves upstream directory perms over existing ones, tar pipe will not overwrite dir perms if the dirs themselves are not tarred.
-
Just curious. Do busybox and GNU tar behave on the same way?
-
Gnu tar does not require the leading / removed from the entries in the list files. Other than that seems the same for above command.
-
Some preserve upstream directory perms, and others preserve existing target directory perms. In extensions, we decided to go with preserving upstream extension directory perms to give the extension maker control over created directories.
Since I am essentially making an extension it would seem rsync would be the best tool to use.
Jason your tar command is quite a master piece.
"tar -C / -T /mnt/sda1/myfilelist.lst -cpf - | (cd /tmp/package ; sudo tar xf -)"
Could you explain how it works.
I get the simple stuff but I don't get the "- | (...)" and the "tar xf -)" parts.
Sorry to keep bothering you.
-
We used a tar pipe during a release candidate, and from what I understand:
From the / directory, tar up the stuff in the list file, and pipe it to tar to extract it in the /tmp/package directory.
-
I get the simple stuff but I don't get the "- | (...)" and the "tar xf -)" parts.
The "-f" option is used to tell tar where to write or read a tar file; "-" means standard output when creating a tar file and standard input when extracting a tar file. The default is to create tar files on standard output and read tar files from standard input, so the "-f -" options not needed.
tar -c ... | (cd /tmp/package ; tar -x ...)
creates a tar file on standard output where it is piped into subshell that executes a tar extraction process within the directory /tmp/package. I believe that that is equivalent to
tar -c ... | tar -x -C /tmp/package ...
You can also do things like
tar -c ... | ssh tc@somewhere.else tar -x -C /tmp/package
transfer files from one machine to the directory /tmp/package on a remote machine. (Rsync can also transfer files over an ssh link.)
-
As it turns out, tar does not maintain directory permissions as I thought it did. I did some more testing as follows:
tc@box:/tmp$ sudo rm -R package/
tc@box:/tmp$ mkdir package
tc@box:/tmp$ sudo tar -C / -T /mnt/sda1/myfilelist.lst -czf /mnt/sda1/restoremys
tc@box:/tmp$ sudo tar xvzf /mnt/sda1/restoremysys.tar.gz -C /tmp/package
etc/sudoers
etc/shadow
home/tc/.wbar
home/tc/.profile
tc@box:/tmp$ cd package/home/
tc@box:/tmp/package/home$ ls -l
drwxr-xr-x 2 root root 80 Jan 27 17:45 tc/
The /home/tc directory ownership was changed from tc to root.
So it looks like if directorys don't exist the new ones that are created use the current user name just like cpio does.
However, "rsync" appears to get it right:
tc@box:/tmp$ sudo rsync -av --files-from=/mnt/sda1/myfilelist.lst / /tmp/package
building file list ... done
created directory /tmp/package
etc/
etc/shadow
etc/sudoers
home/
home/tc/
home/tc/.profile
home/tc/.wbar
sent 1373 bytes received 97 bytes 2940.00 bytes/sec
total size is 1036 speedup is 0.70
tc@box:/tmp$ cd package/
tc@box:/tmp/package$ ls -l
drwxr-xr-x 2 root root 80 Jan 27 17:38 etc/
drwxr-xr-x 3 root root 60 Jan 27 17:28 home/
tc@box:/tmp/package$ cd home/
tc@box:/tmp/package/home$ ls -l
drwxr-sr-x 2 tc staff 80 Jan 27 17:39 tc/
This has all been quite a learning experience. Thank you Vitex and Jason for providing all the information.
-
That's because the directories weren't included whereas the rsync one did.
This is probably due to -not -type d
-
That's because the directories weren't included whereas the rsync one did.
This is probably due to -not -type d
Could you elaborate on this. Is the above code to be added to the cpio or tar command?
-
Whem creating the list file, using "find -not -type d" is what excludes the directories. If you want to preserve upstream directory perms, then do not use "-not -type d" with find and tar will preserve upstream directory perms when working from the list.
-
Whem creating the list file, using "find -not -type d" is what excludes the directories. If you want to preserve upstream directory perms, then do not use "-not -type d" with find and tar will preserve upstream directory perms when working from the list.
Thanks for the tip.
I'm not using find to create my file list.
I'm using the tc application "/usr/bin/add2file myfilelist.lst"
This works great for adding files to a list that are scattered around the system.
I assumed this is what is used by the tc system for adding files to the "backup" list which are then "tar'ed"
Here's my file list:
tc@box:/mnt/sda1$ cat myfilelist.lst
home/tc/.profile
home/tc/.wbar
etc/shadow
etc/sudoers
What would have to be different?
-
Add any directories that you want to myfilelist.lst
-
Add any directories that you want to myfilelist.lst
Vitex brought this up as well.
So if I added:
home
home/tc
etc
to the myfilelist.lst then the directories are created with the correct ownership. Is that what you mean.
myfilelist.lst
home/tc/.profile
home/tc/.wbar
etc/shadow
etc/sudoers