Fastest way of deleting first line of a large file

[Rich]

Recently while formatting a log file I ran into a problem
caused by the first line in the file. I originally I used this
command to remove it (sed.tcz was installed):

Code: [Select]

sed -i '1d' FilenameWhen I tried it on a larger test file I felt it was a little slow.
So I ran some timing tests on several different techniques.

Running under TC10 x86 Dell Dimension E310:

Code: [Select]

tc@E310:~/Scripting/Timing$ ./DeleteHeaderTiming.sh

# Delete first line from Test.Log timing test.
File Test.Log size: 4102673 bytes containing 35557 lines

 # Alter original file, GNU.
 0.438    /usr/local/bin/sed -i '1d' Test.Log

 # Alter original file, busybox.
 0.094    busybox sed -i '1d' Test.Log

 # Using a second file, GNU.
 0.068    /usr/local/bin/sed '1d' Test.Log > Test.Log2

 # Using a second file, busybox.
 0.094    busybox sed '1d' Test.Log > Test.Log2

 # Using a second file, GNU.
 0.030    /usr/local/bin/tail -n +2 Test.Log > Test.Log2

 # Using a second file, busybox.
 0.026    busybox tail -n +2 Test.Log > Test.Log2

 # Using a second file, GNU.
 0.208    /usr/local/bin/awk 'NR>1' Test.Log > Test.Log2

 # Using a second file, busybox.
 0.536    busybox awk 'NR>1' Test.Log > Test.Log2

 # Using a second file, GNU.
 0.147    /usr/local/bin/awk 'NR==1 { print > /dev/null; next } 1' Test.Log > Test.Log2

 # Using a second file, busybox.
 0.546    busybox awk 'NR==1 { print > /dev/null; next } 1' Test.Log > Test.Log2
Turns out my choice was one of the slower ones. Interestingly, it
turns out the busybox version would have been a better choice.
Fastest was this solution:

Code: [Select]

tail -n +2 Test.Log > Test.Log2Considering how close the execution times were, I'd say the GNU
and busybox versions are pretty much equal.

I decided to repeat the test on a more capable machine.
TC14 x86_64 HP G62:

Code: [Select]

tc@HP-G62:~/SystemHealth$ ./DeleteHeaderTiming.sh

# Delete first line from Test.Log timing test.
File Test.Log size: 4102779 bytes containing 35558 lines

 # Alter original file, GNU.
 0.063    /usr/local/bin/sed -i '1d' Test.Log

 # Alter original file, busybox.
 0.076    busybox sed -i '1d' Test.Log

 # Using a second file, GNU.
 0.033    /usr/local/bin/sed '1d' Test.Log > Test.Log2

 # Using a second file, busybox.
 0.058    busybox sed '1d' Test.Log > Test.Log2

 # Using a second file, GNU.
 0.014    /usr/local/bin/tail -n +2 Test.Log > Test.Log2

 # Using a second file, busybox.
 0.010    busybox tail -n +2 Test.Log > Test.Log2

 # Using a second file, GNU.
 0.123    /usr/local/bin/awk 'NR>1' Test.Log > Test.Log2

 # Using a second file, busybox.
 0.162    busybox awk 'NR>1' Test.Log > Test.Log2Once again, the  tail  version was the clear winner.
Once again, the execution times were so close that the GNU and
busybox versions could be considered to be equal.

Prior to creating a file, each technique begins with:

Code: [Select]

# Make sure all tests start with a clean cache.
sudo cache-clear
sync

# Create our test file.
busybox cp -f "$File" "$TestFile"

# Remove any previous results.
rm -f "$TestFile2"