Pick fastest mirror hang

[neonix]

When I choose pick fastest mirror in appsbrowser it can freezee forever, if one of mirrors don't responding.
There should be timeout mechanism in code of mirror picker. Does wget have timeout mechanism?

[curaga]

There is. Wget has one type of timeout by itself and another we patch in.

[neonix]

There is. Wget has one type of timeout by itself and another we patch in.

Google says that default wget timeout is 900 seconds.
Sometimes wget can freezee. Does mirrorpicker couldn't have its own timeout 120 second or SIGKIL?.

[curaga]

Our patch sets the timeout to 10 sec. If you can reproduce this issue, maybe you can find out why wget hangs for this specific url, and we can fix it in wget. "ps aux" would show which wget is still alive.

[CardealRusso]

I've experienced this kind of "freeze" with wget, both on linux and windows.

My suspicion is that, momentarily, your connection to the server becomes absurdly slow, a few bytes/s? which theoretically shouldn't trigger the timeout.

[neonix]

I coudn't recreate this error by simply unplug internet cable.
I think this problem is realted with bad routing or bad DNS.

Few days ago there was problem with this forum and repo.
I have to wait long time when I type forum.tinycorelinux.net in my browser.

[CentralWare]

@neo: Are you using the full wget or the busybox version?  Full wget would, in theory, resort back to its compiled default timeout.  BB wget "should" timeout after 10 seconds of complete silence, but if a mirror answered and error'ed (such as an FTP server which answers the caller, but the DIRECTORY in question no longer exists) wget wouldn't "fail" and thus timeout may get bypassed -- again, this is in theory.

@everyone: Just a suggestion with the mirror system; the way we do things here is to...

• Take /opt/tcemirror and create a fresh mirror list with it (just in case tcemirror contains a custom URL)
• Take mirrors.tcz and put that list of links into our fresh list from above
• PING each of the listed servers (3 times/1 sec interval) with a wget timeout of 10 seconds, KILL the threads that time-out after OUR OWN 5 seconds
• Of the remaining servers which passed the PING test, DOWNLOAD a copy of mirrors.tcz (~4KB) from each; the first to finish replaces /opt/tcemirror

A stripped version is included which simply doesn't include the PING tests and should act as a close replacement to the existing mirror selector.
I didn't build a GUI version; the goal was with or without desktop and for my own personal speed.
(Note: repo.tinycorelinux ignores ICMP so it doesn't allow pinging anyhow, thus why it's left out here.)

OUR TIMEOUT can be set in the first few lines of the script.  This should be <10 seconds none the less.

Code: [Select]

#!/bin/sh

TIMEOUT=5

#### Make sure we're not running as ROOT
if [ "$(whoami)" == "root" ]; then
    . /etc/init.d/tc-functions
    echo "${YELLOW}You must not run this as ROOT${NORMAL}"
    exit 1
fi

#### Make sure we have mirrors.tcz installed and loaded
if [ ! -d /tmp/tcloop/mirrors ]; then
    if [ ! -f /etc/sysconfig/tcedir/optional/mirrors.tcz ]; then
        tce-load -w mirrors.tcz
    fi
    tce-load -i /etc/sysconfig/tcedir/optional/mirrors.tcz
fi

#### Clean and create our testing grounds
mkdir -p /tmp/mymirrors && rm /tmp/mymirrors -fR
mkdir -p /tmp/mymirrors/cmd /tmp/mymirrors/results /tmp/mymirrors/junk

##############################################
#### Go straight through our mirror list and WGET test them all! ####
##############################################
VER=$(version | awk -F. '{print $1}')
URI="${VER}.x/x86/tcz"

# Be sure we include /opt/tcemirror in case it's custom #
if [ ! -f /tmp/mymirrors.txt ]; then
    cat /opt/tcemirror > /tmp/mymirrors.txt
    cat /usr/local/share/mirrors >> /tmp/mymirrors.txt
fi

CNT=0
clear
while read -r line
do
    test=$(echo $line)
    if [ ! "${test}" == "" ]; then
        SERV=$(echo $line | awk -F"/" '{print $3}' | awk -F: '{print $1}')
        echo "#!/bin/sh" > /tmp/mymirrors/cmd/${CNT}.sh
        echo "wget --no-check-certificate -O /tmp/mymirrors/junk/${SERV} ${line}${URI}/mirrors.tcz 2>/tmp/junk.txt" >> /tmp/mymirrors/cmd/${CNT}.sh
        echo "echo \"${line}\" >> /tmp/mymirrors/test2.lst" >> /tmp/mymirrors/cmd/${CNT}.sh
        chmod +x /tmp/mymirrors/cmd/${CNT}.sh
        CNT=$(expr $CNT + 1)
    fi
done < /tmp/mymirrors.txt

echo "Testing Mirrors..."
cd /tmp/mymirrors/cmd
for file in *
do
    sh $file >/dev/null 2>&1 &
done

sleep $TIMEOUT

# Kill only processes which are WGET and contain "junk" (2>/tmp/junk.txt) JUST IN CASE :)
for item in $(ps -a | grep wget | grep junk | grep -v grep | grep -v sh | awk '{print $1}')
do
    sudo kill $item
done

while read -r line
do
    if [ ! "${line}" == "" ]; then
        echo $line > /opt/tcemirror
        break
    fi
done < /tmp/mymirrors/test2.lst

cp /tmp/mymirrors/test2.lst /opt/tcemirror.lst
cat /opt/tcemirror
cd /opt
rm /tmp/mymirrors -fR


[CentralWare]

SIDE NOTE: Our repositories are on-site, so they're virtually never down and we usually don't disconnect ethernet or wireless so I didn't think to put outages into consideration in the enclosed demo.  It may be prudent to put in a backup to prevent /opt/tcemirror from ever being empty when "online" isn't on-line.

[neonix]

I think I am using busybox's wget becouse the problem occurs on fresh instalation.
Does wget has retry option? Here is example of bad address.
wget http://mirror.uce.edu.ec/tinycorelinux/

I also noticed that wget can behave differently on moblile internet and LAN internet.
Sometimes when I installed mc.tcz on 2G (20 KB/s) it freeze with stalled communicate,
Now I use (LTE 300 KB/s) but sometimes I have to download again becouse wget loose signal.

If there was kill button in mirror picker it will solve my problem.

[CentralWare]

@neonix: The UCE mirror "answers" so to speak (which is why even in your browser it waits a LOT longer than usual before timing out) however they're seemingly having technical issues with their server(s) or their network itself which wget would never be able to predict.  This could be a routing problem (where a visitor is being routed to network A, then network B and then C - but then gets redirected back to A because "D" says there's a problem - all of which wget has no clue about) from the way it's acting...  I didn't dig too deeply to investigate.

Suggestion: Take the above script and COPY the contents into a file called mymirrors and place it into /opt; then run it from the shell/terminal:

Code: [Select]

cd /opt
sh mymirrorsThe UCE mirror will likely act in the same fashion as it's doing right now and get skipped after the five second time-out in the script (not wget's timeout, that's ignored by our method.)
When the above script is completed, it will show you the "fastest" mirror to your location and update /opt/tcemirror to reflect the changes.
Once that's done, try a fresh download (ie: try installing MC again)

NOTES:

• If you're on a slow internet/network you may need to increase the script's time-out to compensate
• The script does not test/detect an active internet connection; it's assumed you're already online
• The script doesn't know how your persistence is set up; when you reboot, if /opt isn't persistent, the changes and "mymirrors" will be lost.