Teej
10-29-2006, 03:10 AM
It seems there's a transparent HTTP proxy somewhere on the route between the VPS and the outside world that is breaking HTTP connections on port 80.
Over the weekend I've been trying to transfer several large gzip files (60MB, 250MB) from my local web server.
From a shell on the VPS I used wget to fetch the data files but the connection was frequently and consistently freezing after as few as 50KB or as many as 2MB.
]# wget http://domain.tld/archive.tar.gzip
I then used the FTP client to connect to the source-server's FTP service. Sessions ran fine but unfortunately (due to sleep deprevation I think!) I ran them in ASCII mode instead of BINARY and the transferred files were therefore corrupt :eek: Then, mid-way through the second attempt, a network outage killed the binary transfers!
So I decided to have another go with wget, this time setting a short time-out and allowing it to continue where it left off. This was suffering frequent read-errors, despite using a command-line designed to cope with time-outs:
]# wget -dc --read-timeout=10 -w 10 http://domain.tld/archive.tar.gzip
Setting --continue (continue) to 1
Setting --read-timeout (readtimeout) to 10
Setting --wait (wait) to 10
DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
...
Disabling further reuse of socket 3.
Closed fd 3
06:30:05 (1.93 KB/s) - Read error at byte 4327031/59308011 (Connection timed out). Retrying.
...
Disabling further reuse of socket 3.
Closed fd 3
06:30:31 (6.36 KB/s) - Read error at byte 4428863/59308011 (Connection timed out). Retrying.
And those errors just go on and on and on!
To begin with I thought it was a problem at the source, but then I recalled something that affected thousands of customers when I was IT Director of a large ISP here in the UK - a badly configured Transparent Proxy at one of the peering partners.
To check this out I added port 81 to the list of listening ports on the source web server and then started a fresh transfer with wget from port 81:
]# wget -dc --read-timeout=10 -w 10 http://domain.tld:81/archive.tar.gzip
Setting --continue (continue) to 1
Setting --read-timeout (readtimeout) to 10
Setting --wait (wait) to 10
DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
206 Partial Content
Registered socket 3 for persistent reuse.
Length: 59,308,011 (57M), 54,879,148 (52M) remaining [application/x-gzip]
100%[++++======================================================>] 59,308,011 24.09K/s ETA 00:00
07:10:44 (22.40 KB/s) - `archive.tar.gzip' saved [59308011/59308011]
As you can see the transfer went perfectly with no connection problems. I'm monitoring the bandwidth being used on the source-server and its fully saturated all the time - no pauses or breaks.
So, the question is, where is the Transparent Proxy server, who controls it, and how to get it fixed?
I'm pretty sure its not on my ISP's network because similar transfers to other locations work perfectly over port 80. Also (at least late-Friday/early-Saturday) I used wget to fetch a bunch of open-source archives from various sourceforge mirrors (in the USA and UK) to the VPS without any problems. This could be an intermitent or temporary issue but it'd be good to pin it down!
Here's the route:
traceroute to 84.12.34.246, 30 hops max, 38 byte packets
1 vzlin2.primarydns.com (72.4.174.5) 0.052 ms 0.028 ms 0.022 ms
2 207.234.128.46 (207.234.128.46) 0.261 ms 0.224 ms 0.241 ms
3 tdcscr05-vl-17.miakc.ustdata.net (66.231.240.53) 42.847 ms 22.057 ms 0.729 ms
4 g1.ba21.b015447-0.mia01.atlas.cogentco.com (66.28.21.125) 0.870 ms 0.849 ms 0.724 ms
5 g9-1-3525.core01.mia01.atlas.cogentco.com (38.112.35.177) 1.745 ms 1.826 ms 1.989 ms
6 p5-0.core01.atl01.atlas.cogentco.com (66.28.4.138) 9.367 ms 9.340 ms 9.111 ms
7 p13-0.core01.atl01.atlas.cogentco.com (154.54.3.197) 15.862 ms 15.961 ms 15.481 ms
8 p1-0.core01.jfk02.atlas.cogentco.com (154.54.5.97) 99.813 ms 109.906 ms 42.591 ms
9 p15-0.core02.jfk02.atlas.cogentco.com (66.28.4.14) 33.347 ms 33.084 ms 33.208 ms
10 p4-0.core02.lon01.atlas.cogentco.com (66.28.4.254) 103.310 ms 103.784 ms 103.789 ms
11 t2-4.mpd02.lon01.atlas.cogentco.com (130.117.2.22) 106.310 ms 106.158 ms 106.170 ms
12 te1-3.cr05.tn5.bb.pipex.net (195.66.224.29) 106.687 ms 106.756 ms 106.686 ms
13 ae1-0.cr02.gs1.dsl.pipex.net (62.72.137.14) 106.525 ms 106.522 ms 106.679 ms
14 g4-0.ar20.gs1.dsl.pipex.net (84.12.224.30) 105.560 ms 104.658 ms 104.303 ms
15 84.12.34.241 (84.12.34.241) 574.434 ms 558.906 ms 388.389 ms
16 *
Over the weekend I've been trying to transfer several large gzip files (60MB, 250MB) from my local web server.
From a shell on the VPS I used wget to fetch the data files but the connection was frequently and consistently freezing after as few as 50KB or as many as 2MB.
]# wget http://domain.tld/archive.tar.gzip
I then used the FTP client to connect to the source-server's FTP service. Sessions ran fine but unfortunately (due to sleep deprevation I think!) I ran them in ASCII mode instead of BINARY and the transferred files were therefore corrupt :eek: Then, mid-way through the second attempt, a network outage killed the binary transfers!
So I decided to have another go with wget, this time setting a short time-out and allowing it to continue where it left off. This was suffering frequent read-errors, despite using a command-line designed to cope with time-outs:
]# wget -dc --read-timeout=10 -w 10 http://domain.tld/archive.tar.gzip
Setting --continue (continue) to 1
Setting --read-timeout (readtimeout) to 10
Setting --wait (wait) to 10
DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
...
Disabling further reuse of socket 3.
Closed fd 3
06:30:05 (1.93 KB/s) - Read error at byte 4327031/59308011 (Connection timed out). Retrying.
...
Disabling further reuse of socket 3.
Closed fd 3
06:30:31 (6.36 KB/s) - Read error at byte 4428863/59308011 (Connection timed out). Retrying.
And those errors just go on and on and on!
To begin with I thought it was a problem at the source, but then I recalled something that affected thousands of customers when I was IT Director of a large ISP here in the UK - a badly configured Transparent Proxy at one of the peering partners.
To check this out I added port 81 to the list of listening ports on the source web server and then started a fresh transfer with wget from port 81:
]# wget -dc --read-timeout=10 -w 10 http://domain.tld:81/archive.tar.gzip
Setting --continue (continue) to 1
Setting --read-timeout (readtimeout) to 10
Setting --wait (wait) to 10
DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
206 Partial Content
Registered socket 3 for persistent reuse.
Length: 59,308,011 (57M), 54,879,148 (52M) remaining [application/x-gzip]
100%[++++======================================================>] 59,308,011 24.09K/s ETA 00:00
07:10:44 (22.40 KB/s) - `archive.tar.gzip' saved [59308011/59308011]
As you can see the transfer went perfectly with no connection problems. I'm monitoring the bandwidth being used on the source-server and its fully saturated all the time - no pauses or breaks.
So, the question is, where is the Transparent Proxy server, who controls it, and how to get it fixed?
I'm pretty sure its not on my ISP's network because similar transfers to other locations work perfectly over port 80. Also (at least late-Friday/early-Saturday) I used wget to fetch a bunch of open-source archives from various sourceforge mirrors (in the USA and UK) to the VPS without any problems. This could be an intermitent or temporary issue but it'd be good to pin it down!
Here's the route:
traceroute to 84.12.34.246, 30 hops max, 38 byte packets
1 vzlin2.primarydns.com (72.4.174.5) 0.052 ms 0.028 ms 0.022 ms
2 207.234.128.46 (207.234.128.46) 0.261 ms 0.224 ms 0.241 ms
3 tdcscr05-vl-17.miakc.ustdata.net (66.231.240.53) 42.847 ms 22.057 ms 0.729 ms
4 g1.ba21.b015447-0.mia01.atlas.cogentco.com (66.28.21.125) 0.870 ms 0.849 ms 0.724 ms
5 g9-1-3525.core01.mia01.atlas.cogentco.com (38.112.35.177) 1.745 ms 1.826 ms 1.989 ms
6 p5-0.core01.atl01.atlas.cogentco.com (66.28.4.138) 9.367 ms 9.340 ms 9.111 ms
7 p13-0.core01.atl01.atlas.cogentco.com (154.54.3.197) 15.862 ms 15.961 ms 15.481 ms
8 p1-0.core01.jfk02.atlas.cogentco.com (154.54.5.97) 99.813 ms 109.906 ms 42.591 ms
9 p15-0.core02.jfk02.atlas.cogentco.com (66.28.4.14) 33.347 ms 33.084 ms 33.208 ms
10 p4-0.core02.lon01.atlas.cogentco.com (66.28.4.254) 103.310 ms 103.784 ms 103.789 ms
11 t2-4.mpd02.lon01.atlas.cogentco.com (130.117.2.22) 106.310 ms 106.158 ms 106.170 ms
12 te1-3.cr05.tn5.bb.pipex.net (195.66.224.29) 106.687 ms 106.756 ms 106.686 ms
13 ae1-0.cr02.gs1.dsl.pipex.net (62.72.137.14) 106.525 ms 106.522 ms 106.679 ms
14 g4-0.ar20.gs1.dsl.pipex.net (84.12.224.30) 105.560 ms 104.658 ms 104.303 ms
15 84.12.34.241 (84.12.34.241) 574.434 ms 558.906 ms 388.389 ms
16 *