[Nolug] ok, let's try forwarding the right email this time...

Ugh, I sent the wrong email. Here is the correct one, about going around web
proxies.

--Joey

---------- Forwarded Message ----------
Subject: OpenNIC Discuss: Working around proxies -- the smelly underside of
port 80
Date: Tue, 26 Mar 2002 00:26:56 -0800 (PST)
From: user@host
To: discuss@opennic.unrated.net

Having slashdotted OpenNIC (sorry, Robin) I got some input that
worked! Recall, I'm having trouble getting to OpenNIC named sites
over my home network because port 80 is intercepted by a ``transparent
proxy'' that my ISP operates. The proxy fails to be transparent when
I browse the web, because it tries to re-resolve the hostname in the
URLs I browse. That means that whenever I try to connect to a
non-ICANN web server, my browser displays only an error message from
the proxy.

Many others are behind proxies like mine, and most people can't get
rid of them without switching ISPs. [After slashdotting my ISP with
an article about how they were stonewalling me, I'm not likely to get
their proxy policy changed either...]

Here's a little mini-report based on the replies I got, some RFC's I
read over the last couple of days, etc. The short answer is: you can
work around most proxies pretty simply, and I'm writing a little tool
to do it. The simplest approach is probably a local filtering proxy.

There are many nuances to the HTTP interchange, which aren't really
apropos here, but the way you're supposed to specify a GET request is:
  GET <URI> HTTP/<NUM>
  <other tags>

or something similar. The request is terminated by a double-newline.
Now, that "URI" isn't a typo -- it's an identifier and it's allowed
(and encouraged by the RFC) to be relative, e.g.
   GET / HTTP/1.1
   Host: www.dev.null

The problem is that broke a whole bunch of early caching proxies caches
that used only the HTTP 1.0 standard where you had to tell the server
   GET <URL> HTTP/1.0
(that's "URL" there, as in full "http://host.name.domain/whatever.html").
That's because none of the old proxies knew how to parse the keyword list,
or to recognize the "HTTP/1.1" string, so they'd barf on the URI.

To combat that problem, ALL browsers that I have tried emit only the
allowed-but-deprecated absolute URL form of the URI:
  GET http://www.dev.null HTTP/1.1
  Host: www.dev.null

That fixes all those legacy proxies. But because of that practice, people
who write new proxies appear to sort only on the first line (or so some
silverback IP geek said), more or less because of inertia. I tried sending
relative-URI requests out port 80, and my ISP's proxy treated them
correctly, forwarding them on to dev.null and sending the correct
Host: tag (so that I could get several different virtual hosts by picking
different Host strings).

In short, if you send
   GET <relative-URI> HTTP/1.1
   Host: virtual-host-name

instead of the more standard
   GET <full-URL> HTTP/1.1
   Host: host-name

then you might be able to get OpenNIC web pages even through a broken
transparent proxy.

The problem is that the web browsers I know about have the
absolute-URL emission hard coded into them. Sometime this week, I'll
be writing a quick-and-dirty bouncing web proxy (that I think will be
about 10 lines of perl) to munge ordinary URLs into relative URI's. The
little proxy can run on the local host and munge all your outgoing
HTTP requests into proper relative form.

In the longer term, sensible web browsers probably ought to have
absolute - vs - relative HTTP requesting settable as an option.

Anyhow, that's kind of a long message but I'll announce the little proxy when
it's ready. It'll probably come in handy for many OpenNIC types.

Cheers,
user
######################################################################
This is the discussion list for the Open Network Information
Center. You can unsubscribe by sending an email containing the words
"unsubscribe discuss" in the body of the message to
"majordomo@opennic.glue" or "majordomo@opennic.unrated.net".
######################################################################

-------------------------------------------------------

-- 
Joey Kelly
< Minister of the Gospel | Computer Networking Consultant >
http://joeykelly.dhs.org
"When Government fears the people, it's liberty.
When people fear the Government, it's tyranny."
-- Benjamin Franklin
Ich möchte ein Berliner.
___________________
Nolug mailing list
nolug@nolug.org