Subject: Fwd: The case of the 500-mile email.

From: Muli Ben-Yehuda <[email-masked]>
Date: Tue, 26 Nov 2002 07:32:30 +0200

Apologies for fwding from another list, but this is entertaining
enough and the list is not widely known. 

Also, check out
http://www.livejournal.com/talkread.bml?journal=jwz&itemid=94645#cutid1

For a cool engineering fit. Makes those pesky bugs seem
... incosequential. 

----- Forwarded message from Nev Dull <[email-masked]> -----

Date: Mon, 25 Nov 2002 15:05:00 -0500 (EST)
From: Nev Dull <[email-masked]>
To: [email-masked] (/dev/null)
Subject: The case of the 500-mile email.

From: Trey Harris <[email-masked]>
Forwarded-by: Kirk McKusick <[email-masked]>

Here's a problem that *sounded* impossible...  I almost regret posting
the story to a wide audience, because it makes a great tale over drinks
at a conference. :-)  The story is slightly altered in order to protect
the guilty, elide over irrelevant and boring details, and generally make
the whole thing more entertaining.

I was working in a job running the campus email system some years ago
when I got a call from the chairman of the statistics department.

"We're having a problem sending email out of the department."

"What's the problem?" I asked.

"We can't send mail more than 500 miles," the chairman explained.

I choked on my latte.  "Come again?"

"We can't send mail farther than 500 miles from here," he repeated.  "A
little bit more, actually.  Call it 520 miles.  But no farther."

"Um... Email really doesn't work that way, generally," I said, trying
to keep panic out of my voice.  One doesn't display panic when speaking
to a department chairman, even of a relatively impoverished department
like statistics.  "What makes you think you can't send mail more than
500 miles?"

"It's not what I *think*," the chairman replied testily.  "You see, when
we first noticed this happening, a few days ago--"

"You waited a few DAYS?" I interrupted, a tremor tinging my voice.  "And
you couldn't send email this whole time?"

"We could send email.  Just not more than--"

"--500 miles, yes," I finished for him, "I got that.  But why didn't
you call earlier?"

"Well, we hadn't collected enough data to be sure of what was going on
until just now."  Right.  This is the chairman of *statistics*. "Anyway,
I asked one of the geostatisticians to look into it--"

"Geostatisticians..."

"--yes, and she's produced a map showing the radius within which we can
send email to be slightly more than 500 miles.  There are a number of
destinations within that radius that we can't reach, either, or reach
sporadically, but we can never email farther than this radius."

"I see," I said, and put my head in my hands.  "When did this start?
A few days ago, you said, but did anything change in your systems at
that time?"

"Well, the consultant came in and patched our server and rebooted it.
But I called him, and he said he didn't touch the mail system."

"Okay, let me take a look, and I'll call you back," I said, scarcely
believing that I was playing along.  It wasn't April Fool's Day.  I
tried to remember if someone owed me a practical joke.

I logged into their department's server, and sent a few test mails.
This was in the Research Triangle of North Carolina, and a test mail to
my own account was delivered without a hitch.  Ditto for one sent to
Richmond, and Atlanta, and Washington.  Another to Princeton (400 miles)
worked.

But then I tried to send an email to Memphis (600 miles).  It failed.
Boston, failed.  Detroit, failed.  I got out my address book and started
trying to narrow this down.  New York (420 miles) worked, but Providence
(580 miles) failed.

I was beginning to wonder if I had lost my sanity.  I tried emailing a
friend who lived in North Carolina, but whose ISP was in Seattle.
Thankfully, it failed.  If the problem had had to do with the geography
of the human recipient and not his mail server, I think I would have
broken down in tears.

Having established that -- unbelievably -- the problem as reported was
true, and repeatable, I took a look at the sendmail.cf file.  It looked
fairly normal.  In fact, it looked familiar.

I diffed it against the sendmail.cf in my home directory.  It hadn't been
altered -- it was a sendmail.cf I had written.  And I was fairly certain
I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option.  At a loss, I
telnetted into the SMTP port.  The server happily responded with a SunOS
sendmail banner.

Wait a minute... a SunOS sendmail banner?  At the time, Sun was still
shipping Sendmail 5 with its operating system, even though Sendmail 8 was
fairly mature.  Being a good system administrator, I had standardized on
Sendmail 8.  And also being a good system administrator, I had written a
sendmail.cf that used the nice long self-documenting option and variable
names available in Sendmail 8 rather than the cryptic punctuation-mark
codes that had been used in Sendmail 5.

The pieces fell into place, all at once, and I again choked on the dregs
of my now-cold latte.  When the consultant had "patched the server," he
had apparently upgraded the version of SunOS, and in so doing
*downgraded* Sendmail.  The upgrade helpfully left the sendmail.cf
alone, even though it was now the wrong version.

It so happens that Sendmail 5 -- at least, the version that Sun shipped,
which had some tweaks -- could deal with the Sendmail 8 sendmail.cf, as
most of the rules had at that point remained unaltered.  But the new
long configuration options -- those it saw as junk, and skipped.  And
the sendmail binary had no defaults compiled in for most of these, so,
finding no suitable settings in the sendmail.cf file, they were set to
zero.

One of the settings that was set to zero was the timeout to connect to
the remote SMTP server.  Some experimentation established that on this
particular machine with its typical load, a zero timeout would abort a
connect call in slightly over three milliseconds.

An odd feature of our campus network at the time was that it was 100%
switched.  An outgoing packet wouldn't incur a router delay until hitting
the POP and reaching a router on the far side.  So time to connect to a
lightly-loaded remote host on a nearby network would actually largely be
governed by the speed of light distance to the destination rather than by
incidental router delays.

Feeling slightly giddy, I typed into my shell:

$ units
1311 units, 63 prefixes

You have: 3 millilightseconds
You want: miles
        * 558.84719
        / 0.0017893979

"500 miles, or a little bit more."

Trey Harris
-- 
I'm looking for work.  If you need a SAGE Level IV with 10 years Perl,
tool development, training, and architecture experience, please email
me at [email-masked].  I'm willing to relocate for the right opportunity.

----- End forwarded message -----

-- 
Muli Ben-Yehuda				    http://www.mulix.org/
[email-masked]:~$ sctrace strace /bin/foo  http://syscalltrack.sf.net/
Quis custodes ipsos custodiet?

---

From: Adi Stav <[email-masked]>
Date: Tue, 26 Nov 2002 15:07:34 +0200

> enough and the list is not widely known. 

Cool one :) 

Reminds me of the "phase of the moon" bug, although that old one always
sounded rather fake to me.

Or the "kernel reboot" bug from about a year ago on LKML: someone found
out that if you just execute the ungzipped Linux kernel after compiling
it, it actually reboots the machine into Linux.  The person's main 
complaint was that it rebooted into the OLD kernel and not the newly-
compiled one, and he also expressed surprise over the fact that it even 
worked when running the kernel under an unpriveleged UID.  The kernel 
hackers, who certaintly did NOT remember implementing such a feature or 
anything like it, wanted to check out this bug.  The person who reported 
it warned them that the system reboots without killing the processes or 
unmounting the filesystems, but apparently the kernel hackers were so 
sure that "it can't be" that you could see maybe ten or fiften messages
from the biggest names on LKML in succession saying things like "damn it, 
I should have known better", "I'll believe bug reports next time...", and 
"I just HAD to check, hadn't I?" which provided endless amusement for the
week.

In the end they found out it that it was merely a bug with the ELF binary
loader which GOTOed into garbage in kernel mode. That was I it worked for
non-root too.

(Heh, a fitting random sig.)

--
"Kernel programming - if it's predictable or reproducible, it's boring" 
-- Muli Ben-Yehuda

---

From: Oleg Goldshmidt <[email-masked]>
Date: 26 Nov 2002 10:03:21 +0200

> http://www.livejournal.com/talkread.bml?journal=jwz&itemid=94645#cutid1

Awesome!!! 

> "We can't send mail more than 500 miles," the chairman explained.

This is cool, too, but I have a silly design question, and a serious
physical comment.

> connect call in slightly over three milliseconds.

Here is a design question. Does it make any sense to allow such short
timeouts at all. What would that be useful for? On the other hand,
would it be useful to protect against malice, stupidity, or
negligence, and have a (configurable? at compile time?) minimal and default
timeouts? the actual timeout would then be set to the default whenever
there is something wrong with the configuration file(s), including too
short (less than minimal) timeout. 

What are the design considerations in this situation? No, I have not
thought about it at all, yet...

> You have: 3 millilightseconds

Here is the physical comment. Something is wrong here. The calculation
gives 500 miles in 3 ms indeed for the speed of light of 300000 km/s,
which we all know to be the speed of light *in vacuum*. The speed of
light in the wire is (significantly) less. In fact, I recall a paper

http://xxx.lanl.gov/abs/physics/0201053

that dealt with measuring the speed of light in ethernet wires using
ping. The measurements presented in that paper indicate, IIRC, that
the speed of light in cat-5 cable is about 200000 km/s, i.e. 2/3 of
the vacuum value.

-- 
Oleg Goldshmidt | [email-masked] 
========================================================
First binary search algorithm         - J. Mauchly, 1946        
First correct binary search algorithm - D.H.Lehmer, 1960

---

From: Oleg Goldshmidt <[email-masked]>
Date: 26 Nov 2002 10:16:27 +0200

> incidental router delays.

Something is wrong here as well. This is true only of there is one hop
from origin to destination, isn't it? Their campus network might be
100% switched (I am willing to believe, for the sake of the argument,
that this means the delay inside the campus is negligible compared to
3 ms), but the moment a packet gets out into the big bad world it will
fall prey to router queues, for which 3 ms is not so long.

In addition, I need to check, but maybe someone knows it cold: isn't
timeout referring to round-trip time rather than one-way? I mean, the
connection drops if an ACK is not received in timeout seconds.

Cool story, but the numbers don't work out for me (yet?).

---

From: Shlomi Fish <[email-masked]>
Date: Tue, 26 Nov 2002 12:59:43 +0200 (IST)

On 26 Nov 2002, Oleg Goldshmidt wrote:

> > the remote SMTP server.Some experimentation established that on this
> > switched.An outgoing packet wouldn't incur a router delay until hitting
> > the POP and reaching a router on the far side.So time to connect to a

What if everything that connected these networks were fiber optics
Point-to-Point lines? Maybe they have a close to vacuum light-speed.

BTW, from what I remember from my studies, the speed of light within
certain materials can exceed the speed of light within vacuum, for some
particular frequencies. Still passing information from one point to another
can only be done at the speed of light.

Regards,

	Shlomi Fish

> To unsubscribe from this group, send an email to:
> [email-masked]
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>

----------------------------------------------------------------------
Shlomi Fish        [email-masked]
Home Page:         http://t2.technion.ac.il/~shlomif/
Home E-mail:       [email-masked]

"Let's suppose you have a table with 2^n cups..."
"Wait a second - is n a natural number?"

---

From: Oleg Goldshmidt <[email-masked]>
Date: 26 Nov 2002 13:21:52 +0200

> Point-to-Point lines? Maybe they have a close to vacuum light-speed.

There are two factors (at least) that affect the speed of
propagation. One is the material filling the fiber, and the other is
geometry. Just the second one would reduce the group velocity. The
material in fiber is some sort of glass. I don't know what its
refraction index is (anyone?). The refraction index of the ordinary
glass is about 4/3, meaning that the speed of light in the glass is
3c/4.

> particular frequencies. 

This can be the case for phase velocity, not group velocity, which is
the one we are discussing here (to make the long story short, it is
the one that determines how fast information can be transmitted by
light).

> Still passing information from one point to another

Yep, and that's what matters.