Subject: RE: [hackers-il] Summary of a Bash Lecture

From: Chen Shapira <[email-masked]>
Date: Wed, 1 Nov 2000 19:12:47 +0200

> The moral of the story is: use modprobe!

Or in Generalized form:

Never implement yourself using shell what is already implemented with C.

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 1 Nov 2000 20:33:37 +0200

> Never implement yourself using shell what is already implemented with C.

This is a tautology, and a biased one at that (if you remove the words "shell"
and "C", the sentence will still be true, and all you did was added a bias
against the shell) ;)

But to return to the issue - 'eval': watch out for this command. It is nice
for interactive use, but dangerous (security-wise) for shell-scripts.
For example, consider the following example: You do
	A=3
	B=A
and then you want to do something like $$B to get the value 3. Unfortunately,
in neither bash nor zsh none of the "obvious" methods work: $$B, ${$B}, ${${B}}.
(I can explain why these do not work - it's not a bug - but let's forget it now)
So, you decide to do
	eval echo \$$B

Now, what happens if someone manages to set B to "zzz hello >/etc/passwd"
to "; rm -rf /", or even "(rm -rf /)"? You've guessed it :)
[if you try the above examples, please replace the rm -rf / with something
less harmful! the last example also shows another problem with the $(...)
construction]

So watch out :)

P.S. You can do the above example in zsh as
	Z=\$$B; echo ${(e)Z}
but this still has the problem with evaluating also the dangerous $(...).

-- 
Nadav Har'El                        |    Wednesday, Nov 1 2000, 4 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Linux is just like a wigwam: no Windows,
http://nadav.harel.org.il           |no Gates and an Apache inside.

---

From: Moshe Zadka <[email-masked]>
Date: Wed, 1 Nov 2000 20:38:58 +0200 (IST)

> Never implement yourself using shell what is already implemented with C.

Though one should not forget the companion lesson: never implement in C,
what can be easily implemented with shell. 

the big paradox!
--
Moshe Zadka <[email-masked]> -- 95855124
http://advogato.org/person/moshez

---

From: mulix <[email-masked]>
Date: Wed, 1 Nov 2000 21:04:40 +0200 (IST)

both of these can be abstracted as
"use the right tool for the task at hand"
or its negation
"to a man with only a hammer, everything looks like a nail"

which does bring up an interesting questions: assume all tools being
equally suited to the task at end, in that each tool has unique
advantages and disadvantages. how do you choose?

--
mulix

linux/reboot.h: #define LINUX_REBOOT_MAGIC1 0xfee1dead

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 1 Nov 2000 21:53:07 +0200

> advantages and disadvantages. how do you choose?

Obviously, in some cases it is not clear if it is best to use C, shell,
perl, awk, or whatever. But in many cases one language has a definate
advantage over another. Only if you know all the languages (or a good
selection of them), you'll be able to choose the appropriate one for each
task.

For example, if you want to write a short (say <100 line) program, which
most of its work is to call other external programs and handle string
constants, why use C which is horribly complex with handling strings (and
needing to allocate eveything, etc.)?

Experience is also a factor. For example, I have about 14 years of
experience with Awk, but only about 3 years with Perl, so when I want to
write a small (say, 50 lines) string-handling script I instinctively
use Awk, only turning to Perl when I need something bigger or one of Perl's
extra features (e.g., socket programming, HTTP connection, etc.).

The funniest example I've seen is where someone I worked with wanted to
write a program which interpolates "include" commands into Fortran source
code (kind of like the C preprocessor's #include). Since Fortran is the
only language he knew well (and I'm talking about Fortran 77, not the
newer C-wannabe-Fortran-90) he wrote this program in Fortran. What could
have been a ~50 line awk script (for example) turned out a hillarious
Fortran program with several hundreds of lines and limitations that only
Fortran addicts will understand (e.g., source lines were not copied as is,
but instead filled with spaces to be 72 character wide, or truncated after
the 72nd character - this is because Fortran doesn't have variable-length
strings (!)).

-- 
Nadav Har'El                        |    Wednesday, Nov 1 2000, 4 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |When everything's coming your way,
http://nadav.harel.org.il           |you're in the wrong lane.

---

From: Moshe Zadka <[email-masked]>
Date: Thu, 2 Nov 2000 00:28:50 +0200 (IST)

> less harmful! the last example also shows another problem with the $(...)

Actually, rm -rf / is not that harmful: as long as anyone remembers
never to try out shell code he's not sure of as root, there's plenty
of time to CTRL-C

---

From: Moshe Zadka <[email-masked]>
Date: Thu, 2 Nov 2000 09:16:37 +0200 (IST)

> advantages and disadvantages. how do you choose?

Ummmmmm.....weigh which advantages and disadvantages are most crucial,
based on my previous experiences?

Which is another way of saying that there is no such situations: in every
situation there is one tool most suited. Which tool it is depends on many
factors, and is the nexus of many flamewars (Python/Perl/Tcl, Emacs/vi, 
Linux/FreeBSD, Lisp/Scheme and so on).

---

From: Moshe Zadka <[email-masked]>
Date: Thu, 2 Nov 2000 09:35:51 +0200 (IST)

> the 72nd character - this is because Fortran doesn't have variable-length

That's because real engineers write in Fortran.
Including string and list processing.
--
Moshe Zadka <[email-masked]> -- 95855124
http://advogato.org/person/moshez

---

From: Oleg Goldshmidt <[email-masked]>
Date: 07 Nov 2000 20:41:59 +0200

> Including string and list processing.

At the risk of over-citing:

I guess I am a real engineer, since I have written (let alone read)
tons of Fortran code (yes, Fortran 77, though my first was Fortran
IV). Some comments without debugging or testing (I *am* a real
engineer, after all):

1) [At least some] string processing is much more convenient in
   Fortran than in other languages, such as C. As a trivial example,
   consider string concatenation.

2) Fortran has an INCLUDE statement of its own. It is not clear to me
   what the purpose of that project was.

3) I don't think that anything prevents you from running cpp on a
   Fortran source. As long as the result of pre-processing is legal 
   Fortran that your compiler will handle, it's OK.

4) At Bloomberg/BFM we extensively use artificial "preprocessor"
   constructs such as

C#IFDEF MACRO
	....... some Fortran code, e.g. debugging, profiling, etc
C#ENDIF MACRO

   and use a set of awk/sed scripts to comment/uncomment/strip/add
   the conditional code as needed. 

5) The 72-character lines could come about because legal Fortran code
   cannot extend beyond column 72 (backwards compatibility), and not
   for other nefarious reasons.

-- 
Oleg Goldshmidt <[email-masked]> 
"... We work by wit, and not by witchcraft;
 And wit depends on dilatory time." [Shakespeare]

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 8 Nov 2000 10:28:12 +0200

> IV). Some comments without debugging or testing (I *am* a real

"my first fortran" was Fortran 77. I've also programmed in Fortran quite
extensively, but I must admit I don't remember every detail now so maybe
I was (and am) wrong on a point or two, so feel free to correct me.

>    consider string concatenation.

Builtin features may be easy to use. But when you want to write a function
that takes a string as an argument, it becomes very hard. Strings have fixed
lengths, and you don't have a notion of how what length you did fill in the
string - so you choose a maximum string length you'll support (e.g., 100)
and every string which is shorter you pad by spaces...

>    what the purpose of that project was.

True, but this guy wanted to have a more suphisticated search for these include
files (to look them up in some sort of table, the exact details aren't really
important for the point I was trying to make...). Let's assume that the guy
did have to write the program - the question was only about the choice of
language to write it in.

>    Fortran that your compiler will handle, it's OK.

You can do that (you'll probably need to filter out the '#line' lines that CPP
adds), but as I said the point wasn't just adding *any* include statement -
he wanted it to make lookups into some sort of special table, and he didn't
want to use CPP (I'm not sure he was even aware of CPP at the time, but that's
besides the point).

>    the conditional code as needed. 

See? awk/sed is a good language to do these things. You didn't write these
scripts in Fortran, did you? :)

>    for other nefarious reasons.

This is not true. (Most) Fortran compilers do ignore any character beyond
the 72nd column, but it doesn't mean you can go and cut your source code
at the 72nd column: this also cuts long comments (this was a real problem for
users of this script). And besides, I didn't say it just _cut_ lines at
the 72nd column - lines that were originally shorter than 72 columns were now
padded with spaces to be 72 characters, and after this transformation source
files became much bigger!

As I said, the reason why this happened is not some deliberate choice, but just
a result of the fact that Fortran 77 cannot easily deal with strings of
variable length, so he choise the most "sensible" fixed length: 72. I'm sure
that with enough effort you can "invent" variable length string processing in
Fortran, just as I've seen people add something like dynamic memory allocation
to it (Fortran 77 does not have dynamic memory allocation!!), or like you can
try to add garbage collection to C.  But Fortran is certainly not a natural
language to tackle this sort of problem.

-- 
Nadav Har'El                        |   Wednesday, Nov 8 2000, 10 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Linux is just like a wigwam: no Windows,
http://nadav.harel.org.il           |no Gates and an Apache inside.

---

From: Oleg Goldshmidt <[email-masked]>
Date: 08 Nov 2000 23:49:06 +0200

> that takes a string as an argument, it becomes very hard. 

I am not sure. Maybe I don't get the picture that you have in mind,
but things like [UNTESTED!]

	subroutine foo(s)
	character*(*) s
	s = 'Nadav'
	return
	end

usually work (the string will even be null-terminated, so you can call
this beast from C, I think). Not that I like it, mind you. To tell you
the truth, I am not 100% sure that the above is 100% portable, and I
am not inclined to research this right now. I am not trying to be a
devil's - oops, Fortran's - advocate here.

> Let's assume that the guy did have to write the program - the
> question was only about the choice of language to write it in.
 
Oh, yes, I quite agree - and did so from the beginning. It really
wasn't clear to me what the guy was doing.

> scripts in Fortran, did you? :)

No, no :) I mentioned that because I did agree from the start that it
was an awk/sed task. Actually, that was in support of your statement
that a limited pre-processing capability was a relatively simple task
for awk/sed.
 
> This is not true. 

I believe it is. AFAIK, the Fortran standard defines a line of code as
a sequence of exactly 72 characters. It is true that (modern)
compilers often ignore that, in particular allowing that comments
spill beyond position 72 (this is possible because the compiler can
essentially ignore the tail of a comment, I guess). However, the
standard maintains 72. Try

http://www.fortran.com/fortran/F77_std/rjcnf0001-sh-3.html#sh-3.2

for reference. I have not re-read it carefully though, so maybe I am
missing something. And I wouldn't want to appear a language lawyer,
just pointing out that maybe your friend did know Fortran very well ;-)

> But Fortran is certainly not a natural language to tackle this sort
> of problem.

Absolutely.

---

From: "Nadav Har'El" <[email-masked]>
Date: Thu, 9 Nov 2000 11:47:21 +0200

> just pointing out that maybe your friend did know Fortran very well ;-)

This discussion is becoming a little silly (it appears that everything on
hackers-il is silly these days - maybe we should offer a Fortran varient
of the Hello World program...)...

Anyway, for people who don't know, the reason for the 72 character limit is
this (as far as I know): punch cards worked the following way - each card was
one line, with 80 characters (each column coded one character, in 6 or 7
bits - I don't remember). So what happens if you had a 10,000 line program,
i.e., 10,000 cards, in a big case, and it fell on the floor? How can you know
the order of the lines? Well, the characters 73..80 on the card are ignored,
and serve as a comment, which you can use as a line number, or whatever.
So it is not quite true to say that Fortran disallows more than 72 characters -
what the standard should say (and I admit I didn't read it), or imply, that
only the first 72 characters have a meaning to the compiler - any characters
after that do not have a meaning under the standard and should be ignored.
Every Fortran 77 compiler I know (and, unfortunately, I used f77 on Alpha,
IBM, Sun, and Linux) indeed ignores characters beyond column 72 (which is a
PITA to debug, if you accidentally write a line too long and part of it is
ignored). Besides, let me reiterate that lines (although not cards) can be
_shorter_ than 72 characters - Fortran does not force you to pad them with
spaces to exactly 72 characters (of course on punch-cards you did pad them
with spaces, because you couldn't have half a card...).

For the curious, the Fortran cards had other special columns except the
73..80 I mentioned. Column 1 was a flag whether this is a comment line: "C"
or "*" meant this is a comment, while a space meant that it is a regular
line. Alternatively, columns 1..5 may contain a numeric "label", that will
be possible to "goto" to (like in Basic). Column 6 was used to mark a
continuation line (instead of Unix's convention of ending the previous line
with a backslash).

IBM's punch cards had special markings on them to remind you of these special
columns, and that you should only use columns 7..72 for the actual code.

All these anachronistic conventions (and a lot of other issues) make
programming in Fortran 77 hell. I wouldn't recommend it :)

-- 
Nadav Har'El                        |    Thursday, Nov 9 2000, 11 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |When everything's coming your way,
http://nadav.harel.org.il           |you're in the wrong lane.

---

From: Moshe Zadka <[email-masked]>
Date: Thu, 9 Nov 2000 12:01:34 +0200 (IST)

> of the Hello World program...)...

No: use libhello instead -- just write a Fortran binding.

> only the first 72 characters have a meaning to the compiler - any
> characters after that do not have a meaning under the standard and
> should be ignored.

I think this is the correct reading. What fun it is when you run
your program through expandtabs or some such -- suddenly a tab, instead
of one character, is transformed into 8, soemtimes pushing lines
over the 72 chars limit. Debug that!
(Especially if your editor displays the tabs, and the untabbification
is an automatic pre-processing stage before the compiler, and you
use 4-place tabs in the editor)

> Besides, let me reiterate that lines (although not cards) can be
> _shorter_ than 72 characters - Fortran does not force you to pad them
> with spaces to exactly 72 characters

Depends on the Fortran compiler, sometimes. I did work with an interpreter
for a Fortran-like language that did have this limitation. Consultation
on facilitating development under these conditions with some vi
trickery is available from me for 200$/hour (which I'll probably
pay my psychiatrist for solving the mental anguish) :-D

> line.

Not accurate: anything other then a space meant comment. C/* is a common
Fortran convention, though.

> programming in Fortran 77 hell. I wouldn't recommend it :)

OTOH, no free compilers for F-90 exist, and C compilers do a much
worse job of optimizing numerical code, especially on super-computers.

Solution: write the numerical code in F-77, and the main loop (which
isn't computation intensive anyway) in Python. Great tools for
hacking F-77 and Python together exist!

---

From: "Nadav Har'El" <[email-masked]>
Date: Thu, 9 Nov 2000 12:18:02 +0200

> over the 72 chars limit. Debug that!

Sun's Fortran compiler actually had an "extended" mode (they should call
it "sane" mode) if the line contained a tab: in this case the line was
allowed to contain as many characters as you wish, and the column numbers
did not have any special meaning. Of course, this was completely non-standard
and could not be used in any program that wanted to be portable.
Of course, Fortran 90 now has a similar mode, but as you mentioned, no free
compiler for it exists. Worse: I know of a project that got stuck for many
months because they couldn't get a Fortran 90 compiler for their choice of
Unix brand. After several months they had to change their choice of Unix
brand (!), to one on which they could get a (commercial) F90 compiler.

> worse job of optimizing numerical code, especially on super-computers.

Many people say that, but I never saw any proof to back it up... It it true
that C programs lack much of the local structure that let's Fortran be
optimized more easily (e.g., Fortran 'FOR' loops compared to C 'for' loops)
but it doesn't mean that C cannot be optimized... I never saw any good
study comparing _actual_ optimization performance.

> hacking F-77 and Python together exist!

This I've got to see :)

-- 
Nadav Har'El                        |    Thursday, Nov 9 2000, 11 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |If God is watching us, the least we can
http://nadav.harel.org.il           |do is be entertaining.

---

From: Moshe Zadka <[email-masked]>
Date: Thu, 9 Nov 2000 13:09:21 +0200 (IST)

> > hacking F-77 and Python together exist!

Look around http://numpy.sourceforge.net

---

From: Oleg Goldshmidt <[email-masked]>
Date: 14 Nov 2000 22:46:04 +0200

> Fortran convention, though.

Sorry, Moshe, it's [Cc*] only. In particular, a numeric label may
start in the first column. 

> > programming in Fortran 77 hell. I wouldn't recommend it :)

hell, I survived it! ;-)

> worse job of optimizing numerical code, especially on

I don't know how true that is. Another point is that an awful lot of code,
including numerical libraries, has been written and hand-optimized
in Fortran. However, Nadav's point that Fortran is not very good for
text-processing, system programming, artificial intelligence, or GUI
(I happen to know one BIG GUI thingy written in Fortran, though) is
well taken.

---

From: Oleg Goldshmidt <[email-masked]>
Date: 14 Nov 2000 22:51:08 +0200

"Nadav Har'El" <[email-masked]> writes:

> optimized more easily (e.g., Fortran 'FOR' loops compared to C 'for'
                                        ^^^

Surely you meant DO loops.

> study comparing _actual_ optimization performance.

That probably depends on your choice of compilers a lot. It's not very
meaningful to compare performance of languages. I suspect there are
cases where f2c/gcc will give you better performance than some Fortran
compilers, for the same Fortran code. However, other Fortran compilers
might beat f2c/gcc on the same problem.

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 15 Nov 2000 00:33:33 +0200

> Surely you meant DO loops.

Oops... Sure, "DO" :) Several months since I last touched Fortran, and my C
indoctrination is beginning to show :)

Reminds me of a Fortran horror-story somebody once told me (I don't know if
it's true, but it makes a good urban legend; also, forgive me in advance
if I "C"-ate more Fortran constructs):

Some company (abroad) was building missiles, and one day a test failed
miserably, due to error in the missile's guiding software. When the software,
written in Fortran, was debugged, the following embaracing error was found:
a programmer meant to write something like
      DO 100 I=1,30
		... do something
100   CONTINUE

which means to repeat something 30 times (the CONTINUE statement does
absolutely nothing, and is just there so that you can put a label (100) on it,
and this label is used to signify the end of the loop).
Anyway, the programmer accidentally wrote the following
      DO 100 I=1.30
		... do something
100   CONTINUE

Can you spot the difference? The comma was changed into a period. Somebody
not familiar with Fortran may be puzzled on how this can compile at all, or
what it does. Well, it compiles ok, and what it does is to run the
"...do something" only once. Can you guess why? If not, scroll down to find
the answer ;)

v

v

v

Ok. Here's the answer: the first line says
      DO 100 I=1.30
Fortran pays absolutely no attention to spaces, so that this line is completely
equivalent to
      DO100I=1.30
Now, because this is not a valid DO line (there is no comma to seperate the
start and end integer), then Fortran cannot parse it this way, and must parse
it as a variable assignment: 1.3 is assigned into the variable "DO100I"!
Now because Fortran (by default) doesn't require variables to be declared
this is a perfectly legal assignment! This assignment obviously has no effect
on the program, as this "DO100I" variable is never used again.
Now as I said the last line,
100   CONTINUE
actually does nothing, and unless something refers to it is just ignored
(needless to say, you're not warned of this fact in compilers I've seen).

So a typo in one character caused the program to do something completely
different!

These Fortran conventions, the insignificance of whitespace and the not
needing declarations, causes a phenomenon which is all-too-familar to
Fortran users, but is almost never found in other languages: the "typo
bug". Fortran users are used to the fact that a typo will cause a hard-to-
find bug, rather than a compilation error!

Another example of a typo bug that happened to me: consider the following
lines:
      COMMON /XYZ/ A, B, C
     1             D
     
(the line with 1 in the 6th column column is a continuation line). This
was _meant_ to declare 4 variables: A, B, C, and D, as "common" (the
Fortran name for global variables). However, I forgot another comma after the
"C"! so to Fortran, this line is equivalent to the command (remember that
whitespace are ignored):
      COMMON /XYZ/ A, B, CD
so that neither C nor D are declared global! Imagine my suprise when the
values I put into C or D didn't seem to be passed correctly between function
calls, or how hard it was to find this bug! A similar bug is in the following
statement:
      CALL SOMEPROC(param1,param2
     1             param3,param4)
You think you're calling SOMEPROC with 4 parameters, but Fortran only sends
3, with "param2param3", a variable that never existed, getting a value of 0
(or a random value). Needless to say, you're never warned that anything is
strange. It took me once several hours to find a bug of this nature...

-- 
Nadav Har'El                        |  Wednesday, Nov 15 2000, 17 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Computers are like air conditioners.
http://nadav.harel.org.il           |Both stop working if you open windows.

---

From: Moshe Zadka <[email-masked]>
Date: Wed, 15 Nov 2000 08:25:18 +0200 (IST)

> might beat f2c/gcc on the same problem.

Also note that it's quite possible many more people hacked on C
optimizations, while few hacked on Fortran specific optimization.

---

From: Moshe Zadka <[email-masked]>
Date: Wed, 15 Nov 2000 09:00:43 +0200 (IST)

> if I "C"-ate more Fortran constructs):

The urban legend (I don't know if it's true either) was that it caused
a space probe (unmanned) to crash.
At least, that's the way Tim Peters tells it.

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 15 Nov 2000 11:08:28 +0200

>...
>...

By the way, if someone is skeptical that Fortran really accepts this statement
as I explained (I know I was skeptical at first!), try the putting the
following in "z.f" and compiling it with f77:

      PROGRAM HELLO
      DO 100 I=1,10
      PRINT*, 'hello'
100   CONTINUE
      END

(make sure you leave the first 6 columns as is - getting the spaces there wrong
will prevent Fortran from compiling).
See how you see "hello" 10 times? Now change the comma in the DO line into
a period. You'll see only one hello :)

-- 
Nadav Har'El                        |  Wednesday, Nov 15 2000, 17 Heshvan 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Christopher Robin Hood steals from the
http://nadav.harel.org.il           |rich and gives to the Pooh.