Subject: Message 3 (Re: About Park - points to consider when designing a new programming language)

From: Omer Zak <[email-masked]>
Date: Fri, 23 Jun 2006 23:08:02 +0300

On Fri, 2006-06-23 at 21:06 +0300, Shlomi Fish wrote:
> Hi Omer!
> 
> I hope you don't mind me CCing this message to shlomif-beta-reviewers. They 
> may be interested in it. I don't see anything particularly personal in it.

Actually, Hackers-IL is a more suitable mailing list for this
discussion.  It is really about how a programming language could support
programming in the large.

> > ref: http://www.shlomifish.org/park-lisp-fooware/
> >
> > I believe that a new non-DSL can justify its existence only if it
> > addresses the problems of developing a giant application (O(10^7 lines
> > of source code)).  I raised the subject in Hackers-IL few years ago.
> 
> Yes, I know. Well, most codebases never reach this kind of volume. I know that 
> Amazon.com has a ~100 Million codebase (which is constantly evolving):
> 
> http://xrl.us/nowb
> 
> (and a lot of it is in Perl and in other high-level, succint languages)
> 
> But this is relatively rare. As MJD points out here:
> 
> http://perl.plover.com/yak/12views/samples/notes.html
> 
> (Search for "LISP") he writes about a dozen one-liners a day. So one thing I'd 
> like to make Park suitable for is writing short scripts and programs.

To do quick one-liners, I think that a properly designed IDE would be
better than a new language.  I have in mind an IDE which integrates with
a powerful search engine:
You type in concepts as keywords.
The search engine searches through library documentation and cookbooks.
You get, as results, links to relevant libraries (from all programming
languages) and relevant coding cookbook entries.

> My expectation from languages is that unless it's a truly estoric language 
> (like Intercal) it can scale well. Hell, even PHP which has some serious 
> maintainability problems (at least IMO) can scale well to large or very large 
> codebases.

A language needs to be both scalable and maintainable.  If PHP has
serious maintainability problems, then it is not suitable as main
language for O(10^7) projects.

> > - Support for mixed-language software development.
> 
> Do you mean mixed-programming language or mixed human-language (I.e: Unicode, 
> etc.)

I mean mixed-programming languages, as in Inline::Guile, which mixes
Scheme into Perl (when it works).

> > - Multi-versioning of support code (it would be cool to be able to link
> > one part of a giant application with version 1.5 of a library, and
> > another part of the same application with version 1.6.3 of the same
> > library).
> 
> That's interesting. I don't think Perl supports it, but Python might if I 
> understood its module system correctly. Of course, we also have to take into 
> consideration a Perl "Safe;" like protected environment module which is often 
> used for IRC eval bots, embedded Perl etc..
> 
> The Pythoneers tried implementing something like that in Python, but they kept 
> discovering security problems, and ended up abandoning by concluding it was a 
> difficult problem.
> 
> One possible solution would be to set a different root namespace or do a 
> namespace assignment. If each namespace (and sub-namespace) is a first-order 
> object (which was what I planned for Rindolf) then we can mix and match them.

This is more of distribution (in the sense of Gentoo, Ubuntu or
Mandrake) issue.  Different programs can link to different libraries,
and be glued together using a script file to form a single gigantic
application.

For example, consider what is needed to be able to keep both versions
1.4 and 1.9 of automake in a Debian system.

> > - Testability (probably supported by aspect orientation).
> 
> What do you mean by "Testability"? I'm waiting for an answer.

Let's keep it open-ended question here.  Additional brains are invited
to discuss implications of testability upon language and IDE design.

An hypothetical language, which does not allow you to run separately a
module from a big application, with reasonable amount of scaffolding -
does not support testability.

> > - Debugability (a language which can help detect heisenbugs can be a
> > winner).
> 
> Funny that you mention it. I recently encountered a heisenbug in "perl -d" and 
> one in the entire debugger interface. Now the default perl debugger is very 
> hairy and error-prone, and I could only reproduce this heisenbug 
> occasionally, and not too many others could reproduce it.
> 
> One suggestion for handling heisenbugs that has been raised is a reversible 
> debugger. See for example:
> 
> http://use.perl.org/article.pl?sid=06/06/22/1210201
> 
> Do you have any other suggestions?

Not really.  But can try to use theory (http://lambda-the-ultimate.org/)
to specify a language, in which it is possible to prove that certain
types of heisenbugs cannot occur.  For example, require explicit
declaration of variables, which are shared among threads, so that the
compiler (or interpreter) can check, at compile time, that accesses to
those variables are properly locked.

> > - Several views of the source code (LabVIEW is suited to certain
> > personality types, among whom I am not).  Syntax coloring.  Finding all
> > references to a concept [variable, procedure, object] without being
> > confused by having its name reused elsewhere; or having several names
> > aliased to it.
> >
> 
> That's pretty much impossible to do all the time in Lisp (if I understand you 
> correctly). While someone can implement a syntax coloring editor which will 
> work if your code does not use macros or fancy stuff like that too much. 

Can you design Park to support this, even when macros are used?
It may be a good idea to design an IDE, which would allow a developer to
have a view of code, which could be generated by a LISP macro, as the
macro is being developed.

I think that it is more of IDE issue than language issue, however.

                                          --- Omer

-- 
You haven't made an impact on the world before you caused a Debian
release to be named after Snufkin.
My own blog is at http://tddpirate.livejournal.com/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 24 Jun 2006 01:01:17 +0300

On Friday 23 June 2006 23:08, Omer Zak wrote:
> > I hope you don't mind me CCing this message to shlomif-beta-reviewers.
> > They may be interested in it. I don't see anything particularly personal
> > in it.
>

Yes. Replying to Hackers-IL.

> > Yes, I know. Well, most codebases never reach this kind of volume. I know
> > that Amazon.com has a ~100 Million codebase (which is constantly
> > evolving):
> >
> > http://xrl.us/nowb
> >
> > (Search for "LISP") he writes about a dozen one-liners a day. So one
> > thing I'd like to make Park suitable for is writing short scripts and
> > programs.
>

Well, I don't write my one-liners using an IDE. I write them in a Konsole 
(KDE's Terminal app) window with Bash. Sometimes I fire up gvim if the code 
becomes too hairy and doesn't do what I want, at which point it often becomes 
more than one line.

I sometimes also consult various man pages or perldoc's (perldoc is perl's 
online help system, similar to man). Usually I don't need Google or a 
different search engine.

I believe most one liners assume the programmer has a ready, instant, 
knowledge of most of the core functions and concepts he needs to write stuff. 
Like I told people a million times already, when I started seriously working 
on UNIX, I wrote most of my shell scripts in Perl. (And I know of several 
other programmers who got introduced to UNIX shell scripting thru Perl or 
Python or whatever.). It simply that I didn't had enough ready knowledge of 
the shells that were available on my host UNIXes to write it using it.

My intention in Park is to have an actually useful Lisp dialect that people 
could use in place of Perl, Python, etc.

> > My expectation from languages is that unless it's a truly estoric
> > language (like Intercal) it can scale well. Hell, even PHP which has some
> > serious maintainability problems (at least IMO) can scale well to large
> > or very large codebases.
>

Maybe maintainability is not the right word. PHP code if written properly is 
maintainable. (and if it isn't written properly, but still works, it can 
usually be refactored to be maintainable) What I meant is that in PHP it is 
easier to introduce various bugs if you're not careful. It's not impossible 
to write perfectly good code in PHP, just requires more discipline. 

( I don't mean to spread FUD on PHP here - this is just my opinion and my 
impression. I can prove it by various means.)

I believe PHP would be suitable for writing > n*10M LOCs codebases. But you'll 
probably need to have very good and discplined programmers for achieving 
this. (But this goes without saying for any other language you'd like to 
maintain such codebases in).

> > Do you mean mixed-programming language or mixed human-language (I.e:
> > Unicode, etc.)
>

OK, I see.

Well, if Park would be C-hosted (which I'd like to - I started writing a 
section about it - it's on: http://xrl.us/no6o ), then it can somehow have 
Inline::Perl5, Inline::Python, and Inline::Perl5::Inline::*. :-)

If Park would be implemented above Parrot (or a different portable VM 
like .NET), then it should be able to interface with other languages written 
for this virtual machine. 

There is also the concept of a Common Request Broker, Web Services, YAML, 
POE/Twisted/etc., SWIG, and plain ol' TCP/IP.

These are mostly an implementation detail.

Note that this is a very hard problem, and I do not claim to have a perfect 
(or even too good) solution for it.

> > understood its module system correctly. Of course, we also have to take
> > into consideration a Perl "Safe;" like protected environment module which
> > is often used for IRC eval bots, embedded Perl etc..
> >
> > The Pythoneers tried implementing something like that in Python, but they
> > kept discovering security problems, and ended up abandoning by concluding
> > it was a difficult problem.
> >
> > namespace assignment. If each namespace (and sub-namespace) is a
> > first-order object (which was what I planned for Rindolf) then we can mix
> > and match them.
>

I see.

I'd like to think about it a bit. But thanks for raising this problem.

> does not support testability.

I see. Well, the way I see it, without the Law of Demeter, and general 
modularity concerns, you can always design such a problematic program in any 
language no matter how suitable. I'm not aware of any such problems in Perl 5 
or Python, because they provide the means for easily mocking (or scaffolding  
as you call it) objects and modules. (Albeit from my impression some of the 
generic MockObject APIs are not worth the extra trouble of using).

I heard that .NET had a problem in creating a generic mock object, but I'd 
rather leave it to other people here. 

> > Funny that you mention it. I recently encountered a heisenbug in "perl
> > -d" and one in the entire debugger interface. Now the default perl
> > debugger is very hairy and error-prone, and I could only reproduce this
> > heisenbug occasionally, and not too many others could reproduce it.
> >
> > One suggestion for handling heisenbugs that has been raised is a
> > reversible debugger. See for example:
> >

Do you include static run-time checking there? (Like the Stanford checker, 
etc.)

I'd rather not go too much into the direction of a language in which bugs can 
be proven not to exist for fear of the following quote:

<<<
"UNIX was not designed to stop its users from doing stupid things, as that 
would also stop them from doing clever things." – Doug Gwyn

So I'm not sure whether such a feature would prevent doing smart things like 
doing self-writing code, or even generating functions on the fly, or 
dynamically changing the class inheritance graph.

> > That's pretty much impossible to do all the time in Lisp (if I understand
> > you correctly). While someone can implement a syntax coloring editor
> > which will work if your code does not use macros or fancy stuff like that
> > too much.
>

I also think Alan Turing has something to say about it too:

http://en.wikipedia.org/wiki/Halting_problem

If we generalise the Halting problem than we can say a program to conclusively 
analyse a different computer program would be impossible to write. I don't 
see how I can get an IDE to analyse a macro and determine how it does what 
it's supposed to do. The best way would probably be to document the behaviour 
of the macros in the program a la doxygen or javadoc. But this will require 
an extra work from the programmer.

Note that my intention is that the core Park language and support libraries 
will be rich and powerful so that people will often not have to resort to use 
macros to redefine the language. This means there would be some kind 
of "Perly Park" subset of Park where you write programs like you would in 
Perl.

<<<<<<
<adeht> rindolf: a nice way of writing CL code is to imagine your dream 
language for expressing that particular problem, and then realizing it :)
<rindolf> You probably wouldn't use a regex in Python either.
<rindolf> adeht: I see.
<rindolf> adeht: I'm not going to implement Perl in Common Lisp. :-D
	--- #lisp-il today.

BTW, tomorrow I intend to write my classic Graham's Function program ( 
http://www.shlomifish.org/Vipe/lecture/Perl/Graham-Function/ ) in Smalltalk 
and/or Common Lisp. I already have versions in Perl 5 (which is the 
original), in O'Caml and in Perl 6. Does anyone volunteer to write it in 
COBOL or Intercal? ;-)

Regards,

	Shlomi Fish

---------------------------------------------------------------------
Shlomi Fish      [email-masked]
Homepage:        http://www.shlomifish.org/

95% of the programmers consider 95% of the code they did not write, in the
bottom 5%.

---

From: Omer Zak <[email-masked]>
Date: Sat, 24 Jun 2006 02:10:22 +0300

> > languages) and relevant coding cookbook entries.

[... discussion how Shlomif works on his code was snipped ...]
The issue is that each one has his own style of working.  The optimal
environment for shlomif is not optimal for me and vice versa.

> knowledge of most of the core functions and concepts he needs to write stuff.

A properly-designed IDE with a properly-designed search engine and
properly-catalogued Internet-based database of documentation and code
examples - would extend the effective "ready, instant knowledge" used by
the programmer.

> could use in place of Perl, Python, etc.

Again, I believe more in designing the whole environment rather than in
polishing a single component (the programming language).  The Smalltalk
developers did develop a whole environment.  The FORTH inventor (Chuck
Moore) developed a whole environment together with FORTH (alebit very
unconventional one, as he has very unconventional mind).

> maintain such codebases in).

For a O(10^7) software development project, you need disciplined
programmers anyway.

I am now thinking that I have been barking on the wrong tree.

If we want to improve the productivity of software developers, then
investment of neuron firings will be more profitable, if it is made at
design and development of a multilingual environment (which I refer to
as IDE elsewhere), rather than at design of a single new language.

However this is a different problem, which is probably of no interest to
people, who make an hobby of designing new programming languages.

> dynamically changing the class inheritance graph.

It is possible to combine both concerns, by language design which
enforces type checking (including legality of unlocked accesses) by
default, but allows the software developer to explicitly override it.

An example:  the coercion operator in C/C++.  It allows you to shoot an
atom bomb at your foot, but you have to explicitly declare that this is
what you are doing.

> an extra work from the programmer.

One possible (and relatively simple) way to implement it:
You set the IDE to execute a macro, with arguments which you supply,
each time you modify its definition.  Then display the resulting code in
a separate window pane.  Then you can see immediately the effects of
changes in the macro definition.
                                           --- Omer
-- 
You haven't made an impact on the world before you caused a Debian
release to be named after Snufkin.
My own blog is at http://tddpirate.livejournal.com/

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 24 Jun 2006 11:27:52 +0300

Hi Omer!

On Saturday 24 June 2006 02:10, Omer Zak wrote:
> > > better than a new language.  I have in mind an IDE which integrates
> > > with a powerful search engine:

Right. Someone mentioned here that he was using vim for editing Java code 
after he found that Eclipse was not his cup of tea. I used gvim to edit 
Scheme code for my SICP exercises and other Scheme experimentation.

> examples - would extend the effective "ready, instant knowledge" used by

Possibly. However, many (most ?) programs will just fire-up vim, joe, edit.exe 
or their different idea of a lightweight and convenient command line editor 
to write the script (or edit it directly in the shell prompt), if a one-liner 
is all they want to write.

Only when editing, longer and usually large-scale program does an IDE 
justifies itself.

When I said "ready, instant knowledge" I meant knowledge that you do not have 
to lookup. For example, when programming Perl, I already know the syntax for 
regular expressions, most of the useful functions out of perlfunc, the 
operators that I need, etc. Some people know it for Python or Ruby or PHP or 
a different language. Part of the reason I'm still using Perl for my scripts 
is that looking everything up for a different language would be more time 
consuming, so even if the resultant script would be shorter, it would take me 
longer to write.

>
> > My intention in Park is to have an actually useful Lisp dialect that
> > people could use in place of Perl, Python, etc.
>

I've recently started learning Smalltalk. I first tried Squeak Smalltalk which 
is a Smalltalk environment written in C. (by the original Smalltalk 
designers) It looks very nice and all, but its conventions are unconventional 
and therefore annoying. I did not like its own internal editor (or could get 
the hang of it), and tried to find a way to write code in gvim. The people at 
#squeak even told me there was no way to write everything using plaintext. 
(No to mention the code exported by Squeak to a Unix system was \r instead of 
\n terminated.)

Meanwhile, I'm using GNU Smalltalk instead which allows me to edit everything 
as text and uses the native filesystem. Obviously Squeak is probably an 
extreme case of making an environment. Other environments like Eclipse or 
Emacs do play nicely with the other kids on the block.

Guy once posted a message about enhancing the "Unix IDE":

http://groups.yahoo.com/group/hackers-il/message/3681

>
> > I believe PHP would be suitable for writing > n*10M LOCs codebases. But
> > you'll probably need to have very good and discplined programmers for
> > achieving this. (But this goes without saying for any other language
> > you'd like to maintain such codebases in).
>

Right.

> > > Not really.  But can try to use theory
> > > (http://lambda-the-ultimate.org/) to specify a language, in which it is
> > > possible to prove that certain types of heisenbugs cannot occur.  For
> > > example, require explicit declaration of variables, which are shared
> > > among threads, so that the compiler (or interpreter) can check, at
> > > compile time, that accesses to those variables are properly locked.
> >
> > Do you include static run-time checking there? (Like the Stanford
> > checker, etc.)
> >
> > I'd rather not go too much into the direction of a language in which bugs
> > can be proven not to exist for fear of the following quote:
> >
> > <<<
> > "UNIX was not designed to stop its users from doing stupid things, as
> > that would also stop them from doing clever things." – Doug Gwyn
> >
> >
> > So I'm not sure whether such a feature would prevent doing smart things
> > like doing self-writing code, or even generating functions on the fly, or

I see. Well the best type system I saw was in Haskell and O'Caml which are 
truly strongly typed (but see [1]), but also functional. It is possible a 
similar type system exists in such languages as Module 3 and Ada 95, but I 
don't know them yet.

Only problem is that such a type system does not correspond very well with the 
spirit of Lisp until now, nor with my plans for Park. I personally find such 
types too limiting to think of in day-to-day-programming. Plus, while they 
may help write more bug-free code, they are not a necessity for large scale 
projects. (See my quote about doing smart things and stupid things in the 
previous message).

Python has a type system but it is still dynamically typed where each 
container can contain hetrogeneous values. Perl 6 aims to also have a 
stricter type system than Perl 5, but from what I understood you can do 
Haskell-like type signatures like $foo is a hash whose keys are string and 
his values are hash refs whose "a" element is a string, and "omer" element is 
a float. But I may have mis-concepted the Perl 6 specs.

I don't see such type systems belonging in Park. Park would have an object 
system with a nice Object hierarchy (based on that of another language, but 
possibly somewhat different.) and everything would be an object there. Now, 
containers will generally manage references to the base class (in a similar 
way to Smalltalk or Java) as values. Since Park would be a symbolic language, 
then like Perl 5, you won't need any casting to call a method of any of the 
derived classes the object belongs to.

> > > It may be a good idea to design an IDE, which would allow a developer
> > > to have a view of code, which could be generated by a LISP macro, as
> > > the macro is being developed.
> > >
> > If we generalise the Halting problem than we can say a program to
> > conclusively analyse a different computer program would be impossible to
> > write. I don't see how I can get an IDE to analyse a macro and determine
> > how it does what it's supposed to do. The best way would probably be to
> > document the behaviour of the macros in the program a la doxygen or
> > javadoc. But this will require an extra work from the programmer.
>

Hmmm... wouldn't it still require some work from the programmer? Can you point 
to an implementation of this for a different Lisp dialect?

Regards,

	Shlomi Fish

[1] - http://perl.plover.com/yak/12views/samples/notes.html - search 
for "Strongly Typed".

--

---

From: Omer Zak <[email-masked]>
Date: Sun, 25 Jun 2006 00:51:52 +0300

> is all they want to write.

The search engine can be available in a separate window, say inside a
Web browser.  It would be open all the time.

> consuming, so even if the resultant script would be shorter, it would take me 

Limiting the tools used to write one-liners limits the power of one's
one-liners.  My suggestion is about expanding the scope of tools
immediately available for this.  Especially, example code fragments will
be very useful for writing one-liners in a language which is not being
frequently used.  And this is very good when a different than usual
language needs to be used due to better match of its paradigms and the
problem in hand.

> http://groups.yahoo.com/group/hackers-il/message/3681

The task of looking for references and examples is just one of the tasks
which Guy's approach would optimize.  I claim that optimizing this task
has huge payoff.

> to an implementation of this for a different Lisp dialect?

I never saw such an implementation.

The programmer has to tell the IDE that he wants to have the macro
automatically invoked with certain arguments each time it is modified;
and to have the resulting code automatically displayed in a pane.

Then it will work somewhat like Watch windows in a debugger IDE (the
windows which display variable or register values, which change each
time you step through an instruction in a debugger).

                                              --- Omer

---

From: Shlomi Fish <[email-masked]>
Date: Sun, 25 Jun 2006 11:22:15 +0300

On Sunday 25 June 2006 00:51, Omer Zak wrote:
> > > examples - would extend the effective "ready, instant knowledge" used
> > > by the programmer.
> >
> > Possibly. However, many (most ?) programs will just fire-up vim, joe,
> > edit.exe or their different idea of a lightweight and convenient command
> > line editor to write the script (or edit it directly in the shell
> > prompt), if a one-liner is all they want to write.
>

I see. Well, as far as Konsole+bash+vim (or different implementations of any 
of them) is concerned, I could use Firefox or Konqueror (or possibly Opera, 
which I try not to use too much because of its licence) very effectively. If 
Park's online documentation and community will be good enough, which is a 
goal I hope to achieve (and mention in the informal spec), then people can 
use such a browser with some customisations.

BTW, is anyone aware of anything like "perldoc -f $SOMEFUNC" for Perl 
(e.g: "perldoc -f length" or "perldoc -f open") for different P-languages. 
What I mean is a way to access documentation to individual functions, which 
is very convenient. For perl I'd also like to have something 
like "perldoc --var '$@'" for built-in variables, but this is not applicable 
for Python for instance.

> > When I said "ready, instant knowledge" I meant knowledge that you do not
> > have to lookup. For example, when programming Perl, I already know the
> > syntax for regular expressions, most of the useful functions out of
> > perlfunc, the operators that I need, etc. Some people know it for Python
> > or Ruby or PHP or a different language. Part of the reason I'm still
> > using Perl for my scripts is that looking everything up for a different
> > language would be more time consuming, so even if the resultant script
> > would be shorter, it would take me longer to write.
>
> one-liners.  

I don't see how. One-liners are things you write using an existing language. 
They are not impossible to do (e.g: the halting problem), probably polynomial 
complexity, within the how-to-do-something knowledge of the programmer, and 
don't require any extraenous knowledge. Obviously, if a programmer only knows 
C (without regexps, etc.) and tries to write a one-liner in it, it'll take 
him quite a while to write. Most Perl one-liners will also probably take a 
while to write with Visual Basic Classic. 

I wouldn't write the same one-liner now that I wrote when I started using 
UNIX, but that's because I'm not as clueless today as I used to be. 
Nevertheless, my one-liners back then were ok for what I needed to do.

BTW, I (electronically) know someone who writes his one-liners in Haskell. 
However, after I analysed one of his programs (which took me quite a while) I 
found out i had an O(N^2) complexity instead of the optimal O(N). A different 
program I wrote in Haskell, using recursion and matching rules, was in fact 
O(N). He said his program worked for him, etc.

> My suggestion is about expanding the scope of tools 
> frequently used. 

That's right. But I hope Park will be frequently used.

> And this is very good when a different than usual 

Hmmm... yes. 

> which Guy's approach would optimize.  I claim that optimizing this task

Right. I see your point.

> > > each time you modify its definition.  Then display the resulting code
> > > in a separate window pane.  Then you can see immediately the effects of
> > Hmmm... wouldn't it still require some work from the programmer? Can you
> > point to an implementation of this for a different Lisp dialect?
>

OK, this is not very comforting.

> time you step through an instruction in a debugger).

I see. Well, this may require some extra work on part of the programmer. And 
most programmers are lazy (and impatient and hubrisful), and wouldn't want to 
do that. It's like most code out there doesn't have doxygen documentation, or 
much less written using literate programming tools.

Regards,

	Shlomi Fish