Subject: The Two Schools of Data Representation

From: Shlomi Fish <[email-masked]>
Date: Wed, 21 Sep 2005 14:06:28 +0300

An IRC discussion I took part in led to me asking why the various dialects of 
LISP did not support useful primitives like random file I/O, sockets or 
regular expressions. I was told that file I/O and sockets were not portable 
enough because some systems like ITS or the LISP Machine where LISP could run 
on did not have them.

This led us to a revelation that there were in fact two schools of data 
representation:

1. The UNIX/C/TCP/IP school where data is repsresented as octet streams that 
are passed from application to application and processed. (or permanently 
stored).

2. The LISP/LISP Machine/LISPORBA (where LISPORBA is a yet to be invented RPC 
layer for LISP that does remote operations using LISP objects). In that daata 
is represented as objects (not necessarily in the OOP sense): handled to 
nested data structures, where the semantics of the datum is determined 
according to its place.

So, my partner argued that because regular expressions were intended to 
manipulate octet streams they had no place in an object-manipulating  
environment such as LISP.

However, octet streams are perfectly natural. Humans have thought and 
communicated using streams of characters since the invention of speech. 
Throughout human history a great deal of documents were written using 
characters in unicode. So even if you're using LISP, it is not inconceivable 
that you will need to process an octet stream. (Unless you can find a way to 
convert the Bible or the Complete Works of Shakespeare into objects).

Octet streams are also natural as far as the implementation of computers is 
concerned. Computers are made out of electrical wires and logical gates, 
which pass and manipulate bits and words.

In a way, the object-representation is also natural to humans, but in a 
different way than byte streams. We think in concepts, and often make mental 
relationships by drawing diagrams, etc.

Objects are generally more efficient to manipulate than octet streams. 
Replacing a substring in a buffer with one of different length is an O(N) 
operation, and some common algorithms would be even less efficient. Using 
efficient data structures can often make a huge improvement.

I believe a good programming environment should have good capabilities with 
regards to both schools, as well as flexible ways to serialize and 
de-serialize data. Sometimes the best thing to do is have complex objects 
that contain buffers of bytes. The agile languages (Perl, Python, etc.) are a 
good example of languages that offer very good capabilities in both fields 
and in what's between them.

And naturally, because chances (at least 99.9% in my approximation) that your 
LISP virtual machine runs on a UNIX-like environment, I believe the standard 
for new LISP dialects should offer rich primitives for manipulating octet 
streams. Otherwise, it would be close to useless. Like it or not, our 
underlying systems still think in octet streams, and reality to be conquered 
must be obeyed. 

Regards,

	Shlomi Fish

---------------------------------------------------------------------
Shlomi Fish      [email-masked]
Homepage:        http://www.shlomifish.org/

95% of the programmers consider 95% of the code they did not write, in the
bottom 5%.

---

From: Omer Zak <[email-masked]>
Date: Fri, 23 Sep 2005 05:15:36 +0300

> enough because some systems like ITS or the LISP Machine where LISP could run 

[... snipped ...]

> underlying systems still think in octet streams, and reality to be conquered 

A similar dilemma led to the demish of FORTH.  The FORTH advocates
refused to or were split about the issue of standard libraries.  Instead
of providing standard libraries while retaining the powerful FORTH
mechanisms of overriding those libraries, people advocated doing without
libraries, forcing each FORTH application developer to re-invent the
wheel.

Standards, such as the ones advocated above, should not be standard core
language features but standard libraries.

The philosophy for specifying libraries for languages like FORTH and
LISP/Scheme should be that software developers, who do not demand the
ultimate in performance and power from a certain API - would use the
provided library rather than re-invent their own wheel.  Those, who are
more demanding, should be assisted in re-building their wheel from parts
provided by the libraries glued together with parts custom-written by
the developer.

Therefore the libraries should be structured in such a way that it would
be possible to override low-level handling of the various entities
manipulated by the library.
                                     --- Omer
-- 
Every good master plan involves building a time machine.  Moshe Zadka
My own blog is at http://www.livejournal.com/users/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 24 Sep 2005 21:21:51 +0300

Hi Omer!

Just wanted to note that I agree with almost everything you said.

Regards,

	Shlomi Fish