Subject: Re: When we cannot waste 1MB per datum.

From: Omer Zak <[email-masked]>
Date: Mon, 25 Jun 2012 22:21:34 +0300

Nice to remind the world that I existed also at 2002.  :-)
Obviously, representing a single FC position is not something that
should, need to or even could consume a full 1MB RAM.

What I had in mind was AI stuff.

--- Omer

On Mon, 2012-06-25 at 17:14 +0300, Shlomi Fish wrote:
> Hi all,
> 
> Omer Zak (CCed to this message) wrote this old (2002) Hackers-IL message:
> 
> http://tech.groups.yahoo.com/group/hackers-il/message/1894
> 
> Titled "How to exploit 1MB data structures?" this message reads:
> 
> <<<<
> The future computers will have terabytes of memory (mostly in disks, but
> some in RAM). They will be able to manipulate megabytes at the time
> needed by yesterday's computers to manipulate single bytes.
> 
> I wonder how can all those extra bytes be harnassed. (AI anyone?)
> 
> Let's try to use 1MB to represent the string "The beep has booped".
> The data structure will contain:
> >>>>
> 
> Now I'd like to give a view from the other side of the fence: when have
> milliards (10^9s) of data items and we need to fit them all in RAM (first -
> 128 GB - now I also got access to a machine with 512 GB), because storing and
> retrieving them on disk is far too slow. This is the case of Freecell Solver's
> dbm_fc_solver where I've spent a lot of time reducing its RAM consumption:
> 
> * http://tech.groups.yahoo.com/group/fc-solve-discuss/message/1114
> 
> And there's another optimisation coming up:
> 
> * http://tech.groups.yahoo.com/group/fc-solve-discuss/message/1115
> 
> If every position in the game’s graph consumed 1MB of RAM, then I could fit
> at most 512*1024 of them in 512 GB of RAM, which is all I have at the moment.
> And I can reach much more than 512K of positions on my underpowered home
> computers. Furthermore, manipulating these 1MB data will likely be much more
> time-consuming than the compact ones I'm using at the moment. Some of my scans
> reached 3 milliard stored positions, and such an amount would have consumed
> roughly 3 peta bytes (3 * 10^15), which I suspect are beyond any single HPC
> machine at the moment (to say nothing of consumer computers).
> 
> I think that if we have large amounts of RAM resources at our disposal, we
> should not carelessly waste them like Omer suggests (unless of course we ran
> into a use-case that necessitates that).
> 
> Regards,
> 
> 	Shlomi Fish 

-- 
PHP - the language of the Vogons.
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html