http://www.shlomifish.org/me/blogs/ NYTProf-3 is Out! http://community.livejournal.com/shlomif_tech/44836.html <a href="http://blog.timbunce.org/2009/12/24/nytprof-v3-worth-the-wait/">Tim Bunce writes on his blog about the new features in Devel-NYTProf version 3</a>. Devel-NYTProf is a profiler for the <a href="http://perl-begin.org/">Perl programming language</a>, which has put all the previous attempts in profiling in the dust, and now it's even better than before. Enjoy! (Thanks to <a href="http://mail.pm.org/pipermail/sanfrancisco-pm/2010-February/002704.html">Fred Moyer's post</a> on the San-Fransisco Perl Mongers mailing list). Shlomi Fish ( shlomif@iglu.org.il ) 3 version perl profiler release nytprof http://community.livejournal.com/shlomif_tech/44836.html http://community.livejournal.com/shlomif_tech/44836.html Fri, 05 Feb 2010 16:31:51 GMT Project Euler Problem #10 in Haskell, Perl and C http://community.livejournal.com/shlomif_tech/44006.html zerothorder told <a href="http://zerothorder.blogspot.com/2009/11/after-drinking-functional-immutable-and.html">how he found a solution for Project Euler's Problem 10 in Haskell</a>. The problem is "Find the sum of all primes less than 2,000,000". It is given here below: <pre> primes :: [Integer] primes = 2 : filter isPrime [3, 5 ..] where -- only check divisibility of the numbers less than the square root of n isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes divides n p = n `mod` p == 0 result = sum $ takeWhile (< 2000000) primes main = do putStrLn( show result ) </pre> He says "If this doesn't give you a nerdgasm, I don't know what will.". The problem is that this nerdgasm will last a long time. Benchmarking this program gives that 20 iterations of it run at 310 seconds - less than - about 15 seconds each (on my Pentium 4 2.4GHz machine running Mandriva Linux Cooker). So next I tried a better Haskell implmenetation that I recalled from a thread I started in the Haskell Café mailing list about implementing <a href="http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes">a sieve of Eratosthenes in Haskell</a>: <pre> import Data.Int primes :: Int64 -> [Int64] primes how_much = sieve [2..how_much] where sieve (p:x) = p : (if p <= mybound then sieve (remove (p*p) x) else x) where remove what (a:as) | what > how_much = (a:as) | a < what = a:(remove what as) | a == what = (remove (what+step) as) | a > what = a:(remove (what+step) as) remove what [] = [] step = (if (p == 2) then p else (2*p)) sieve [] = [] mybound = ceiling(sqrt(fromIntegral how_much)) --main = print (length (primes 1000000)) main = print (sum (primes 2000000)) </pre> This does not involve costly operations such as modulo or division and 20 iterations of it run at 135 wallclocks seconds - over two times faster than zeroth's Haskell version. Now how about Perl? Since Perl has assignment, we have the advantage that we can create a vector of bits that we will mark with the primes by iterating over all numbers up to the root of the limit. Here is the code: <pre> #!/usr/bin/perl use strict; use warnings; use Math::BigInt lib => 'GMP'; my $limit = 2_000_000; my $primes_bitmask = ""; my $loop_to = int(sqrt($limit)); my $sum = 0; my $total_sum = Math::BigInt->new('0'); for my $p (2 .. $loop_to) { if (vec($primes_bitmask, $p, 1) == 0) { $sum += $p; my $i = $p * $p; while ($i < $limit) { vec($primes_bitmask, $i, 1) = 1; } continue { $i += $p; } } } for my $p ($loop_to .. $limit) { if (vec($primes_bitmask, $p, 1) == 0) { if (($sum += $p) > (1 << 30)) { $total_sum += $sum; $sum = 0; } } } $total_sum += $sum; print "$total_sum\n"; </pre> 20 runs of it run at 95 walclock seconds - even faster than the Haskell version. But it gets better. Since all the primes we encounter greater than 2 are not even, we can create a map of their pseduo-halves and conserve on memory and iterations. This is the Perl version: <pre> #!/usr/bin/perl use strict; use warnings; use Math::BigInt lib => 'GMP'; my $limit = 2_000_000; my $primes_bitmask = ""; my $loop_to = (int(sqrt($limit)))>>1; my $half_limit = ($limit-1)>>1; my $sum = 0+2; my $total_sum = Math::BigInt->new('0'); for my $half (1 .. $loop_to) { if (vec($primes_bitmask, $half, 1) == 0) { my $p = (($half<<1)+1); $sum += $p; my $i = ($p * $p)>>1; while ($i < $limit) { vec($primes_bitmask, $i, 1) = 1; } continue { $i += $p; } } } for my $half ($loop_to .. $half_limit) { if (vec($primes_bitmask, $half, 1) == 0) { if (($sum += (($half<<1)+1)) > (1 << 30)) { $total_sum += $sum; $sum = 0; } } } $total_sum += $sum; print "$total_sum\n"; </pre> Running this 20 times takes 73 wallclock seconds, close to half that of my Haskell version. Then I wondered how long C will take. Here is a C implementation without the halving: <pre> #include <string.h> #include <math.h> #include <stdint.h> #include <stdio.h> #define limit 2000000 int8_t bitmask[(limit+1)/8]; int main(int argc, char * argv[]) { int p, i; int mark_limit; long long sum = 0; memset(bitmask, '\0', sizeof(bitmask)); mark_limit = (int)sqrt(limit); for (p=2 ; p <= mark_limit ; p++) { if (! ( bitmask[p>>3]&(1 << (p&(8-1))) ) ) { /* It is a prime. */ sum += p; for (i=p*p;i<=limit;i+=p) { bitmask[i>>3] |= (1 << (i&(8-1))); } } } for (; p <= limit; p++) { if (! ( bitmask[p>>3]&(1 << (p&(8-1))) ) ) { sum += p; } } printf("%lli\n", sum); return 0; } </pre> This was too fast to measure with 20 runs alone, so 500 runs of it took 15 seconds, two or three orders of magnitude faster than the fastest Haskell or Perl versions. But naturally, we can apply the halving paradigm there too: <pre> #include <string.h> #include <math.h> #include <stdint.h> #include <stdio.h> #define limit 2000000 int8_t bitmask[(limit+1)/8/2]; int main(int argc, char * argv[]) { int half, p, i; int half_limit; int loop_to; long long sum = 0 + 2; memset(bitmask, '\0', sizeof(bitmask)); loop_to=(((int)(sqrt(limit)))>>1); half_limit = (limit-1)>>1; for (half=1 ; half <= loop_to ; half++) { if (! ( bitmask[half>>3]&(1 << (half&(8-1))) ) ) { /* It is a prime. */ p = (half << 1)+1; sum += p; for (i = ((p*p)>>1) ; i < half_limit ; i+=p ) { bitmask[i>>3] |= (1 << (i&(8-1))); } } } for( ; half < half_limit ; half++) { if (! ( bitmask[half>>3]&(1 << (half&(8-1))) ) ) { sum += (half<<1)+1; } } printf("%lli\n", sum); return 0; } </pre> 500 runs of it take 10 wallclock seconds - 54.35 times per second, and 50% better than the previous C version. And I still haven't applied platform-specific gcc optimisations. I should also note that the executables generated by ghc are extremely large in comparison to their C ones: <pre> $ ls -l c_* haskell_* -rwxr-xr-x 1 shlomi shlomi 6082 2009-12-04 07:46 c_mine -rwxr-xr-x 1 shlomi shlomi 6103 2009-12-04 07:46 c_mine_half -rwxr-xr-x 1 shlomi shlomi 6092 2009-12-04 07:46 c_mine_micro_opt -rwxr-xr-x 1 shlomi shlomi 796825 2009-12-04 07:56 haskell_mine -rwxr-xr-x 1 shlomi shlomi 571717 2009-12-04 07:46 haskell_zeroth </pre> <tt>c_mine_half</tt> is less than 1% the size of <tt>haskell_mine</tt> (and runs faster). When talking about this to other people, they said that Haskell has a very optimised primes sequence generator, which I can try using (which should be over 3 times as fast), and that it has two kinds of integers, which the other type is faster, and that it has a better way to emulate assignment. But the bottom line is that the naïve and intuitive way to write such programs in Haskell is under-performant, even in comparison to Perl, and 100 or 1,000 times as much in comparison to C. I've written this as a separate post and not as a comment to the original blog post because I'm very limited with the markup in the commenting there (I'm going to post a comment there with a link to this blog post, though). I should note that you can find all the code I mentioned inside <a href="http://github.com/shlomif/Project-Euler-Problem-10-Comparison">a dedicated Github repository</a>, and you can experiment with it further. Shlomi Fish ( shlomif@iglu.org.il ) problem summary primes project optimize optimise optimization c haskell perl 10 speed optimisation euler http://community.livejournal.com/shlomif_tech/44006.html http://community.livejournal.com/shlomif_tech/44006.html Sat, 30 Jan 2010 12:50:07 GMT XML-Grammar-Fortune version 0.0200 Was Released http://community.livejournal.com/shlomif_tech/43620.html I recently released version 0.0200 of the Perl 5 CPAN Module <a href="http://web-cpan.berlios.de/modules/XML-Grammar-Fortune/">XML-Grammar-Fortune</a>. XML-Grammar-Fortune is a module to maintain a collection of Unix-like fortune cookies as XML and convert them to XHTML and plain-text. New in this release are a heavily fixed and enhanced code for rendering the XML to plaintext and a more enhanced META.yml file. XML-Grammar-Fortune was successfully utilised for <a href="http://www.shlomifish.org/humour/fortunes/">my collection of Fortune Cookies</a>. Shlomi Fish ( shlomif@iglu.org.il ) release unix text fortune grammar hacktivity xml-grammar-fortune perl xml fortunes cpan http://community.livejournal.com/shlomif_tech/43620.html http://community.livejournal.com/shlomif_tech/43620.html Mon, 25 Jan 2010 15:49:53 GMT The CPAN Dependencies of Website Meta Language http://community.livejournal.com/shlomif_tech/43345.html In the past few days I resumed work on converting the <a href="http://thewml.org/">Website Meta Language</a> (WML) build-system from the GNU Autotools (also known as "GNU Autohell") to <a href="http://www.cmake.org/">CMake</a>. This involved implementing a lot of the existing makefile and autoconf logic in CMake and Perl. Yesterday, I converted the "wml_common" sub-directory which turned out to contain some ancient versions of CPAN distributions with a forced-upon way to install under the WML prefix. I decided to forego installing all of them and just check for their existence in the perl version that wml uses using some WML and "perl -MMyModule" code. I was able to find all the needed modules in the Mandriva repository, except one - <a href="http://search.cpan.org/dist/File-PathConvert/">File-PathConvert</a> , which I've never heard about. I checked it on CPAN, and saw that it was deprecated and superseded by File::Spec and Cwd. So I grepped the tree for its use and found it was used only in one Perl program which I quickly converted to File::Spec and Cwd. Now, the future build system of Website Meta Language will be simpler and it won't use deprecated modules. Oh! And <a href="http://blog.mozilla.com/blog/2010/01/21/firefox-3-6-release/">Firefox 3.6 (Gamma) is out</a>! I've been using the betas and RCs on Mandriva Cooker for some time now, and now I've upgraded to the stable version and it's still great. Shlomi Fish ( shlomif@iglu.org.il ) eumm wml language cmake perl firefox gnu autohell website meta cpan http://community.livejournal.com/shlomif_tech/43345.html http://community.livejournal.com/shlomif_tech/43345.html Fri, 22 Jan 2010 19:22:40 GMT Perl: Using Vim's snipMate for Perl 5 snippets http://community.livejournal.com/shlomif_tech/43262.html I noticed that there was some Perl 5 code that I had to type or copy-and-paste again and again on many occasions. So I decided to find a way to put it in one place and then recall it. I thought of writing it myself, but then recalled the <a href="http://www.vim.org/scripts/script.php?script_id=2540">snipMate snippets extension for Vim</a> (which <a href="http://www.catonmat.net/blog/vim-plugins-snipmate-vim/">Peteris Krumins covered in a blog post</a>), which allows that. After a little reading, I was able to prepare the snippets. I placed the following under <tt>~/.vim/snippets/perl/_slurp.snippet</tt> : <pre> sub _slurp { my $filename = shift; open my $in, "<", $filename or die "Cannot open '$filename' for slurping - $!"; local $/; my $contents = <$in>; close($in); return $contents; } </pre> And now I can type "_slurp<TAB>" to recall it. The existing "slurp" snippet in <tt>~/.vim/snippets/perl.snippets</tt> is quite evil, with an ugly inline line, typeglobs, and a two args open without a die statement. Thanks, but no thanks. Afterwards I added the following in <tt>~/.vim/snippets/perl/_ltestb.snippet</tt> <pre> local $Test::Builder::Level = $Test::Builder::Level + 1; </pre> And now I can type "_ltestb" (short for "local Test::Builder") and put this line there to create my own custom Test::More/Test::Builder tests. Thus, using snipMate, you can create your own short-hands for commonly-used snippets like that. Shlomi Fish ( shlomif@iglu.org.il ) test tip local slurp perl snipmate vim tip builder snippets vim http://community.livejournal.com/shlomif_tech/43262.html http://community.livejournal.com/shlomif_tech/43262.html Thu, 21 Jan 2010 16:26:13 GMT Report on the TelFOSS "Moose for Beginners" Meeting http://community.livejournal.com/shlomif_tech/42895.html Last Sunday, <a href="http://tel.foss.org.il/">Tel Aviv Open Source club (TelFOSS)</a> met to listen to Sawyer's presentation about "Moose for Beginners". Despite the fact that the meeting was well-publicised, very few people came. Sawyer did not prepare a lot of slides and since there were not many questions he finished quickly. After all that, he went to demonstrate how to implement several use cases that the audience has proposed as Moose programs, and despite that we still finished early. I asked about implementing an "XML parser", but in fact I wanted to implement a processor for a certain XML grammar. In any case, he misunderstood and told me to use Parse-RecDescent for that. I've grown to dislike P-RD for most stuff though. In any case, I'm getting tired of the low attendance in TelFOSS meetings. We do so much work in organising and publicising the meetings, and most people don't bother to come. Are the people too busy? Did they run out of steam? Why don't they come? This makes me frustrating and unwilling to further organise the TelFOSS meetings. Shlomi Fish ( shlomif@iglu.org.il ) telux telfoss meeting open perl club sawyer moose tel aviv source report http://community.livejournal.com/shlomif_tech/42895.html http://community.livejournal.com/shlomif_tech/42895.html Wed, 20 Jan 2010 12:35:04 GMT New Solver for "Black Hole" Solitaire in Perl http://community.livejournal.com/shlomif_tech/42309.html I wrote <a href="http://www.shlomifish.org/open-source/projects/black-hole-solitaire-solver/">a new solver for the solitaire "Black Hole"</a>. Here's the complete story: Yesterday, I was going to play a game on <a href="http://www.brainbashers.com/">Brain Bashers</a> when I remembered the Penguin solitaire variant, which is similar to Freecell and which I wanted to cover on the Cards Wikia. So I did a Google search for "penguin solitaire" and saw it had <a href="http://en.wikipedia.org/wiki/Penguin_%28solitaire%29">a page on the wikipedia</a>. There I saw that it was invented by <a href="http://en.wikipedia.org/wiki/David_Parlett">David Parlett</a> and I saw there that he also invented <a href="http://en.wikipedia.org/wiki/Black_Hole_%28solitaire%29">a solitaire called "Black Hole"</a> So I looked for it in <a href="http://pysolfc.sourceforge.net/">PySolFC</a>, read the instructions there and started to play. The game involves putting a card that is one above or below the foundation (wrapping from kings to aces). I noticed that there wasn't any over-populated talon or something like that there, which meant that I could probably build a DFS-based solver for it. So I set out to see if it was possible. The first thing I did was to adapt my PySol/PySolFC game generator to generate the initial board of the PySolFC deals. This turned out to be a very complicated and frustrating task for me, because I had to understand what's going on with the code. But after a lot of playing with it, I was able to get it to generate the initial deals. The changes for that are in the Freecell Solver trunk now. Then I started working on the solver. I decided to write it in Perl 5 because it's a good prototyping language and I know it well. When writing it, I heavily optimised for speed and low memory consumption by using bit fields ( see <a href="http://perldoc.perl.org/functions/vec.html">the vec() function</a> ) and a gigantic hash lookup. Then, after it was written, came the moment of truth: I ran it on the board and it reported success. Great! But what's the solution? So I added some solution tracing logic, to output the cards that should be moved. Then I was able to play it and the game was solved. Yay! I tried it on another game and it also worked. Then I ran it on the first few games. Game #1 was reported as unsolvable, Game #2 was solved after a while, and Game #3 consumed over 15% of my RAM and then was solved. So it seemed to be working nicely. Today I wrapped the code inside its own CPAN package and prepared a nice app-in-a-module and a script for it. he solver's code is made available under the permissive MIT/X11 licence - share and enjoy. And as I mentioned before <a href="http://www.shlomifish.org/open-source/projects/black-hole-solitaire-solver/">here is its home-page</a>. Shlomi Fish ( shlomif@iglu.org.il ) automated patience black perl games hole game cards solver solitaire solvers davi python http://community.livejournal.com/shlomif_tech/42309.html http://community.livejournal.com/shlomif_tech/42309.html Mon, 11 Jan 2010 17:32:29 GMT libtap-1.02 Was Released http://community.livejournal.com/shlomif_tech/42035.html libtap is a C library to implement the <a href="http://testanything.org/">Test Anything Protocol (TAP)</a>, which is used by the test suites of perl 5, CPAN modules and other programs. libtap allows one to write C code that can be analysed by TAP harnesses such as <a href="http://web-cpan.berlios.de/modules/Test-Run/">Test-Run</a>. I started maintaining Nik Clayton's libtap, which he seems to have abandoned <a href="http://www.shlomifish.org/open-source/projects/libtap/">on my homepage</a>. There's also <a href="http://svn.berlios.de/svnroot/repos/web-cpan/libtap/">a Subversion repository</a>. Reading from the NEWS file: <blockquote> tap-1.02: <ol> <li> Add the COPYING file. </li> <li> Now disabling the thread-safety by default (in order to have it add <tt>-DLIBTAP_ENABLE_BROKEN_THREAD_SAFE</tt> . </li> <li> Changed configure.in to configure.ac. </li> </ol> </blockquote> Shlomi Fish ( shlomif@iglu.org.il ) test perl release anything qa testing tap new c libtap protocol http://community.livejournal.com/shlomif_tech/42035.html http://community.livejournal.com/shlomif_tech/42035.html Fri, 08 Jan 2010 11:18:19 GMT Next TelFOSS Meeting: Moose, the Perl OOP System on 17-January-2009 http://community.livejournal.com/shlomif_tech/41624.html The <a href="http://www.cs.tau.ac.il/telux/">Tel Aviv Open Source Club</a> will host <a href="http://wiki.osdc.org.il/index.php/Tel_Aviv_Meeting_on_17_January_2010">the talk "Moose, the Perl OOP System" for beginners</a> by Yaron Meiry (Sawyer) on Sunday, 17-January-2009, at 18:00 (note the change of time since last year) in Tel Aviv University, Holcblat Hall No. 007 in the corridor of the exact sciences building (note the change of place from last year). Further details, maps for arrival, etc. can be found on <a href="http://www.cs.tau.ac.il/telux/">the web-site</a> <a href="http://wiki.osdc.org.il/index.php/Tel_Aviv_Meeting_on_17_January_2010">and in the Wiki</a>. Attendance is free, it is not necessary to RSVP and everyone are welcome. With any other problems, feel free to <a href="http://www.shlomifish.org/me/contact-me/">contact the organiser</a>. Abstract <a href="http://moose.perl.org/">Moose</a> is a post-modern Object-Oriented Programming system for Perl 5. It was written since its originator (Stevan Little) was jealous of the capabilities that Perl 6 provided in regards to OOP, and so instead of switching to Ruby, he worked on developing a similar system for Perl 5. Moose drew inspiration from the Object Oriented Programming capabilities of many languages such as Perl 6, Smalltalk, Common Lisp (CLOS), Ruby, Java, OCaml and other languages, while remaining faithful to its Perl 5 roots. About the Lecturer Yaron Meiry is a systems' administrator and a Perl developer. He gives talks about open source, free software, security and programming standards. Yaron has previousl given the <a href="http://wiki.osdc.org.il/index.php/Tel_Aviv_Meeting_on_28_June_2009">presentation about red flags in programming of very high languages</a> in the Tel Aviv Club. <hr /> We are always looking for people who will volunteer to give presentations on various topic that are related to open source code and to computers. In case you are interested to give a talk, or that you have a suggestion for a talk that interests you, we'll be happy to hear from you. Shlomi Fish ( shlomif@iglu.org.il ) announcement activities open source israel telux telfoss meeting foss perl linux moose tel aviv http://community.livejournal.com/shlomif_tech/41624.html http://community.livejournal.com/shlomif_tech/41624.html Tue, 05 Jan 2010 08:48:18 GMT פגישת קוד-פתוח-תל-אביב: Moose למתחילים - מערכת האובייקטים לפרל http://community.livejournal.com/shlomif_tech/41232.html <div dir="rtl" align="right"> <a href="http://www.cs.tau.ac.il/telux/">מועדון הקוד הפתוח התל-אביבי (תלוקס)</a> ייפגש שוב כדי לשמוע את <a href="http://wiki.osdc.org.il/index.php/Tel_Aviv_Meeting_on_17_January_2010">הרצאתו של ירון מאירי (Sawyer) אודות "Moose, מערכת תכנות מונחה העצמים לשפת פרל (למתחילים)"</a>. ההרצאה תתקיים ביום ראשון, 17 בינואר 2010, בשעה 18:00 (שימו לב לשינוי בשעה משנה שעברה), באולם הולצבלט, מס' 007 במסדרון הבניינים למדעים מדויקים (שימו לב לשינוי במיקום משנה שעברה) באוניברסיטת תל אביב. פרטים נוספים, מפות להגעה וכיוצא בזה, ניתן למצוא <a href="http://www.cs.tau.ac.il/telux/">באתר</a> <a href="http://wiki.osdc.org.il/index.php/Tel_Aviv_Meeting_on_17_January_2010">ובוויקי</a>. הנוכחות בהרצאה היא חינמית ולא נדרשת הרשמה מראש. <a href="http://moose.perl.org/">Moose</a> הינה מערכת תכנות מונחה-עצמים פוסט-מודרנית לשפה פרל 5. היא נכתבה מכיוון שההוגה המקורי שלה (סטיבן ליטל) קינא במה שפרל 6 סיפקה בנוגע לתכנות מונחה עצמים, ולכן במקום לעבור לרובי הוא שקד על פיתוח מערכת דומה לפרל 5. Moose שאבה השראה מיכולות ה-OOP של שפות רבות כמו פרל 6, Smalltalk, ליספ, רובי, ג'אווה, OCaml ושפות אחרות כשהיא נשארת נאמנה לשורשי ה-פרל 5 שלה. ירון מאירי הינו מנהל מערכות ומפתח פרל. הוא מרצה על קוד פתוח, תוכנה חופשית, אבטחה וסטנדרטים של תכנות. ירון העביר בעבר את <a href="http://wiki.osdc.org.il/index.php/Tel_Aviv_Meeting_on_28_June_2009">ההרצאה על דגלים אדומים בתכנות עבור שפות עיליות ביותר</a> במועדון התל-אביבי. אנו תמיד מחפשים מרצים שיתנדבו לתת הרצאות בנושאים שונים הקשורים לקוד-פתוח ולמחשבים. במידה שאתם מעוניינים לתת הרצאה, או שיש לכם הצעה להרצאה שמעניינת אתכם, נשמח לשמוע ממכם. </div> Shlomi Fish ( shlomif@iglu.org.il ) announcement mobile activities open source israel telux telfoss meeting foss perl linux moose tel aviv phones cellphones http://community.livejournal.com/shlomif_tech/41232.html http://community.livejournal.com/shlomif_tech/41232.html Sun, 03 Jan 2010 20:52:46 GMT