http://www.shlomifish.org/me/blogs/ A Perl-based →with(...) method http://community.livejournal.com/shlomif_tech/46303.html Lately, I've been writing some <a href="http://pdl.perl.org/">PDL (Perl Data Language)</a> code and noticed that I've been doing the following: <pre> my $temp_pdl = $pdl->…->final_expr(); my $result_pdl = $temp_pdl->where($temp_pdl < 0); </pre> I've been thinking if I could somehow write it like that: <pre> my $result_pdl = $pdl->…->final_expr()->with(sub { $_->where($_ > 0)}); </pre> Which seems cleaner and more elegant. I went on IRC asking about it and someone suggested <tt>rindolf: (sub { local $_ = shift; $_ * $_ * $_})-gt;( $obj->... );</tt> (up to an episolon, but naturally this reverses the order and messes with the flow of the code). Eventually <a href="http://www.shadowcat.co.uk/blog/matt-s-trout/">Matt S. Trout (mst)</a> proposed the following solutions: <pre> sub UNIVERSAL::with { local $_ = shift; my $sub = shift; return $sub->($_,@_); } my $_with = sub { local $_ = shift; my $sub = shift; return $sub->($_,@_); }; </pre> (I had figured something like that was possible previously, but I was beating around the bush.) The $_with approach is covered in <a href="http://www.shadowcat.co.uk/blog/matt-s-trout/madness-with-methods/">one of Matt's blog posts titled "Madness with Methods"</a>. I wondered if it will work with <a href="http://search.cpan.org/dist/autobox/">autobox</a> and mst said that <q>I don't know and I hope not since the UNIVERSAL approach is evil.</q>. But it does, out-of-the-box: <pre> #!/usr/bin/perl use strict; use warnings; use autobox; sub UNIVERSAL::with { local $_ = shift; my $sub = shift; return $sub->($_,@_); } my $_with = sub { local $_ = shift; my $sub = shift; return $sub->($_,@_); }; print +(10+1)->with(sub { $_ * $_; }), "\n"; print +(10+1)->$_with(sub { $_ * $_; }), "\n"; </pre> This prints 121 twice. I now wonder what to do with this knowledge. I have some aspirations of releasing it to the cpan as a mini-module. Still, it's really cool. Finally, I should note that it seems like I'm going to lose my current Planet Perl Iron Man karma due to inadequate blogging. Oh well. Shlomi Fish ( shlomif@iglu.org.il ) perl fun http://community.livejournal.com/shlomif_tech/46303.html http://community.livejournal.com/shlomif_tech/46303.html Mon, 01 Mar 2010 17:22:55 GMT New Perl Quiz-of-the-Whatever: Short Freecell Solutions http://community.livejournal.com/shlomif_tech/45610.html I published a new <a href="http://article.gmane.org/gmane.comp.lang.perl.qotw.discuss/2655">Perl Quiz-of-the-Whatever about finding short Freecell solutions</a>. Hope you enjoy thinking about it. Here is the <a href="http://perl.plover.com/qotw/">Perl Quiz-of-the-Whatever (formerly "Quiz of the Week") information.</a>. Shlomi Fish ( shlomif@iglu.org.il ) perl fc-solve freecell http://community.livejournal.com/shlomif_tech/45610.html http://community.livejournal.com/shlomif_tech/45610.html Mon, 22 Feb 2010 21:57:30 GMT Solving gaal's Circuit Riddle Using Perl http://community.livejournal.com/shlomif_tech/45409.html <a href="http://gaal.livejournal.com/234795.html">gaal (= Gaal Yahas) mentions a riddle</a> on his blog: <blockquote> There’s a box with three input signals: A, B, C. It has three output signals: ~A, ~B, ~C (that is, the negation of each input). Design a circuit satisfying the above description. You have two NOT gates, and as many AND and OR gates as you like. (If you solve this, don’t tell me the answer — I haven’t solved it yet; I just want to share the misery.) </blockquote> I decided that I will try to solve this riddle on my own and then attempt to write a solution using a Perl 5 program, which I've also thought about how to write. As it turned out, I ended up exploring and eventually completely solving the riddle using a series of Perl programs. I won't mention the final solution here, but I'll tell which programs I wrote to solve them and how I wrote them. The code can be found in <a href="http://bitbucket.org/shlomif/riddle-not-a-not-b-not-c/src/">its Mercurial repository</a> under the MIT/X11 licence, as I mentioned in <a href="http://gaal.livejournal.com/234795.html?thread=936235#t936235">a comment</a> to gaal's post. Well, first of all I decided to write a program that will take two negated expressions of AND and OR gates and see if one can arrive to the final solution using them and the given inputs (A, B and C). To do this I represented every such possible expression as its components of (A + ~A) × (B + ~B) × (C + ~C) - 8 in total which fit in an 8-bit datum, and which I indexed using an array of 256 items to see which ones exist in the population. After doing the coding I realised none of the expressions I tried worked. Then I decided to see if I there are any two possible expressions that when added to A, B an C will yield all the necessary outputs. I ran it and indeed found a single pair (and only a single pair) of such expressions (given as numbers) which I was able to analyse. I should note that the programs as I've written it using some functions and closures was too slow, so I ended up writing another Perl script to preprocess the code, and inline the functions, which has made everything much faster. (You can find this preprocessor in the repository). Then I wanted to see if I can indeed construct these expressions from A , B, and C using two negations. When I had thought about this, I believed I needed to maintain a collection of expressions' populations, and to keep how many negations were in each. But when I got to code it I thought of a simpler way: marry all the inputs using AND and OR until no more can be married, then go over the existing ones and negate each one in turn, then add the negated vector to the population and marry it with all the rest, and then do it again for all the existing inputs. Using this I was able to find a single solution. Then I had to output the solution. For that, I wrote a recursive expression dumper, which ended up outputting with many parentheses, so I had to post- process it a bit. The solution I got to was very complicated and I believe it would have taken me a long time to reach it without resorting to programming. But at least it saved me a lot of time and head-scratching. Shlomi Fish ( shlomif@iglu.org.il ) perl fun http://community.livejournal.com/shlomif_tech/45409.html http://community.livejournal.com/shlomif_tech/45409.html Fri, 19 Feb 2010 13:28:13 GMT Article Recommendation: "What Every CSer Should Know about Floating-Point Arithmetic" http://community.livejournal.com/shlomif_tech/45104.html I've read the paper <a href="http://docs.sun.com/source/806-3568/ncg_goldberg.html">"What Every Computer Scientist Should Know About Floating-Point Arithmetic"</a> and can recommend it as it has many interesting insights. I had received quite a lot of reports for bugs in <a href="http://search.cpan.org/dist/Statistics-Descriptive/">the "Statistics-Descriptive" CPAN distribution</a> that were caused by expected floating-point behaviour. Some of them I was able to resolve using some trickery, and some others possibly can be somewhat mitigated using the insights I've learnt in the article. Shlomi Fish ( shlomif@iglu.org.il ) perl recommendation article http://community.livejournal.com/shlomif_tech/45104.html http://community.livejournal.com/shlomif_tech/45104.html Thu, 11 Feb 2010 05:24:18 GMT NYTProf-3 is Out! http://community.livejournal.com/shlomif_tech/44836.html <a href="http://blog.timbunce.org/2009/12/24/nytprof-v3-worth-the-wait/">Tim Bunce writes on his blog about the new features in Devel-NYTProf version 3</a>. Devel-NYTProf is a profiler for the <a href="http://perl-begin.org/">Perl programming language</a>, which has put all the previous attempts in profiling in the dust, and now it's even better than before. Enjoy! (Thanks to <a href="http://mail.pm.org/pipermail/sanfrancisco-pm/2010-February/002704.html">Fred Moyer's post</a> on the San-Fransisco Perl Mongers mailing list). Shlomi Fish ( shlomif@iglu.org.il ) 3 version perl profiler release nytprof http://community.livejournal.com/shlomif_tech/44836.html http://community.livejournal.com/shlomif_tech/44836.html Fri, 05 Feb 2010 16:31:51 GMT Project Euler Problem #10 in Haskell, Perl and C http://community.livejournal.com/shlomif_tech/44006.html zerothorder told <a href="http://zerothorder.blogspot.com/2009/11/after-drinking-functional-immutable-and.html">how he found a solution for Project Euler's Problem 10 in Haskell</a>. The problem is "Find the sum of all primes less than 2,000,000". It is given here below: <pre> primes :: [Integer] primes = 2 : filter isPrime [3, 5 ..] where -- only check divisibility of the numbers less than the square root of n isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes divides n p = n `mod` p == 0 result = sum $ takeWhile (< 2000000) primes main = do putStrLn( show result ) </pre> He says "If this doesn't give you a nerdgasm, I don't know what will.". The problem is that this nerdgasm will last a long time. Benchmarking this program gives that 20 iterations of it run at 310 seconds - less than - about 15 seconds each (on my Pentium 4 2.4GHz machine running Mandriva Linux Cooker). So next I tried a better Haskell implmenetation that I recalled from a thread I started in the Haskell Café mailing list about implementing <a href="http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes">a sieve of Eratosthenes in Haskell</a>: <pre> import Data.Int primes :: Int64 -> [Int64] primes how_much = sieve [2..how_much] where sieve (p:x) = p : (if p <= mybound then sieve (remove (p*p) x) else x) where remove what (a:as) | what > how_much = (a:as) | a < what = a:(remove what as) | a == what = (remove (what+step) as) | a > what = a:(remove (what+step) as) remove what [] = [] step = (if (p == 2) then p else (2*p)) sieve [] = [] mybound = ceiling(sqrt(fromIntegral how_much)) --main = print (length (primes 1000000)) main = print (sum (primes 2000000)) </pre> This does not involve costly operations such as modulo or division and 20 iterations of it run at 135 wallclocks seconds - over two times faster than zeroth's Haskell version. Now how about Perl? Since Perl has assignment, we have the advantage that we can create a vector of bits that we will mark with the primes by iterating over all numbers up to the root of the limit. Here is the code: <pre> #!/usr/bin/perl use strict; use warnings; use Math::BigInt lib => 'GMP'; my $limit = 2_000_000; my $primes_bitmask = ""; my $loop_to = int(sqrt($limit)); my $sum = 0; my $total_sum = Math::BigInt->new('0'); for my $p (2 .. $loop_to) { if (vec($primes_bitmask, $p, 1) == 0) { $sum += $p; my $i = $p * $p; while ($i < $limit) { vec($primes_bitmask, $i, 1) = 1; } continue { $i += $p; } } } for my $p ($loop_to .. $limit) { if (vec($primes_bitmask, $p, 1) == 0) { if (($sum += $p) > (1 << 30)) { $total_sum += $sum; $sum = 0; } } } $total_sum += $sum; print "$total_sum\n"; </pre> 20 runs of it run at 95 walclock seconds - even faster than the Haskell version. But it gets better. Since all the primes we encounter greater than 2 are not even, we can create a map of their pseduo-halves and conserve on memory and iterations. This is the Perl version: <pre> #!/usr/bin/perl use strict; use warnings; use Math::BigInt lib => 'GMP'; my $limit = 2_000_000; my $primes_bitmask = ""; my $loop_to = (int(sqrt($limit)))>>1; my $half_limit = ($limit-1)>>1; my $sum = 0+2; my $total_sum = Math::BigInt->new('0'); for my $half (1 .. $loop_to) { if (vec($primes_bitmask, $half, 1) == 0) { my $p = (($half<<1)+1); $sum += $p; my $i = ($p * $p)>>1; while ($i < $limit) { vec($primes_bitmask, $i, 1) = 1; } continue { $i += $p; } } } for my $half ($loop_to .. $half_limit) { if (vec($primes_bitmask, $half, 1) == 0) { if (($sum += (($half<<1)+1)) > (1 << 30)) { $total_sum += $sum; $sum = 0; } } } $total_sum += $sum; print "$total_sum\n"; </pre> Running this 20 times takes 73 wallclock seconds, close to half that of my Haskell version. Then I wondered how long C will take. Here is a C implementation without the halving: <pre> #include <string.h> #include <math.h> #include <stdint.h> #include <stdio.h> #define limit 2000000 int8_t bitmask[(limit+1)/8]; int main(int argc, char * argv[]) { int p, i; int mark_limit; long long sum = 0; memset(bitmask, '\0', sizeof(bitmask)); mark_limit = (int)sqrt(limit); for (p=2 ; p <= mark_limit ; p++) { if (! ( bitmask[p>>3]&(1 << (p&(8-1))) ) ) { /* It is a prime. */ sum += p; for (i=p*p;i<=limit;i+=p) { bitmask[i>>3] |= (1 << (i&(8-1))); } } } for (; p <= limit; p++) { if (! ( bitmask[p>>3]&(1 << (p&(8-1))) ) ) { sum += p; } } printf("%lli\n", sum); return 0; } </pre> This was too fast to measure with 20 runs alone, so 500 runs of it took 15 seconds, two or three orders of magnitude faster than the fastest Haskell or Perl versions. But naturally, we can apply the halving paradigm there too: <pre> #include <string.h> #include <math.h> #include <stdint.h> #include <stdio.h> #define limit 2000000 int8_t bitmask[(limit+1)/8/2]; int main(int argc, char * argv[]) { int half, p, i; int half_limit; int loop_to; long long sum = 0 + 2; memset(bitmask, '\0', sizeof(bitmask)); loop_to=(((int)(sqrt(limit)))>>1); half_limit = (limit-1)>>1; for (half=1 ; half <= loop_to ; half++) { if (! ( bitmask[half>>3]&(1 << (half&(8-1))) ) ) { /* It is a prime. */ p = (half << 1)+1; sum += p; for (i = ((p*p)>>1) ; i < half_limit ; i+=p ) { bitmask[i>>3] |= (1 << (i&(8-1))); } } } for( ; half < half_limit ; half++) { if (! ( bitmask[half>>3]&(1 << (half&(8-1))) ) ) { sum += (half<<1)+1; } } printf("%lli\n", sum); return 0; } </pre> 500 runs of it take 10 wallclock seconds - 54.35 times per second, and 50% better than the previous C version. And I still haven't applied platform-specific gcc optimisations. I should also note that the executables generated by ghc are extremely large in comparison to their C ones: <pre> $ ls -l c_* haskell_* -rwxr-xr-x 1 shlomi shlomi 6082 2009-12-04 07:46 c_mine -rwxr-xr-x 1 shlomi shlomi 6103 2009-12-04 07:46 c_mine_half -rwxr-xr-x 1 shlomi shlomi 6092 2009-12-04 07:46 c_mine_micro_opt -rwxr-xr-x 1 shlomi shlomi 796825 2009-12-04 07:56 haskell_mine -rwxr-xr-x 1 shlomi shlomi 571717 2009-12-04 07:46 haskell_zeroth </pre> <tt>c_mine_half</tt> is less than 1% the size of <tt>haskell_mine</tt> (and runs faster). When talking about this to other people, they said that Haskell has a very optimised primes sequence generator, which I can try using (which should be over 3 times as fast), and that it has two kinds of integers, which the other type is faster, and that it has a better way to emulate assignment. But the bottom line is that the naïve and intuitive way to write such programs in Haskell is under-performant, even in comparison to Perl, and 100 or 1,000 times as much in comparison to C. I've written this as a separate post and not as a comment to the original blog post because I'm very limited with the markup in the commenting there (I'm going to post a comment there with a link to this blog post, though). I should note that you can find all the code I mentioned inside <a href="http://github.com/shlomif/Project-Euler-Problem-10-Comparison">a dedicated Github repository</a>, and you can experiment with it further. Shlomi Fish ( shlomif@iglu.org.il ) problem summary primes project optimize optimise optimization c haskell perl 10 speed optimisation euler http://community.livejournal.com/shlomif_tech/44006.html http://community.livejournal.com/shlomif_tech/44006.html Sat, 30 Jan 2010 12:50:07 GMT XML-Grammar-Fortune version 0.0200 Was Released http://community.livejournal.com/shlomif_tech/43620.html I recently released version 0.0200 of the Perl 5 CPAN Module <a href="http://web-cpan.berlios.de/modules/XML-Grammar-Fortune/">XML-Grammar-Fortune</a>. XML-Grammar-Fortune is a module to maintain a collection of Unix-like fortune cookies as XML and convert them to XHTML and plain-text. New in this release are a heavily fixed and enhanced code for rendering the XML to plaintext and a more enhanced META.yml file. XML-Grammar-Fortune was successfully utilised for <a href="http://www.shlomifish.org/humour/fortunes/">my collection of Fortune Cookies</a>. Shlomi Fish ( shlomif@iglu.org.il ) release unix text fortune grammar hacktivity xml-grammar-fortune perl xml fortunes cpan http://community.livejournal.com/shlomif_tech/43620.html http://community.livejournal.com/shlomif_tech/43620.html Mon, 25 Jan 2010 15:49:53 GMT The CPAN Dependencies of Website Meta Language http://community.livejournal.com/shlomif_tech/43345.html In the past few days I resumed work on converting the <a href="http://thewml.org/">Website Meta Language</a> (WML) build-system from the GNU Autotools (also known as "GNU Autohell") to <a href="http://www.cmake.org/">CMake</a>. This involved implementing a lot of the existing makefile and autoconf logic in CMake and Perl. Yesterday, I converted the "wml_common" sub-directory which turned out to contain some ancient versions of CPAN distributions with a forced-upon way to install under the WML prefix. I decided to forego installing all of them and just check for their existence in the perl version that wml uses using some WML and "perl -MMyModule" code. I was able to find all the needed modules in the Mandriva repository, except one - <a href="http://search.cpan.org/dist/File-PathConvert/">File-PathConvert</a> , which I've never heard about. I checked it on CPAN, and saw that it was deprecated and superseded by File::Spec and Cwd. So I grepped the tree for its use and found it was used only in one Perl program which I quickly converted to File::Spec and Cwd. Now, the future build system of Website Meta Language will be simpler and it won't use deprecated modules. Oh! And <a href="http://blog.mozilla.com/blog/2010/01/21/firefox-3-6-release/">Firefox 3.6 (Gamma) is out</a>! I've been using the betas and RCs on Mandriva Cooker for some time now, and now I've upgraded to the stable version and it's still great. Shlomi Fish ( shlomif@iglu.org.il ) eumm wml language cmake perl firefox gnu autohell website meta cpan http://community.livejournal.com/shlomif_tech/43345.html http://community.livejournal.com/shlomif_tech/43345.html Fri, 22 Jan 2010 19:22:40 GMT Perl: Using Vim's snipMate for Perl 5 snippets http://community.livejournal.com/shlomif_tech/43262.html I noticed that there was some Perl 5 code that I had to type or copy-and-paste again and again on many occasions. So I decided to find a way to put it in one place and then recall it. I thought of writing it myself, but then recalled the <a href="http://www.vim.org/scripts/script.php?script_id=2540">snipMate snippets extension for Vim</a> (which <a href="http://www.catonmat.net/blog/vim-plugins-snipmate-vim/">Peteris Krumins covered in a blog post</a>), which allows that. After a little reading, I was able to prepare the snippets. I placed the following under <tt>~/.vim/snippets/perl/_slurp.snippet</tt> : <pre> sub _slurp { my $filename = shift; open my $in, "<", $filename or die "Cannot open '$filename' for slurping - $!"; local $/; my $contents = <$in>; close($in); return $contents; } </pre> And now I can type "_slurp<TAB>" to recall it. The existing "slurp" snippet in <tt>~/.vim/snippets/perl.snippets</tt> is quite evil, with an ugly inline line, typeglobs, and a two args open without a die statement. Thanks, but no thanks. Afterwards I added the following in <tt>~/.vim/snippets/perl/_ltestb.snippet</tt> <pre> local $Test::Builder::Level = $Test::Builder::Level + 1; </pre> And now I can type "_ltestb" (short for "local Test::Builder") and put this line there to create my own custom Test::More/Test::Builder tests. Thus, using snipMate, you can create your own short-hands for commonly-used snippets like that. Shlomi Fish ( shlomif@iglu.org.il ) test tip local slurp perl snipmate vim tip builder snippets vim http://community.livejournal.com/shlomif_tech/43262.html http://community.livejournal.com/shlomif_tech/43262.html Thu, 21 Jan 2010 16:26:13 GMT Report on the TelFOSS "Moose for Beginners" Meeting http://community.livejournal.com/shlomif_tech/42895.html Last Sunday, <a href="http://tel.foss.org.il/">Tel Aviv Open Source club (TelFOSS)</a> met to listen to Sawyer's presentation about "Moose for Beginners". Despite the fact that the meeting was well-publicised, very few people came. Sawyer did not prepare a lot of slides and since there were not many questions he finished quickly. After all that, he went to demonstrate how to implement several use cases that the audience has proposed as Moose programs, and despite that we still finished early. I asked about implementing an "XML parser", but in fact I wanted to implement a processor for a certain XML grammar. In any case, he misunderstood and told me to use Parse-RecDescent for that. I've grown to dislike P-RD for most stuff though. In any case, I'm getting tired of the low attendance in TelFOSS meetings. We do so much work in organising and publicising the meetings, and most people don't bother to come. Are the people too busy? Did they run out of steam? Why don't they come? This makes me frustrating and unwilling to further organise the TelFOSS meetings. Shlomi Fish ( shlomif@iglu.org.il ) telux telfoss meeting open perl club sawyer moose tel aviv source report http://community.livejournal.com/shlomif_tech/42895.html http://community.livejournal.com/shlomif_tech/42895.html Wed, 20 Jan 2010 12:35:04 GMT