There's a new essay on my site that discusses which programming language a non-programmer should start from. It has been brewing for quite a while, and now I'm pretty happy with it.

Thanks to the many people who read it and gave useful commentary. The licence is Creative Commons Attribution Licnce (CC-by) which means you can redistribute it, modify it, make derivative works and relicense it, as long as you give proper credit.

This was the first entry on the shlomif-tech community, and I hope you found it useful.

This is an IRC conversation that took place on Freenode. I am "rindolf".

   <bogdan>  rindolf: so you like being selfish, eh?
   <bogdan>  and you wanted to find a reason to be selfish.
  <rindolf>  bogdan: http://xrl.us/h2kn
  <rindolf>  bogdan: Rand made a mistake by naming her book "The virtue
             of Selfishness".
  <rindolf>  bogdan: she didn't mean it in the sense of "Complete
             disrespect for others".
  <spectei>  GPL > BSD
  <rindolf>  spectei: GPL is more selfish.
 <Dianora_>  BSD > GPL
  <rindolf>  Dianora_: :-)
 <Dianora_>  BSD is free, GPL is not free.
  <rindolf>  Dianora_: not you too.
  <rindolf>  spectei: the BSD is free, even Stallman thinks so.
 <Dianora_>  The GPL is like finding a coupon for $20 for pants on the
             street, the BSDL is like finding a $20 bill on the street.
 <Dianora_>  I would rather have the $20 bill
  <rindolf>  Dianora_: please lets agree to accept Stallman's definition
             of Freedom, and call BSD-style licensed software by another
             name.
  <rindolf>  Dianora_: I don't see the anlogy.
 <Dianora_>  even esr prefers the BSDL
  <rindolf>  Dianora_: yes, I know.
  <rindolf>  That's what I use for my software.
  <rindolf>  If I can help it.
 <Dianora_>  No Stallmans version of Freedom is a political
             mis-labelling of true freedom.
  <rindolf>  Dianora_: I'm not saying he defined freedom.
  <rindolf>  Dianora_: but he coined the term "free softwae
 <Dianora_>  He can try to relabel English anyway he wants, but he is
             not taking my language from me.
 <Dianora_>  BAH!
  <rindolf>  Dianora_: but different things mean different things in
             different contexts.
  <rindolf>  Dianora_: you can call BSD-style software PD-software or
             whatever.
  <rindolf>  Dianora_: they are nutjobs.
  <rindolf>  But they are good people.
  <rindolf>  They're on our side.
 <Dianora_>  BSD is FREE, GPL is a restrictive license, that is NOT free
          *  rindolf is a nutjob but tries to be a good person.
  <rindolf>  Dianora_: fine. The GPL is not free.
  <spectei>  it is free software!
  <rindolf>  "I want to be there, I want to be there".
  <pnbeast>  GPL is, in fact, a very non-restrictive license.
  <rindolf>  "fine you are there."
 <Dianora_>  No, it is NOT free software, the GPL is a restrictive
             license. period.
  <rindolf>  "danny! What are you doing here?"
  <spectei>  Dianora_, it is free software as defined by the FSF
  <rindolf>  "Here? <crying"
  <spectei>  :D
 <Dianora_>  it restricts your right to use that software!
  <rindolf>  "did I say something wrong?"
  <spectei>  Dianora_, in a good way!
  <rindolf>  It's from Sesame street.
 <Dianora_>  the FSF can label black as white and I will still not
             believe them!
  <rindolf>  Dianora_: the GPL is better than the MS EULA.
 <Dianora_>  rindolf yes ;-)
  <rindolf>  Dianora_: I've contributed to GPL software, and I cannot
             improve MSIE.
  <rindolf>  Or windows.
 <Dianora_>  don't get me wrong, I use GPL software, I write in a couple
             of GPL projects, but I disagree that GPL is free.
  <rindolf>  Or Excel.
  <rindolf>  Dianora_: like I said if we want to communicate we have to
             agree on a terminology.
 <Dianora_>  then the GPL is just that, the GPL a restrictive license.
  <pnbeast>  I won't argue "free", but I can't imagine how you think the
             GPL is restrictive. Can you explain that to me?
  <rindolf>  Most people accept the fact that free software is software
             by the FSF FS definition.
 <Dianora_>  no
 <Dianora_>  I am afraid you would be wrong
 <Dianora_>  available source code yes, but free? uh no. never.
  <rindolf>  Let's avoid the term free software.
  <rindolf>  Let's call it open-source.
 <Dianora_>  as I said, GPL is like a $20 coupon
  <rindolf>  Dianora_: do you think the GPL is not open-source?
 <Dianora_>  open-source is a good compromise.
  <rindolf>  Dianora_: ok.
  <rindolf>  Free software can also mean gratis software.
  <rindolf>  And the GPL is sourceware.
 <Dianora_>  you could have sourcecode that was not licensed to be
             changed or used by anyone else, but is available to be
             looked at
  <rindolf>  Dianora_: I hope you're not using qmail.
 <Dianora_>  in fact, everyones favourite whipping boy licenses his code
             that way.
 <Dianora_>  I'd use qmail if it was the best tool for the job, no
             matter what I thought of djb
  <spectei>  opensource doesn't mean anything!
 <Dianora_>  Disliking a person is no reason not to use someones code.
  <spectei>  haha
  <rindolf>  Dianora_: http://www.shlomifish.org/open-source/anti/qmail/
  <spectei>  dan bernstein is a wacko!
 <Dianora_>  djb does not allow his code to be modified ;-)
  <rindolf>  Dianora_: I find Theo objectionable but at least OpenBSD is
             open-source.
 <Dianora_>  naw, I will make up my own mind thankyouverymuch.
  <rindolf>  So I can take it , fork it, etc.
  <rindolf>  Dianora_: I cannot with DJB's software.
 <Dianora_>  OpenBSD is open-source *FREE* code
 <Dianora_>  *correct* both DJB and GPL are open-source but both are
             non-free!
  <rindolf>  Dianora_: and I'm not saying it just because I dislike DJB.
             Although due to the qmail licensing it is a problem.
  <spectei>  GPL is free :D
 <Dianora_>  gosh, you made my point for me. restricted licenses.
  <rindolf>  Dianora_: qmail is sourceware not opensource.
 <Dianora_>  naw rindolf
  <rindolf>  Dianora_: have you read the OSD.
  <rindolf>  ?
  <rindolf>  Or reveled source, or available source, or whatever.
 <Dianora_>  we both agreed that when you could look at the source, it
             was open-source no?
  <rindolf>  Dianora_: no.
  <rindolf>  Dianora_: open source is software that complies with OSD or
             the FSD.
  <rindolf>  Preferably both.
 <Dianora_>  but both GPL and DJB's license are restricted use. ;-)
  <pnbeast>  Come on, Dianora_, won't you define "restricted" for me?
 <Dianora_>  the OSD or FSD can call black white and I will still not
             use their definition. ;-)
  <rindolf>  Dianora_: yes, but the GPL is more usable than the DJB
             terms.
 <Dianora_>  pnbeast: you are defining it yourself.
  <rindolf>  Dianora_: I can fork GPLed code.
  <rindolf>  And I can link against an LGPLed library.
  <rindolf>  But I cannot fork DJB's code.
 <Dianora_>  rindolf: doesn't matter! it's still restricted!
  <rindolf>  Dianora_: it is restricted, yes.
 <Dianora_>  in fact
  <spectei>  Dianora_, bad things are restricted!
  <rindolf>  Dianora_: but it's not black and white.
  <pnbeast>  I guess that's a "no".
 <Dianora_>  the BSDL _is_ also restricted
  <spectei>  closed source proprietary software is bad
 <Dianora_>  pnbeast: You defined it yourself
  <spectei>  ergo it is good that it is restricted
  <spectei>  one can be free and not free to poison a well
 <Dianora_>  GPL poisons wells
  <spectei>  opposite!
  <spectei>  it keeps software free!
 <Dianora_>  GPL is a virus.
  <rindolf>  Dianora_: Windows < DJB < GPL < LGPL < BSD

I've already discussed the Ctrl-R commands in command line mode in this column. Well, I recently discovered that one can use Ctrl+R % there to enter the filename of the currently opened file. That way if you want to save it under a different filename (for example), you can type: :w Ctrl+R % and then edit it.

I started and finished reading the "Big Ball of Mud" article, which was recommended to me by a fellow Israeli Perl monger. It was interesting, but a bit disappointing, and I kind-of felt that it didn't teach me anything I hadn't already known.

I started going over the the Catalyst tutorial in the Catalyst-Manual section, in trying to learn Catalyst, which is a web development framework for Perl.

Today, I've also continued in reading Aristotle' "Categories", and reached the beginning of the third and final section. A while ago I gave up on it, after I saw that its beginnign contained a lot of "is predicated by" statements in the same sentences. However, later it becomes more understandable and quite enlightening. Afterwards, I'd like to read the rest of Aristotle's Organon.

Offline, I've been reading Ayn Rand's "The Virtue of Selfishness" (which I've been neglecting lately), "The Pragmatic Programmer", which I found to be very interesting, and I also returned to reading some parts of Feeling Good.

For a long time, I thought it was a KMail (= the KDE mail client) limitation that it couldn't cut and paste messages. Well, as of KMail 1.9.7 (and perhaps earlier) this was solved.

In the Edit menu, there is a Cut Messages (Alt+Ctrl+X), Copy Messages (Alt+Ctrl+C) and Paste Messages (Alt+Ctrl+V) options that can be used to do exactly that. That way one can easily move a few messages at once to a different KMail window or a different mail folder, without drag and drop.

One can use Ctrl+X in Vim to invoke several sub-commands for auto-completion. For example, "Ctrl+X ; Ctrl+L" completes entire line. So if you have a lot of my $self = shift; or my ($self, $args) = @_; in your Perl code you can start writing that line and press Ctrl+X;Ctrl+L to fill the rest of them.

Ctrl+X;Ctrl+F completes filenames, and there are other completions like that. I found it while reading usr_24 - inserting quickly which contains some other tips. I also realised that Ctrl+U cuts to the beginning of the line in vim editing, or in Emacs or GNU Readline.

Here's a vim tip on how to copy some non adjacent lines to a register ("r), but first the story that made me find out about how to do it exactly. I need to add full POD (Perl's documentation format) coverage to a Perl module I inherited. What it means is that every top-level function (or at least those that do not start with a "_") should have their own entries. In Perl the function definitions goes like that:

sub first_function {

}

sub second_function {

}

sub yet_another_function {

}

Etc. So I thought that I could look for all the lines that start with "sub " , collect them and append them to a register, where I'll paste them. I recalled the :g/re/$cmd that executes a command on all lines matching a certain regex, and the fact that I could append to a register instead of setting its value using a yank command.

So after a little research I tried: :g/^sub /"Fyy. :g searches for the /^sub / and invokes the command on every occurence, and "Fyy, appends the current line to the register "f". However, as it turns out :g executes an Ex command. However, fear not - after a little research, I recalled the :normal command that allows to execute normal mode commands from within the command-line (or Ex) mode.

So the final line that works is :g/^sub /normal "Fyy. You may need to do :let @f = '' before that to clear the "f" register.

Cheers.

A few days ago I ran into some problems when I tried to install and use Common Lisp's iterate construct for my Mandriva SBCL (= Steel Bank Common Lisp) installation. Luckily, after I consulted the wisdom of Freenode's #lisp channel, and an IRC buddy of mine (thanks!), I was able to get it working. With some variations it might be useful for other Common Lisp APIs or Lisp implementation.

So here is how to do it:

  1. Download iterate using darcs using the instructions on this page.
  2. Add the line:

    (require 'asdf)
    

    at the top of your .sbclrc file, in case it's not already there. Then, add this line somewhere after it:

    (push #P"/home/shlomi/Download/iterate/" asdf:*central-registry*)
    

    /home/shlomi/Download/iterate/ should be the path where you downloaded iterate.

  3. To use iterate from your Lisp script write the following at the top:

    (asdf:oos 'asdf:load-op :iterate)
    (use-package :iterate)
    

    (As I discovered the colon in (use-package :iterate) is significant.)

    Then you can write this to test it:

    (iter (for i from 1 to 10)
          (format t "Hello No. ~A~%" i))
    

This should hopefully work. Please comment here if you still have problems.

Here are some of the things I've been doing lately. Recently, I've been having a spell of tiredness and lacked the will to do anything productive. But lately, it got somewhat better and I was able to work on some productive tasks.

I've been learning the Prolog programming language. I started with this tutorial, but was dismayed from the fact it did not have exercises involving writing code, so I switched to learning from this tutorial.

I've also been doing some of the tasks from Project Euler.

I originally had written a Perl script to combine the feeds of my blogs for my aggregated RSS feeds. Originally this script executed an aggregator program for every feed, which fetched the feeds from the network every time. So a few days ago, I made sure the feeds are fetched only once, and then are combined and outputted into the files, using one program.

However, later on, I also decided to do hard-disk cleanup, and accidently deleted it. I restored it from the backup, but it was the copy before the improvements. So today, I set to reproduce my actions (this time using version control), and I think it went quicker this time.

When I gave my presentation about the Joel Test to the Israeli Pythoneers, I noticed that one problem was that all the points appeared at once. So starting from a few days ago, I customised the slidy.js JavaScript code from HTML Slidy and adapted it to my needs for the presentation, to display the points one by one. I removed most of the functionality there and changed a lot of the rest, but with my shallow control of JavaScript it was good to start from a working code. Firebug was a lot of help in getting to the bottom of the bugs I encountered.

Finally, I placed Darien's "Everybody's Free (to Ping Timeout) online on my homepage, after I extracted it from my web-feed aggregator. It disappeared from the Net after the demise of Darien's blog.

For a long time, I had many hangups with the open-source Firefox browser, which happened all the time, but I could not reproduce. Two weeks ago, however, I noticed that when I was using YouTube, after I opened a window with a new video, and closed it, then Firefox freezed. I tried to reproduce it and was consistently able to do so. So I reported it.

Then, after I tried to reproduce the bug in a new profile, I could not reproduce the bug. So per Steve England's advice, I tried to find out which extension is causing this problem, by disabling and re-enabling them. After a lot of binary search, I found out that I could only reproduce the problem using three extensions enabled in the original profile: Firebug, Greasemonkey and "Split Browser".

Then Oded Arbel said he could not reproduce the problem on a clean install using my current description. And indeed I could not reproduce it on a fresh profile either. So there were also some problematic preferences.

I set out to discover it. My most likely suspect was something in the prefs.js file, so I copied it to the new profile, and it indeed caused the problem to re-surface there. Then I gradually removed more and more initial settings from there, to find out the problematic ones. Each time I did a change, I had to try and reproduce the bug. I kept some -bad-$idx and -good-$idx files as a poor man's version control. After spending some time on it, I found out that the problem was with a single preference: extensions.firebug.disabledAlways. So what was needed was installing the three extensions and toggling this preference.

Then using this information I was able to reproduce the problem on my Windows XP machine.

This was a hard bug to properly isolate, and it now seems very pedantic. But it may indicate a more general problem with Firefox. I'll be grateful if you can try to reproduce it on your Firefox setup (in a fresh profile, of course).

I wrote a patch to implement a search dialog for the Linux kernel "make xconfig" configuration applet, and sent it to the Linux kernel mailing list (LKML) back in January. I did it because I got bit by the lack of a search facility there and wanted to improve it. As it turned out, by the time I submitted the patch a search dialog was already implemented, but it was placed in the wrong place in the menus, and was quite inadequate.

So I wrote a patch to relocate the menu item to "Edit" → Find, and it got accepted into the main kernel. Then I neglected working on it.

About a week ago, I started working on it again. I enhanced the find dialog to have substring search as well as regular expression search, and made it search through the description strings. Afterwards, I implemented keyword search.

When trying to submit this patch to LKML, I discovered that my address was "not liked source for email", at least according to the bounce. (This is orobably due to this email.) I re-submitted my first patch using the shlomif-lkml@iglu.org.il address, which would have worked if iglu.org.il was still on qmail, but since the domain was moved to a server running postfix, caused some bounces for people who tried to reply to me in private. Aaarrrggh!

The second version of the patch (almost final, as far as I'm concerned) was sent from my original address, and CCed to two relevant developers, and bounced again. I hope this issue will be addressed after the Linux kernel summit (and posssibly linux.conf.eu) are over.

In any case, I placed the patches I'm working on, in a temporary location until they get applied. Enjoy!

There's a new petition to release MainActor, a non-linear video editing software under an open-source software, as it will be discontinued. See also this opendotdotdot post.

Please sign the petition and help spread the word.

I should note that when the vote is submitted, you end up at the donation page, not at the list of votes. Don't worry - your vote was registered.

This post will explain how to over-ride (or overwrite or replace) a global JavaScript function found on a page with a completely new one using Greasemonkey. Also see this greasemonkey-users discussion where I originally asked the question.

To do that write a global JavaScript function in the Greasemonkey script (ending with .user.js) with the same name as the page-defined JavaScript function that you wish to override, and do the following:

function PageFunction() {
        // SNIPPED
}

function embedFunction(s) {
document.body.appendChild(document.createElement('script')).innerHTML=s.toString().replace(/([\s\S]*?return;)
{2}([\s\S]*)}/,'$2');
}

embedFunction(PageFunction);

embedFunction() is taken from the "Embed a function in the Current Page" snippet from the Greasemonkey wiki.

Happy Greasemonkeying!

STAF stands for "Software Testing Automation Framework" and is a framework for IBM for software testing. I spent a large amount of the last two days trying to get it up and running on my Mandriva Cooker Linux system, in order to fullfill this request for a Linux beta-tester.

Trying to get the binary to work with my Perl failed due to a segfault when running it. So I opted to build it for source. I followed the STAF developer's guide which explains how to build STAF, but still needed a lot of trial and error.

Eventually, I installed ActivePerl-5.8 and used the following shell script, that needs to be sourced (or "."ed) into the current shell, to build everything. You may need to customise it a little.

#!/bin/bash
# One needs staf to build the main things, connprov* to be able to
# run STAFProc on the local machine and perl because I needed it.
export OS_NAME=linux BUILD_TYPE=debug PROJECTS="staf connprov* perl"
export JAVA_LIBS="/usr/java/jdk1.5.0_09/" JAVA_VERSION="1.5"
export JAVA_V12_ROOT="/usr/java/jdk1.5.0_09/" JAVA_V12_LIBDIRS="/usr/java/jdk1.5.0_09/libs/"
export PERL_BUILD_V58=1
export PERL_BUILD_V56=0
export PERL_V58_ROOT="/opt/ActivePerl-5.8"
export PATH="/opt/ActivePerl-5.8/bin:$PATH"
PLIB="$PERL_V58_ROOT/lib/"
A="$PLIB/CORE"
export PERL_V58_INCLUDEDIRS="$A" PERL_V58_LIBDIRS="$A"
# This variable is completely undocumented in the relevant part of the
# STAF documentation, but with its default value it won't work correctly.
export PERL_V58_TYPEMAP="$PLIB/ExtUtils"

After doing it, type "make". Cheers. Now I can go on with my business.

One day, I woke up and started using my Mandriva Linux Cooker system which I left on overnight, and noticed that I couldn't access the Internet host hosting my homepage (which also hosts my Email account). No POP, no HTTP - nothing. I couldn't find any way to restore access there except reboot. Eventually, the problem repeated itself a few times, each time involving a reboot.

Eventually, I reported my findings to Linux-IL, but couldn't get much help there. I asked netdev@vger.kernel.org for help, and was told the problem was with the TCP window scaling problem, but doing "echo 0 > /proc/sys/net/ipv4/tcp_window_scaling" after the problem appeared didn't solve the problem after at least 30 minutes.

Here's what I know so far about this problem:

  1. The IP that causes the problems is 212.143.218.31.
  2. It is exhibited by kernels 2.6.23, 2.6.24-rc1 and 2.6.24-rc2. (At least)
  3. A different computer on the same Home LAN connected via a NAT/router has no problem with that IP, at the same time the Linux installation exhibits the problem.
  4. During one time this happened, I could connect using telnet to port 80 eventualy, but it took an awfully long time.
  5. I have problem with both HTTP and port 80 and POP.
  6. Restarting the network ("/etc/init.d/network restart") does not help - only a reboot.
  7. The network as a whole (Google, etc.) works fine.
  8. Doing "echo 0 > /proc/sys/net/ipv4/tcp_window_scaling" after the problem appeared didn't solve the problem after at least 30 minutes.

Now, in trying to resolve this problem, I decided to switch to my distribution's kernel so I could at least report it in its bug tracker. So I tried to boot my system using the new kernel and it wouldn't boot. After a lot of experimenting and no progress, I decided to install Mandriva again on a new kernel, so I can copy the new kernel settings and initrd. It couldn't boot either.

So I filed a bug to the Mandriva bug-tracker, and have done some further investigation. Apparently, a configuration I created for the Linus kernel, which is similar to the Mandriva config, could not boot the Linus kernel either. On the other hand, my old config did work. So I started bisecting, and eventually reached this conclusion, where I have a working configuration that has some stuff as "Y", and a non-working configuration that resembles the Mandriva one more (that was created by toggling a "Y" option into "M" in "make xconfig").

Bug-investigation is often kind-of like serendipity (or rather Whack-a-mole), where one discovers bugs that prevent one from further investigating the previous bugs. At the moment, I'm still using the vanilla kernel, and am still getting the networking problem that forces me to reboot the computer more often than I'd like to.

If you can shed some light on any of these problems I mentioned, please comment below - any help would be appreciated. Until then, happy, and hopefully bug-free, computing.

GIMP and GEGL

I've recently decided that I'd like to give my life purpose again by working on a large project. I decided to work on GIMP again, joined its channel, and asked how to help in porting it to the GEGL and Cairo libraries. I was told that the HOWTO just involved learning the APIs of these programs and figuring it out for yourself. So I started reading on GEGL. As I read the GEGL homepage and API documentation, I discovered some errors and mis-phrasings, which I reported and are now fixed in the trunk (either by a different committer or by me.).

Next, I realised that GEGL did not have any automated tests, and decided to work on that. For that I'd like to use Ruby and TAP - the Test Anything Protocol. I have to resort to using Ruby instead of Perl because GEGL does not have Perl bindings yet.

Archive-Zip

I recently asked Adam Kennedy for a co-maintainership status for the Archive-Zip CPAN module (so I can close its bug reports) and he kindly granted it. This enabled me to fix and close several of its many bugs. It was sometimes time consuming because the bugs sometimes did not have regression tests. I had to write them myself and make sure they fail before patching and succeed afterwards.

If you're a software enthusiast and reading it, please take note for this: when reporting a bug against a program, please give a full reproduction recipe, and very preferably, a patch containing an automated regression test. Otherwise, you're wasting the developer's time who has to interrogate you, figure out the missing data or code, and why it failed for you. So please, be kind to us, and show us the testcase!

Test-Count

While resolving an Archive-Zip bug, I had to heavily rework a test that someone sent me. Now I had to execute a group of assertions once normally and once with a Perl global variable ("$\" if you must know) set. So I used the Test-Count, $n=0 and then $n=$n+1 paradigm (and finally added two $n to the test count) in order to count the tests. But it didn't work, and the script exited with a weird error. As it turned out when retrieving the value of the variable, I checked for truth instead of whether it's not undef(), and so failed. So I had to take a break from working on Archive-Zip and fix Test-Count in this regard.

My Test-Count testcase, then failed again, and as it turned out it happened because assignments were performed twice due to a weird parsing problem, which I also fixed. After all that was done, I uploaded Test-Count-0.0103 to the CPAN, and got a bootload of failures for it. Turns out that the specification of the dependent modules was wrong, and I fixed it and released Test-Count-0.0104, which passed 27 times so far, while generating one "NA" with perl 5.005 for which it's not intended anyway.

Nouveau

Nouveau is an ongoing project to create open-source UNIX drivers for Nvidia cards. I've been trying to consistently use nouveau instead of the properietary "nvidia" or the limited "nv" driver, but often encountered some problems that drove me back to "nv".

The last time it happened was about a month ago when x11-server-1.4.0 was released and caused various random rendering problems when used with Nouveau, which were too annoying to stand. It also had other non-Nouveau-related bugs, but these were mostly corrected by vendor patches.

So I switched to "nv" again. A few days ago, I've visited the #nouveau channel on IRC, and was told to try Nouveau again. So I downloaded, compiled and installed the latest Nouveau and restarted X with it, and it seems to work fine. The rendering problems are gone, and as a bonus from previous versions of x11-server, I'm no longer getting the visible wallpaper tile-by-tile-redrawing that I experienced previously.

One thing I notice is that when using Nouveau, I can still exhibit this bug, which exists in "nv" as well. (So it doesn't matter too much that I'm using nouveau, in this regard.)

So nouveau works again, and I can happily beta-test it by using it daily.

Ben Collins-Sussman (one of the Subversion developers) wrote an insightful article titled "Version Control and 'the 80%'" on his blog. It received many comments and I wanted to comment on it, too, but:

  1. His blog ate my comment.
  2. When I tried to post it again, it refused to, claiming a similar comment has already been posted.

Given I have no choice, I'll post this comment here, and allow Ben to respond to it. Here it is, with all of its glory:


Nice article. However, are you sure most developers are working on Windows? Not according to the article which I referenced in this Joel-on-Software discussion:

"Linux is becoming the development platform of choice; for example, it is expected that more developers will be developing on Linux than Windows by 2004;"

Also can you please add a "Preview Comment" button? I am unable to tell if my comment is good or not because the only button I have is "Submit Comment".


I should note that I can also recommend Ben's "Welcome to the Food Chain" entry, which isn't technical.

I've been meaning to try KDE 4 for a while now, and finally decided to do it after RC 1 (The first Release Candidate) was announced lately. I decided to install it from the Mandriva KDE 4 RPMs. As I discovered by reading "The Running KDE 4 on Mandriva 2008" linux-tip article, installing it involved typing "urpmi task-kde4". It failed to install the kdebase4-workspace package and I had to fix and report a bug of it (which has since been resolved in cooker), but after installing the fixed package, and doing "urpmi task-kde4" again, I had KDE 4 installed. By then it was too late in the night (past 1 AM) and I decided to go to sleep.

Next morning I looked for a way to start KDE from a virtual console, as I was not using a graphical run-level (with kdm/gdm/etc.). The article did not mention how to do it, and after a little research, I found out that it can be done by typing "kde4" at the virtual console bash command line. This will set up the environment apprporiately and start an X session running KDE 4.

So I had KDE 4 running. Here are my initials impressions:

  1. The decorations of the inactive windows were light gray, which made them hard to distinguish. I could not find a way to change it. (I was told on IRC that it is work in progress.)
  2. There did not seem to be any support for virtual desktops, and I was also told it is work in progress.
  3. In Konsole, there was no "New session" button at its tabs bar to start a new tab. That seems to be a developer decision, and I discovered that I could still start a new tab from the menus, by pressing the keyboard shortcut, or by double-clicking on an empty space in the tabs bar.
  4. There doesn't seem to be a way to drag items from the main menu into the desktop or the quick-launch bar (which doesn't seem to exist) for easy access.

In short, it's not ready for prime time, and I could not find it usable for my day-to-day-use. If it's a release candidate, then I'm a Jackalope.

See also Diego Iastrubni's recent review of KDE 4 (in English, and with some comments).

I started working at a certain workplace downtown, at least temporarily. There I was instructed to set up a Fedora 8 system (Fedora, being the Red Hat-endorsed community distribution) with some networking services, including SMTP (Internet Mail). The installation and post-install configuration of Fedora there, went pretty flawlessly, but then I encountered an annoying bug when trying to monitor the SMTP traffic. Apparently, when using Wireshark the recorded SMTP conversations on eth0 only have one side of the conversation, with the other side completely removed. Here's what I found out about it:

  1. HTTP conversations are recorded fine.
  2. It still happens with iptables being cancelled completely (all ACCEPT), and with SELinux set to permissive.
  3. It happens with all current Fedora 8 updates, including the most up-to-date kernel, applied.
  4. It happens with the Wireshark filter set to empty ones, which should accept all traffic.
  5. It happens with both the Fedora supplied Wireshark, and a Wireshark that I compiled from source and installed under a different prefix.
  6. The traffic originates from a remote computer on the same Ethernet network. putty which is used for the conversation, indicates that the SMTP server has properly received the messages I sent it, and that packets were sent in both directions.

Now I've tried to reproduce the problem here on my home machine, so I installed Fedora 8. At home, however, everything works perfectly. Grrrr...

This has been driving me crazy because it makes it much more difficult to debug the SMTP service. If anyone has any idea to what could be wrong, please comment here.

This has given me a chance to try Fedora again. The first problem I noticed (at home - at work it was OK) was that I didn't have external networking. Apparently, the /etc/init.d/network script was not placed in the rc5.d scripts for some reason. Running /etc/init.d/network start from the command line and using system-control-services fixed it.

I noticed that I could mount XFS partitions, and was able to configure the same /home partition (with the same user) as my Mandriva installation. However, using the Mandriva configuration, Konqueror would just display the code of an HTML page, instead of rendering it. Plus, when I booted the Mandriva system again, I noticed that my KDE screen was a little messed up, and I had to restore its configuration directory from the backup I prepared before installing Fedora. I guess $HOME/.kde/share/config was not intended to be shared like that between Mandriva and Fedora.

One thing that has improved is the support for .mp3's in Amarok. I recall being completely unable to play them properly there using the xine backend, but now it worked after I installed the relevant package as specified in the Fedora FAQ.

At home, I installed Fedora 8 from the Fedora Live KDE CD, so it is possible some of the problems I encountered will not be encountered if one uses the installation DVD.

I've been meaning to try out the Archlinux Linux distribution for a long time now. During the previous weekend I finally dedicated some time to installing it. Here are my first impressions.

The first thing I noticed was that the installation procedure is rather text based and not very friendly. I didn't want to risk it formatting my common /home partition so I eventually opted to configure it after the installation. I had to run the installer twice in order to set a bootloader, so I can tell how to boot Archlinux properly using the bootloader on the Mandriva parition. I wanted to configure the Mandriva bootloader using the files on the Arch partition but couldn't figure out where is the kernel and initrd. Apparently, it keeps the kernel under /boot/vmlinuz26 and the initrd under kernel26.img. Moreover, the mkinitrd program is called mkinitcpio there.

After being able to boot arch, I noticed that my networking was not configured properly, despite the fact I use DHCP for this computer. I opted to configure it manually using ifconfig and route, but I was later told I can add eth0=(dhcp) in the appropriate place in /etc/rc.conf. This would enable DHCP after you do /etc/rc.d/network restart.

Arch also configured my clock improperly and I had to set the timezone and clock settings manually:

HARDWARECLOCK="UTC"
TIMEZONE="Asia/Tel_Aviv"

The time was still wrong, and remained so until I installed and ran ntpdate manually. The documentation on the NTP protocol on the wiki is confusing, and I have yet to figure out how to do it on system startup.

Then it took me a long time to figure out Pacman, the Archlinux package manager. As I was later told, it has a good cheat-sheet on the Archlinux wiki. If you're going to install Arch, I suggest reading it and then keeping it handy on the hard disk in a plain text, because you'll need it, and without it you won't be able to install anything.

I needed the help of the #archlinux IRC channel on Freenode on how to install X and a usable environment. You can install it using pacman -S xorg, and then pacman -S xf86-video-nv (or whatever is your video card), and then pacman -S icewm to install IceWM (or whatever). To install KDE you are better off configuring and using KDEmod than the built-in KDE and there are instructions how to do it on their site.

I still don't know how to do a few things in Archlinux. One is how to search for files inside remote packages (like urpmf on Mandriva or apt-file on Debian/Ubuntu). And it seems that mplayer, xine (and thus - Amarok) don't support playing Module files, because support for them has been configured out during compile time, which is a bit annoying.

I was able to share my /home partition and my user between Mandriva and Arch pretty easily. My KDE exhibited a few problems on Mandriva, after using it with Arch, so it may not be very recommended to do so. Archlinux keeps KDE under /opt/kde, which could cause incompatibilies with the Mandriva /usr-bound KDE.

As a general rule, Archlinux gave me a rather "raw" and non-user-friendly feeling, so I didn't end up using it a lot after the installation. I wouldn't recommend it for beginners because hardly anything works there out of the box. Still, installing it was a good experience so I can see things being done a little differently than I'm used to.

This is a reminder that Israeli Perl Workshop for 2007 will take place next week on Monday, 31 Decemeber 2007. If you're planning to attend, please register at the site, and make the payment.

I already registered and am planning to be there, and I'll hope to see any Perl enthusiasts I know, and those who would like to meet me, there too.

Happy new year!

The ~/.ssh/config accepts the Host meta-configuration that enables one to specify directives for only a specific host. With the HostName directigve, and the User directive this allows defining shortcuts and defaults. For example, this portion out of my file:

Host berlios

HostName shell.berlios.de
User shlomif

Specifies that by doing "ssh berlios" I'll login to shell.berlios.de as the user shlomif (in equivalence to ssh shlomif@shell.berlios.de).

One use I can think of for this feature is to define a common destination, with variable remote usernames, in uploading scripts. Until now, I've used something like: ${IGLU_USERNAME:-$(whoami)}@iglu.org.il:, which retrieved the value of the $IGLU_USERNAME and defaulted to the local user. However, now one can simply use iglu.org.il: or mytask.iglu.org.il: and define the appropriate user in the ssh config file.

Credit is due to a blog post by Diego Iastrubni, where he blogged about this feature. Thanks, Diego!

I found the following projects which I found of interest to me by reading the new Freshmeat.net releases. I'd like to share them here:

  1. GIMP Scripts that work in GIMP-2.4 - forward-ported scripts for GIMP-2.4.
  2. An Open-Source Search Engine for Subversion Repositories - seems to be very useful in some contexts. I submitted a patch to add it to the Subversion links' page.
  3. VisionProject - a non-free issue tracking/project collaboration tool with Subversion integration. It may also be appropriate to add it to the links page.
  4. Yudit - a Unicode text editor for X-Windows - a Unicode text editor with Bi-directionality support that I seem to have missed. When trying to input Hebrew with it, I'm getting the 4-digit-hex-representations and not the letters themselves - I'm probably missing a font. Furthermore, my compose key doesn't work there and so I cannot input accented characters. It has no menu - only a toolbar, and the right mouse button does the same operation as the left mouse button.

    Still, after I sort out all the problems with it, it may prove as a good alternative to KDE's KEdit and to Katoob.

  5. WP Clipart is a public domain, bitmap-based clipart that looks interesting.
  6. BurgerSpace is a clone of a game I used to play on DOS. Trying to play it with the version in Mandriva Cooker, I found that after a while I could no longer throw pepper to attack the enemies, even after I died. But I didn't know if it was a bug or a feature.
  7. CrissCross is another C++ portability library
  8. Deliantra is an open-source Multi-player Online Role-Playing Game (MORPG). One interesting fact about it is that it's partly written in Perl. I did not test it yet because I'm not using the proprietary nvidia drivers.
  9. Another MMORPG is The Mana World, which doesn't require OpenGL or 3-D acceleration. I temporarily gave up on installing it after running into one of its dependencies.

Enjoy!

In continuation to my previous entry about Amarok and Last.fm integration, I found this bug entry regarding the main problem with it that bothers me, and I commented about it and re-opened it. A net-friend who checked it got different results, and commented as well.

One thing I noticed about the Last.fm player is that it allows one to play any arbitrary Last.fm tag, while Amarok only has a set of pre-defined tags for common music genres in its GUI. However, after experimenting with "dcop" I found out one can enqueue any arbitrary Last.fm tag using the following shell script:

#!/bin/bash
tag="$1"
shift
dcop amarok playlist addMedia "lastfm://globaltags/${tag}"

To use it save as "lasttfm-amarok-tag", make it executable (chmod +x) and do ./lastfm-amarok-tag "new age" to play the "new age" tag. You can also type a command line like dcop amarok playlist addMedia "lastfm://globaltags/reggae" directly from the command line, but that's not very convenient.

And finally, here's a recommendation for a puzzle game I saw mentioned on the Free Gamer blog: Hex-a-Hop. I was able to easily compile it on my Mandriva Cooker system, and played it, and it's very nice.

This story has two lessons. The first is that I'm an idiot, but I guess we all already knew that. I'm not going to tell the second one, but it will be available under a headline, in case you want to skip. But let's start with the beginning.

The Bug

It's all started when I did my backup yesterday, converted it into a CD-ROM image (an .iso file) and so used growisofs. But when running it, it completely hogged my system, causing the Amarok music playback to be flaky, and making everything more sluggish. It didn't happen before, so after reporting this bug to the Linux-IL mailing list, I decided to investigate.

I went on the IRC's #kernelnewbies channel, and someone told me that since it worked before, I should start from a working kernel and bisect it. And indeed trying the 2.6.23 kernel made the problem disappear. So I decided to bisect the kernels in between to try where the problem occured first.

Bisecting using git

The history of the Linux kernel is versioned using the git version control system. Searching for "git kernel" yielded the "Kernel Hackers' Guide to Git", which I've skimmed, and found that I should start by typing this in the command line:

git-clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6 in the command line.

Note that one cannot effectively interrupt this command without needing to start at the beginning, so be prepared to wait a while. (I had to wait for it to finish before I could reboot to test another kernel).

Similarly one can use git-clone linux-2.6 new-lin-2.6 to clone the repository on the hard-disk, which also proved to be useful in order to build the latest kernel I used to see if I could still reproduce the problem there.

Then I was told to used git bisect to bisect the kernel. (The "lion in the desert" method). Read the link for usage information. I copied my /boot/config-2.6.23 kernel configuration file to .config in the top-kernel-dir, and ran "make oldconfig" on it. From then on, I used "make oldconfig" on the new kernels, while pressing Enter all the time to accept the defaults (also doable with yes "" | make oldconfig).

This way, I built one kernel after another while rebooting to test each one. As I discovered in the middle of the 2.6.24, something went wrong, and one could no longer run "make modules" in the kernel source tree, so I often had to reboot to a safer kernel. This also slowed me down, but it was resolved since before -rc6.

All Good is No Good

As it turned out, every kernel I built except the 2.6.24-rc6 kernel which exhibited the bug in the first place, did not exhibit the bug (and marked by me and git as "good"). Eventually I ended up with the last patchset reported to be the problem, but it only contained a change of the EXTRAVERSION from "-rc5" to "-rc6" in the top-level makefile. I couldn't understand why it should make so much difference.

So what was the problem? As I discovered later, the v2.6.23 configuration, on which the configurations of most of the kernels I built were based, was slightly different than the one I had for v2.6.24-rc6, and this is what made it work. I originally made the changes to investigate a different bug (where I tried to boot my system using a kernel with a configuration similar to the Mandriva one), and they persisted in the further kernels.

After changing the kernels (including the old ones) to the new configuration, I was able to reproduce the problem in all of them. So it was a bug with my kernel configurations and not with the history.

The Second Lesson of this Story

So the second lesson is that if you're going to bisect kernels make sure you have the same configuration or one that is derived from the other one. Otherwise, the kernels may behave differently.

What I Achieved

So what did I achieve?

  1. I learned more about the git version control system, and how to use it. I could use the knowledge in the future.
  2. I now have 32 entries in my boot-loader menu.
  3. I rebooted my machine 25 times. That's the most I've done in such a short period since I've encountered the problems with my ReiserFS partition back when I started using Linux Mandrake (now Mandriva Linux) 7.1.
  4. I was pratically neutralised throughout all this time. I was able to send some emails, and chat a bit on the IRC, but I was mostly waiting for kernels to finish compiling.
  5. I discovered that this kernel configuration diff has been the root of the problem

    --- config-ok	2008-01-07 20:53:30.698008159 +0200
    +++ config-not-ok	2008-01-07 23:13:17.894346993 +0200
    @@ -1,7 +1,7 @@
     #
     # Automatically generated make config: don't edit
    -# Linux kernel version: 2.6.24-rc5
    -# Mon Jan  7 16:37:58 2008
    +# Linux kernel version: 2.6.24-rc6
    +# Mon Jan  7 13:19:02 2008
     #
     # CONFIG_64BIT is not set
     CONFIG_X86_32=y
    @@ -621,34 +621,31 @@
     CONFIG_BLK_DEV_RZ1000=m
     CONFIG_BLK_DEV_IDEDMA_PCI=y
     CONFIG_BLK_DEV_AEC62XX=m
    -CONFIG_BLK_DEV_ALI15X3=y
    -# CONFIG_WDC_ALI15X3 is not set
    -CONFIG_BLK_DEV_AMD74XX=y
    +# CONFIG_BLK_DEV_ALI15X3 is not set
    +# CONFIG_BLK_DEV_AMD74XX is not set
     # CONFIG_BLK_DEV_ATIIXP is not set
    -CONFIG_BLK_DEV_CMD64X=y
    +# CONFIG_BLK_DEV_CMD64X is not set
     CONFIG_BLK_DEV_TRIFLEX=m
     # CONFIG_BLK_DEV_CY82C693 is not set
     # CONFIG_BLK_DEV_CS5520 is not set
     CONFIG_BLK_DEV_CS5530=m
     # CONFIG_BLK_DEV_CS5535 is not set
    -CONFIG_BLK_DEV_HPT34X=y
    -# CONFIG_HPT34X_AUTODMA is not set
    -CONFIG_BLK_DEV_HPT366=y
    +# CONFIG_BLK_DEV_HPT34X is not set
    +# CONFIG_BLK_DEV_HPT366 is not set
     # CONFIG_BLK_DEV_JMICRON is not set
     CONFIG_BLK_DEV_SC1200=m
    -CONFIG_BLK_DEV_PIIX=y
    +CONFIG_BLK_DEV_PIIX=m
     # CONFIG_BLK_DEV_IT8213 is not set
     # CONFIG_BLK_DEV_IT821X is not set
     # CONFIG_BLK_DEV_NS87415 is not set
    -CONFIG_BLK_DEV_PDC202XX_OLD=y
    -# CONFIG_PDC202XX_BURST is not set
    -CONFIG_BLK_DEV_PDC202XX_NEW=y
    -CONFIG_BLK_DEV_SVWKS=y
    -CONFIG_BLK_DEV_SIIMAGE=y
    -CONFIG_BLK_DEV_SIS5513=y
    +# CONFIG_BLK_DEV_PDC202XX_OLD is not set
    +# CONFIG_BLK_DEV_PDC202XX_NEW is not set
    +# CONFIG_BLK_DEV_SVWKS is not set
    +# CONFIG_BLK_DEV_SIIMAGE is not set
    +# CONFIG_BLK_DEV_SIS5513 is not set
     CONFIG_BLK_DEV_SLC90E66=m
     CONFIG_BLK_DEV_TRM290=m
    -CONFIG_BLK_DEV_VIA82CXXX=y
    +# CONFIG_BLK_DEV_VIA82CXXX is not set
     # CONFIG_BLK_DEV_TC86C001 is not set
     # CONFIG_IDE_ARM is not set
    
  6. Hopefully I found a way to overcome the bug. Now that I've upgraded to kernel version 2.6.24-rc7 (which was released today), I based my configuration on the one that doesn't exhibit the bug, and will test it to see if that's indeed the case.
  7. And like I said earlier, I realised I am an idiot, which is the first step on the path to enlightenment.

So mission accomplished? I guess so. Paul Graham said in What You'll Wish You'd Known that one is happy when being relieved after overcoming worries, and I feel very much relieved now, so I guess I'm happy. I also feel a kind-of-calmness.

Here's to a bug-less and flaw-less use of your favourite open-source operating system. Well, at least we can hope.

In order to play sound / audio on your Mandriva (and possibly other Linux flavours) system, after being logged in an X session with one user, and you've su'ed to a different (also possibly under-privileged) user in the shell, do the following. Just add all the users that should be able to play sound at all times to the "audio" UNIX group in the "/etc/group" file (as root, of course). Then, you may have to run "sg" or "newgrp" for these changes to take effect globally.

Then you can play sound as all the users you've su'ed to that belong to the "audio" group. Also see this thread in the Mandriva Cooker mailing list, where I asked the question originally.

And a more obscure and less useful tip: to exclude dependencies ("requires") on different RPMs (that are calculated implicitly) when editing an Mandriva SPEC use a statement like that:

%define _requires_exceptions /bin/zsh /bin/csh

There's an example of this in the Mandriva xemacs spec.

As a follow-up to my previous post about a networking problem I've encountered on my Linux box I'd like to make a follow-up and re-summarise all the currently available information.

Someone suggested the problem may be caused due to a bad Ethernet card, so I borrowed an Intel Ethernet card, replaced my Ethernet card with it, and tried it for a few days. After keeping the computer on for a while, it started exhibiting similar problems as the old one: no connectivity to the host of www.shlomifish.org, bad connectivity to Google, etc. Solvable only after a reboot.

I've uploaded Wireshark pcap dumps of a "lynx http://www.shlomifish.org/" command from both cards before and after the problem is exhibited.

So here's what I know now:

  1. The symptom is that after the computer is on for a while (two days or so), one cannot connect using TCP to some hosts, and the connection times out.
  2. The IP that causes the problems is 212.143.218.31, but it also affects www.google.com and possibly other hosts.
  3. It is exhibited by kernels 2.6.23, 2.6.24-rc1 and 2.6.24-rc2 and 2.6.24-rc8. (At least)
  4. A different computer on the same Home LAN connected via a NAT/router has no problem with that IP, at the same time the machine running Linux exhibits the problem.
  5. During one time this happened, I could connect using telnet to port 80 eventualy, but it took an awfully long time.
  6. I have problem with both HTTP and port 80, POP and SSH.
  7. Restarting the network ("/etc/init.d/network restart") does not help - only a reboot.
  8. Some other hosts in the network, like Yahoo work fine.
  9. Doing "echo 0 > /proc/sys/net/ipv4/tcp_window_scaling" after the problem appeared didn't solve the problem after at least 30 minutes.
  10. The problem is exhibited by both a RealTek card I have and an Intel Ethernet card. (both 100 Mbps).
  11. pcap dumps of an HTTP connection to the offending site before and after the problem occurs are available for both cards.
  12. I can't ping www.shlomifish.org, because it doesn't answer to pings at all, but when the problem occurs again, I can try pinging www.google.com and see what happens.
  13. iptables is completely off:

    [root@telaviv1 ~]# iptables -L
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    
  14. My household is connected to the Internet through a Sweex NAT/Router that doesn't have any updates yet.
  15. Quoting srlamb:

    The difference in these two packet captures seems to be that in the "bad" case, the www.shlomifish.org->you packets have a bad checksum and are ignored (see it retrying the SYN after it's already gotten a SYN/ACK?). I don't know why this would be, but it seems worthy of mention to the mailing lists you are asking for help.

Some extra discussion can be found at the previous entry.

This evening, after I returned from work, I decided to further my studies of unknown programming languages. The first thing I tried to do was to play the Practical Common Lisp video that I downloaded. However, neither mplayer nor xine could play it due to a missing codec. It worked fine a few days ago, so I don't know what's wrong yet (but I intend to find out in the future).

Then I decided to go over Pick Axe Book instead, but as it turns out, at the time of this writing all the chapters give a "File Not Found" error.

Then I decided to watch the "Practical Common Lisp" video this time online using the Flash applet. However, towards the start, I pressed the "More" links in the related videos frame, and instead of a nice AJAXy thing, I ended up overwriting the current page. As a result, the video had to be reloaded from the beginning, and I could not skip and load the middle of it.

So I feel a bit frustrated from all these things going wrong. Luckily, I jogged on the way back from work, and so felt more calm and relaxed previously. I still hope to do some programming languages-related learning this evening, but it will have to be something else than what I tried so far.

A few days ago I noticed that some operations in the Amarok music player became very sluggish. For example, the track switching took a very long time. I recalled that something like that happened some months ago, at which point I ended up renaming away the KDE base directory ($HOME/.kde/share/apps/amarok/) which solved it. This time per advice from someone on the #amarok IRC channel, I simply moved away the SQLite collection.db file. After the database was re-populated, the sluggishness indeed disappeared.

However, I wanted to solve the problem permanently without having to keep resetting the database. So I decided to switch to a PostgreSQL backend, which hopefully won't get congested. As it turns out, if you can select in "Settings → Configure Amarok → Collection → Collection Database" PostgreSQL or MySQL, then they are supported. So I created a database, set up a user for it and set up Amarok accordingly. If it's not supported on your system, you may wish to consult this page for Mandriva and Amarok on the Amarok wiki and similar pages there for other distributions.

I also had a long-standing problem that when I double-clicked a video file, it tried to load it using Noatun, which caused artsd to crash. As someone on #mandriva-cooker told me, Noatun was no longer maintained, and I should use Kaffeine instead. So a few urpmi commands later, and after tweaking the file types, Kaffeine played my video files beautifully.

I hope you found this entry useful or at least a bit interesting.

Using tcpdump to record TCP sessions (or conversations) to a file, use the following command (as root or a privileged user):

tcpdump -w dump-file.pcap 'host 212.143.218.31'

Replace the 'hsot 212.143.218.31' with any tcpdump filter. Then you can run "wireshark" on the dump-file.pcap file. Note that I got incomplete results when I tried it now on my home machine and would love to be further enlightened.

I last took a look at KDE 4 three months ago. Since then, KDE-4.0.0 and KDE-4.0.1 had been released. Yesterday, an IRC pal of mine said he was informed us he was going to compile KDE-4.0.1 on his Gentoo system and test it. This prompted me to take another look at KDE-4.0 to see if and how much it improved since my last review. I am writing this report on KDE 4.

As I discovered (possibly again) to run KDE 4 one needs to run kde4env from the command line, and then type startx. Running it inside the .xinitrc/.Xclients file is not enough.

The first thing I noticed is that double-clicking the icons on the desktop does nothing, nor could I find any other way to invoke them. It's probably a bug. On the other hand, one can resize the icons, which is a pretty nifty effect. So you can have large or small (but useless) icons.

Moreover, when browsing the file system using Konqueror, I couldn't find the tree-view, which was one of my favourite views. I don't know if it was removed deliberately, or is still not implemented.

The Amarok progress bar of a song does not advance. It used to happen with videos, but now it happens with .mp3s and .oggs too. Amarok still doesn't display the visual part of videos. I thought it was supposed to be added in this version, but perhaps I had a mis-conception. Plus, I could not drag-and-drop files from Konqueror to its playlist, and Amarok doesn't enqueue a new double-clicked .mp3 file after it got opened by the first double-clicked file. Today I discovered one can double click files in the Amarok file-browser to open them, which makes it somewhat more usable.

One should note that Amarok 2.0 (the KDE 4.0 version) is pre-Alpha, so such bugs are expected, and the developers have asked not to report bugs.

KMPlayer seems to play the videos I double-clicked on fine.

On my setup none of the Alt+Tab, Ctrl+Tab, Alt+Space and Alt+F4 keyboard shortcuts worked, and the keyboard shortcuts applet in the configuration centre did not display anything.

In KDE 4.0.1, the taskbar displays applications from all virtual desktops. Moreover, I could not configure it to behave otherwise, because the right-click on the taskbar does not work. I was told I'll have to wait for KDE 4.0.2 for that.

The fonts in gtk+ applications in KDE 4 seems smaller than usual for some reason, which is a bit annoying.

In short, KDE 4.0.1 is better than the -rc I tried last time, but still too buggy to be ready for prime-time.

My Vim tips column could use a more comprehensive entry about window management (a.k.a "split view") in Vim, but this is a small tip in this regard, that bothered me for a long time and that I want to record for prosperity and share.

When having a verticla split, one can use Ctrl+W,Left or Ctrl+W,H to move to the left split, and Ctrl+W,Right or Ctrl+W,L to move to the right split. However, it seems that for some reason Ctrl+W,Ctrl+Left (while holding the control) does not work. To fix it, add the following lines to your .vimrc:

" mapping to be able to move to the left and the right windows
" without needing to leave the Ctrl key.
map <C-W><C-Right> <C-W><Right>
map <C-W><C-Left> <C-W><Left>

I've been annoyed by this behaviour for a long time, and now the salvation feels sweet. I should note that other Ctrl+W,Ctrl+Someting key-combinations work perfectly, so this aliasing is not needed there.

Arc is a new dialect of Lisp by Paul Graham and others. Recently, Graham has released a working codebase for Arc, which started a small community of people who play with it and improve it. As part of it, there's a world-writable version control repository for Arc called "Anarki".

Seeing that a comprehensive test suite was absent, I decided to start writing one. I decided to use the Test Anything Protocol (or TAP for short), which is a simple-to-implement STDOUT and STDERR based mechanism for test scripts to emit results of their test assertions to the TAP harness. This was partly because I expected the Arc test suite to be written in more than one language, and partly because I have been involved with the development of Test::Run, which is a test harness for TAP, and other TAP efforts, and so wanted to eat my own dog food.

So I've written (and later on enhanced) a rudimentary implementation of a TAP emitter in Arc. This TAP emitting module can be found in arctap.arc in the repository, and is used by the Arc tests. Some tests were written in Perl, primarily to test and fix bugs in the Arc TAP implementation. The Perl tests use Test::More, which is a TAP-emitting module for Perl, and Test::Trap. The latter is used to trap the STDOUT and STDERR of Arc programs that are run by the Perl scripts. This is done to test the sanity of the Arc interpreter and the Arc-TAP implementation.

So far the work on the test suite uncovered several bugs in Arc and one bug in Test::Run, and also resulted in some enhancements to Arc. Plus, I was able to better familiarise myself with Arc, and naturally to write test suite code which helps in preventing future regressions.

One feature of Test::Run that helped here was its AlternateInterpreters plugin (which is an idea I originally derived from something Curtis "Ovid" Poe (also a TAP worker) had blogged about). Currently, my interpreters.conf.yml file reads:

---
- cmd: ./arc.sh
  pattern: \.arc(?:\.t)?\z
  type: regex

This uses ./arc.sh to run the tests by default. The exact configuration is a bit more complicated than that (see Test.sh), but still the prove utility that is part of Test-Harness-2.xx, which is the older TAP harness, would not do. This is because it is impossible to assign a she-bang at the start of the Arc files, because it seems it confuses the Arc interpreter.

Currently 66 assertions in 7 files were written so there's still a long way for a decent coverage of the Arc features. But so far it seems that using TAP for the Arc test suite has been a sound choice, and I'm happy with it, and it probably facilitated putting the test suite together.

(Thanks to infi, mauke and simcop for reviewing earlier drafts of this post.)

I recently tried to install the proprietary Nvidia Linux drivers on Mandriva. It took me about two hours, but eventually the solution turned out to be very simple. Then I discovered there was a better way.

First the better way: if you're running a distribution kernel, you can install the Nvidia drivers by using the dkms package. To do this, go to Easy Urpmi and configure the non-free packages' source for your distribution. Afterwards run XFdrake, and select the nvidia card you have, and it will ask you if you want to install the proprietary driver.

If, on the other hand you wish to install from the Nvidia installer, as I initially did, and you are using the distribution kernel, make sure you installed the kernel-desktop-devel-latest or the equivalent kernel-*-devel package for your kernel.

That's it. After I did that, I was able to run and tweak REnouveau in order to help the Nouveau project. I also was able to run Extreme Tuxracer, and Compiz Fusion. The latter did not play nice with KDE and misbehaved badly.

So yesterday I decided that my homesite's markup may have had too much extraneous whitespace. So I looked at HTML Tidy to try to remove it. I searched for a way to tell it to remove whitespace on the Net, but most pages I found only said it was possible, but didn't say how. After some trial and error I found that this tidy.rc file:

indent: 0
indent-attributes: no
vertical-space: no
wrap: 0
indent-spaces: 0
break-before-br: 0
tab-size: 0

And a command line that specifies it with -config worked the best. It still left a lot of newlines, which could be removed, but I guess it's better than nothing.

While running it, I discovered that it complained about having empty files. And I checked that the files generated were indeed empty. Then I experienced with Website Meta Language (or WML) on Mandriva and found out that piping into it even simple documents resulted in empty results.

I found out that the update with the fix for the WML CVE issue broke WML. So I had to found out what the problem was generate a better patch and submit it. As it turns out the WML test suite still failed, but it didn't cause the RPM "%check" target to fail for some reason. After my fix, all tests passed again.

Then I was able to process the pages again. So back to tidy. Tidy still complained about some stuff like missing "summary" attributes in "table" tags. Sometimes, it exited with an error status without emitting any warnings or errors, so I had to tell my build system to skip these offending files.

So tidy ran, but it wasn't the end of my problems. When trying to build the homepage to run tidy with a clean copy, I found out that TTML did not run. This was caused by the upgrade to perl-5.10.0, and I looked for a way to prepare an up-to-date RPM out of it again. Both cpan2rpm and Ovid (the Perl module, not the Perl author) failed, and so I looked into cpan2dist. After experimenting with it a bit I got it running using CPANPLUS-Dist-Mdv and it gave me a working RPM. Now I'd like to prepare an equivalent package for constructing Fedora RPMs.

That was not the end of my problems. When I tried to clean-build my site, I found out that the directory structure did not get built before some of the targets. So I had to play with the make targets. Afterwards, everything went fine. The difference that tidy made was:

Before size: 4903322
After size: 4781819

What next? On the site I still have many links that start with "./". This prefix is completely redundant, and I can save some bandwidth by eliminating them. I also contemplated converting the navigation menus to JavaScript, which will read the menus' contents from the site in JSON format. Now that I think about it, the problem with this approach is that it will hinder navigating the site by web-crawlers, and may diminish the Page Rank of the various pages. It also won't work on clients which have JavaScript disabled. One thing I can do is also generate an HTML file with the current view of the navigation menu for each page and link to it directly for JavaScript-challenged clients. I still have to think about it.

Recently, I wanted to buy a USB Disk-on-a-key to replace my old 32 MB one. I went to Zap and compared the prices. The cheapest offer was from the Pandas.co.il shop, so I went to buy it there. The site appears to be written in PHP and I was able to fully operate it using the KDE Konqueror browser, and to complete the buying. I assume the Firefox browser will also work nicely there.

The 8 GB disk-on-a-key arrived a few days ago, and it works great. So I'd like to thank Pandas.co.il for making their site compatible with non-MSIE browsers, and I'll consider them for my future computer parts orders.

Windows are one of Vim's most convenient features, and learning some commands could prove to be useful. Some of the commands are available from the "Window" menu in gvim, or using the mouse, but using them from the keyboard is often more convenient.

To split a viewport in two, use Ctrl+W,S for a horizontal split or Ctrl+W,V for a vertical split. To move between windows use Ctrl+W followed by the arrow keys or h,j,k,l (but see this previous tip). To close a viewport use Ctrl+W,C.

Ctrl+W,r rotates the windows, and Ctrl+W, shift+R rotates them in the opposite direction. Finally, Ctrl+W,+ and Ctrl+W,- increase and decrease the height of the viewport by one line (in case you are mouse-deprived). If that's too little you can do something like 10,Ctrl+W,+ to increase the height by 10 lines.

In this tip, I'll explain how to debug C/C++ programs in Eclipse using a remote GDB session (gdbserver). To do this:

  1. Install CDT - The Eclipse C/C++ Development Tooling.
  2. Set up a C/C++ project so it will build and compile.
  3. Install gdb and gdbserver. The latter does not seem to be available in any Mandriva package and had to be compiled from the gdb source, where it is built by default.
  4. Run gdbserver on the generated executable using a command like gdbserver localhost:10900 ./my-exe
  5. In Eclipse go to Run → "Open Run Dialog...", and go to the upper debugger tab and select gdbserver Debugger in the Debugger drop-down. Afterwards, go to the "Connection" tab under Debugger Options and configure your connection. For my demonstration I've chosen "Type: TCP" "Host: localhost" and "Port number: 10900".
  6. Select the Run → Debug in Eclipse option and start debugging the application.

The reason I needed it was that I wanted to use "remote" debugging so I can debug a C++ program that must be ran as root, and would rather not run Eclipse itself as root. I'm still not sure it can serve as a good debugger front-end, but I became very annoyed with ddd and plain gdb start to seem to be inadequate on visualising the rest of the source code. So I looked for an alternative and thought Eclipse might be OK.

Back when I worked on Freecell Solver , I used to prepare binary distributions of the command line solver. Now it acted as a filter, and when invoked without arguments it just waited for standard input. As a result, when the Windows users double-clicked the fc-solve.exe executable, it just popped an empty console, which they couldn't understand what to do. And a lot of them sent me a "How do I use this" program email with the exact description.

Since I became annoyed of it, I decided to output the following to the standard error by default:

Reading the board from the standard input.
Type "fc-solve --help" for more usage information.
To cancel this message set the FREECELL_SOLVER_QUIET environment variable.

After the poor Winnies read it, it probably made them decide the program was not for them and to look for an alternative, but at least most of the emails I received about it stopped.

A year ago, Joel on Software published an article about having remarkable customer services, in which he said that one needs to fix everything two ways: first by helping the user, and then by making sure other users don't encounter and complain about the problem again by tweaking the behaviour. As one can see, what I did for Freecell Solver was essentially what Joel advised to do. (Albeit much later than I incorporated the fix.).

More recently, I started the Better SCM site, which compares and advocates several version control systems. However, git has been absent due to the fact a suitable maintainer did not come forward. And as a result, I received many emails asking "Where's git?". A few days ago I decided to add a Frequently Asked Questions (F.A.Q.) to the site where I answered this and other questions. However, today I received another email with "Git?" in the subject about that.

Another thing Joel had said is that "People Don't Read" or at least read very selectively. So I added a placeholder page about git with explanation why it is absent and a call for help, and added it to the comparison with the text "Unknown. Due to a lack of a maintainer for this system." with a link to the F.A.Q..

This should be enough to get rid of that particularly repetitive stream of emails, and hopefully will also yield me a contributor to add a better solution. Often designing web-sites and doing "customer service" (or feedback) requires designing the site properly based on the feedback one gets.

I didn't particularly think of what Joel said when I decided to do what I did with the F.A.Q. and the placeholders, but now that I think of it, I guess he was right.

On my homepage, I have many <h2>, <h3>, etc. tags with id attributes in them so one can link directly to the middle of the page. I have written a Website Meta Language API that allows one to generate a table of contents for the page based on them. However, recently I also looked for a way to have a link to their anchors somewhere close to them.

I eventually decided to try doing it using JavaScript and jQuery. It took me a bit of information lookup, trial and error and consulting people on IRC, but I ended up with:

<script type="text/javascript">
<!--
$("h1[id],h2[id],h3[id],h4[id],h5[id],h6[id]").each(
function(i){
$(this).append( ' <span class="selfl">[<a href="#' +
this.id + '">link</a>]</span>' )
})
-->
</script>

Now for some explanations:

  1. The $("h1[id]...") construct selects all the headings with id's. There may be a shorter way to do it (comments are welcome.
  2. The each method iterates over all of them and calls the closure inside. The closure sets the "this" variable to the current element, and accepts its index there (the "i" variable). In our case, we're not making use of the index.
  3. $(this) constructs a jQuery object from "this".
  4. .append() appends an expression to the inner HTML of the element. I add a little HTML there. this.id may result in an XSS attack if you have a really funky (and probably invalid) ID, but since I have control over my ID's it's ok.

You can see the result, in the headings of the presentations' page (for example), as long as you don't have JavaScript turned off. jQuery seems very nice, and I'm looking forward to making even more use of it where appropriate.

Andy Lester has written a great article about some of the social and philosophical problems with the Perl world. He says that while Perl 5 is a great technology, has a very comprensive collection of reusable, open-source code called "CPAN", and has a lot to show for, it has suffered from concentration of effort and other such problems.

This, in turn, contributed to the fact that many programmers concluded it was "dead", "dying", "not good enough", or other such negative hype and negative myths. While Lester's article suffers from some problems, it still makes a good read even if you're not an avid Perler.

2008-06-03: Next OSDClub Tel Aviv Meetings

On Tuesday, 3-June-2008 (tomorrow or today depending how you look on it), on 18:30, the OSDClub of Tel Aviv, which is a joint venture of the Tel Aviv Linux club, and Perl-Israel (and some other FOSS clubs who want to join the fun), will hold a social meeting. It will take place at the Café of Tel Aviv University near the junction of Einstein and Haim Levanon Streets. This social was scheduled because Zvi is in Israel. (If you don't know who he is, then come to meet him.)

Much later, on Sunday, 15-June-2008, OSDClub Tel Aviv will meet to share some Vim/gvim (= the text editor) Tips and Tricks. We'll meet in Schreiber (Math & CS) building in Tel Aviv University. We have more meaty presentations planned for July, but we hope these two meetings will be a a useful start after a long neglect.

2008-06-04: Mozilla/Firefox Tip: Getting xml:lang to Work

XHTML 1.1 has deprecated the lang="" HTML attribute in favour of the more standard xml:lang. I like to use XHTML 1.1 because I find the extra strictness (like no <a name="...">) useful, but in order to not completely alienate Microsoft Internet Explorer users, I'm serving it with a Content-Type of text/html.

However, it seems that Firefox as of v2 or v3 completely ignores "xml:lang" when the XHTML is being served as "text/html". It took me a long time to figure out a way to solve it, but here's what I did:

  1. Loaded jQuery:

    <script type="text/javascript" src="../../js/jq.js"></script>
    
  2. Added onload="lang_load()" to the opening "<body>" tag.
  3. Linked to a short jQuery-based script that implements lang_load(), which I've written.

What the script is doing is go over all elements and if the "xml:lang" attribute is set, then set "lang" to its value. It's a kludge, but it works. You can see the results on my homesite.

2008-06-06: Ubuntu is Dead

As illustrated by this bug (which got closed as "invalid"), the Ubuntu world has become infested with red tape, abuse, unfriendliness and hubris. So my suggestion is to stay a clear mile away of anything Ubuntu and to switch to Debian, Mandriva, Archlinux, FreeBSD, Fedora, CentOS or any of the other fine distributions with much healthier communities. Which one you choose depends on your needs and the situation.

This is especially in consideration of all the horror stories I heard of Ubuntu Hardy getting hang-up. I have spoken against Ubuntu before, but I may have missed the point, and it's possible the problems I've discovered recently are more recent.

The bug in question was closed because it wasn't a software bug, but that is also the case for the infamous bug #1.

In short, Ubuntu is dying now, but with enough determination on the part of its leaders, it can be resurrected. But this will require a huge inside motivation, and a lot of serious soul-searching. In the meanwhile, don't get near Ubuntu.

2008-06-16: Firefox 3 Slogan

I said it on IRC and on IM several times, but here it is on the Web for the upcoming release of Firefox 3:

Firefox 3 Runs Like a Fox on Fire.

It really does. And it leaks less memory and is very stable. Thanks to the Mozilla and Firefox developers for all the work they've placed there.

(This time I think the term "a fox on fire", may be inspired by the tale of Samson and foxes in the Old Testament, but who knows.)

2008-06-27: Summary of the Firefox-IL Meeting (from my POV)

This is the summary of the Firefox users' meeting that took place today at Park Hayarkon in Tel Aviv. I slept well today starting from 1 AM so was annoyed by the 8 PM clock. (I still felt energetic throughout the day.) After waking up and buying Bourekas (not enough time for breakfast - didn't even shave), I tried to find a bus station with the bus going there, but couldn't. So I took a cab.

The cab driver was very friendly. He didn't have a computer at home, but I was able to explain to him about software and open-source. One of his daugthers had studied Maths+CS+EE in Bar-Ilan and the other one is a lawyer. We did not talk about Computers exclusively, of course. I directed him to the Paz gas station and we ended up at a different one, and had to do a detour. Oh well.

After I got there, I took a detour and slipped on some mud approaching the Firefoxers. But I was ok (I suppose the grass was being sprinkled with water shortly beforehand.). There were already a few people there, and we started chatting. This was an anti-conference, and so we didn't eventually hold any formal thing like presentations/talks or introductions, but rather chatted, drank, ate and were happy. We also got some Firefox swag, but pitchefkes (= the Yiddish word for "small stuff") are not really an obsession of mine.

We had some beers there, and we had water and sodas, and we had some Croissants. I didn't eat anything, but I drank some water and 7'up. Lots of fun was had.

Here are some highlights from the conversations:

  1. We discussed the super-domains: "http://beauty/" which L'oreal wants and why people are less likely to want "ugly". But then I said "http://paris.hilton.is.ugly/" would be cool. Then we discussed "http://hilton/" and "http://ibm/".

  2. We discussed the fact that Israel had 100,000 downloads of Firefox so far, and that Iran is one of the top-10 countries there. BTW, I've been talking with a few Iranians on IRC and they've been very friendly to me in spite of knowing I'm an Israeli. I suppose most of the people in Iran are not the problem, but rather the regime.

    On the ride back (with Ori), I said that I felt that the Israeli politicians are a joke and do not actually run Israel. Who does is a good question, but it's possible that Israel just runs itself, with some entities, like the Military having more influence.

    Looking back, I can say, that the Israeli Law still has many bad elements, and should be changed because it's causing a lot of damage.

  3. We discussed the "israeli girls" search on Flickr and how it is dominated by female soldiers. People told me it was Digg'ed too. I said that Flickr's diversification algorithm was probably not as good as Google's, if there is one at all.

  4. A bit less political and off-topic, I had a long conversation with a bio-informatician from Ben-Gurion University. He is using Perl and "R" (because they are free-as-in-speech and do what he need.) He said their budget is pretty big and they have a program running on the server with Java and Oracle and stuff. Then he said that he bought guitars instead of going on trips abroad and so, like me, remained in Israel most of the time. He seemed nice, and said he remembers my name and will MSN me.

    We discussed some machine-learning algorithms. How he has to optimise over 500 parameters and decide which to include and which not. Naturally the sets of all subsets for 500 paramaters is huge, and so cannot be done by brute force. I told him about my greedy algorithm for finding a meta-scan in Freecell Solver (hi Muli!).

  5. I said that I think that non-x86 (= non-Pentium etc.) architectures are going to make a comeback in servers, and that we'll see more peopple increasingly buy machines running UltraSPARC, PowerPC, and other non-x86 CPUs. (More about that later). Tomer noted that he does PHP software development at work and that such architectures were where all the big money was made. He said some companies have ancient IBM PowerPC-based setups which they don't upgrade.

  6. There were quite a few girls there: the Pink Fakatsa, ailaG, and two new girls I didn't recognise. I also met Kobi Zamir, who is known as the libhocr guy, and he was very nice. He is tall and slender, and I told him most Kobis I know had a bigger build, and he said that he felt so too but said he was an exception. We discussed people with similar names.

    Israelis have many common private names, but many names are very common. The situation is much worse in Russian, where they have less than 20 common masculine private names.

  7. I met Meni whom I didn't see for a long time in Telux and other FOSS meetings. He said that he's now working in a different place, which is harder to come to Telux from, and that he now has a car, which he didn't need when he worked in Hertzeliyah Pituakh. We discussed the fate of Atelis, where he and other three esteemed developers worked, and now has been split into pieces, due to bad investors.

I'll probably write some further stuff in comments, but I'm just doing my duty to people who did not attend. In short - a fun meeting. About 20 people came and left, and a lot of fun was had. Next time, I suggest doing it on a weekday during the evening or afternoon, and optionally not at the summer. A more formal schedule and place would be good too, and I suppose I can make some arrangements at Tel Aviv University.

I returned home with Ori, and we discussed some stuff, including Shania Twain. I thought I'd upload some of her songs online, but that's what YouTube is for, no? Knock yourself out with Shania goodness! (or Shania Badness if you happen to think so.

Stay on fire (foxes)!

2008-06-24: Predicting Three Comebacks

Well, it's a common belief that comebacks in technology don't happen often, but this time I predict three comebacks. I cannot recommend that you base your decisions (much less investment plans) based on it, but here they are.

Comeback #1: Non-x86 Machines

Once upon a time, companies spent a fortune on costy UNIX servers or even IBM mainframes, just to be able to run their operation. Since then, x86-based computers have become faster and more capable, enough to make people be able to run Microsoft Windows NT or Unix-based Operating systems on them. This is now the most popular setup on the Internet for servers (much less clients and workstations).

However, these x86 machines are not as integrated as their non-x86 equivalents (UltraSPARC, PowerPC, etc.), are still much slower (due to the limitations of the x86 architecture vs. the superior RISC family of architectures). This costs in a lot of time of maintenance, many hardware failures, and sub-optimal lifetime. It is well-known that there are some PC XT machines which are still running. However, finding a low-end desktop Pentium I machine that's still OK is an exercise in futility. And even the high-end x86 servers suffer from many problems.

Human time is much more costy than shelling some money for getting a good RISC machine. And with the portability and stability of the GNU system, and portable and free operating systems such as Linux, or the BSDs, you can easily run a server there at ease.

So what can Sun and IBM (and friends) do about it?. They should phase out their own OSes in favour of Linux, as people have grown to dislike Solaris (to say nothing of AIX, HP-UX, or greater brain-damages). Don't get me wrong - Solaris has its place and I'm sure is a fine piece of engineering. But GNU/Linux is better for trying to promote the hardware.

Sun should also make the OpenSolaris licence GPLv2 and GPLv3 compatible, to allow it to borrow code from Linux and other GPLed-products (and vice versa). And they should also start supporting Perl, PHP, CPython, CRuby, Mono etc. etc. instead of just Java or whatever has been hyped. (Similar to what Microsoft have been doing recently for its MS-Windows OSes).

All of this will allow selling more hardware, getting more support and ergo - more profits. I think the transition to non-x86 machines will happen anyhow, but that will make it faster, and much more profitable for them.

Note that it is not the end of either Intel or much less the end of Microsoft. I still see workstations dominated by x86 in the near future, and both MS and Intel are insanely profitable and have enough time to re-invent themselves. But I think (and hope) we'll be seeing more and more non-x86 servers and to a lesser extent workstations.

Comeback #2 : Mandriva

Once upon a time, the Mandriva Linux distribution (formerly Mandrake) was the de-facto standard for home Linux desktops. Then vocal Debian people started spreading a lot of pro-Debian and anti-everything-else FUD, and Ubuntu also became the over-hyped distribution-of-the-moment. Meanwhile, naturally, Mandriva suffered from lack of popularity and negative FUD, but also still continued to improve.

Right now, as Ubuntu Hardy (which I nicknamed "Hardly") tends to hang up or worse so often, and Ubuntu as a general rule seems to be infested with red-tape, it seems that there's an opening for Mandriva and distributions that are very good, but were not as hyped as Ubuntu.

Here are some Israelis who converted away from Ubuntu lately:

These are all Linux experts and FOSS contributors. It's harder to tell what the common joe who wants to install Linux at home will choose. But I think that they will tend to hear a recommendation from the experts, too.

Will Mandriva become the hottest distribution? Maybe not. But it doesn't matter - Mandriva was never about hype. Naturally, Mandriva has had and still have its share of bugs and problems (mostly non-critical). But so do most other distributions, including such whose users were spreading anti-hype against it.

Writing this from a Cooker (which is the Mandriva equivalent of something like Debian Testing or Unstable) system.

Comeback #3: Perl

And now we come to the third and final comeback - Perl. You can often here that "Perl is dead", or that "Perl is dying", etc. It's a matter of image, more than technology or mindshare. Many people would blame it on the anti-Perl FUD we can hear from various sources. But I believe the problem has lied in the Perl world itself. To quote an email I sent:

The Perl technology is in an excellent shape. It's fast, feature-rich, powerful, easy to use and learn, under active development, relatively bug-free, with many automated tests, and with many 1st party, 2nd party and 3rd party enhancements (re CPAN).

However, the Perl community is sick. Not dying - but sick.

This sickness was not caused by external attacks. Instead it was created from within the Perl community. By a leadership/intelligencia who were more keen on writing and selling closed books (see my essay about it) than on promoting the core Perl technology, documentation and community. It was caused by an "official" IRC network (which is still the first hit on a Google search for "irc perl") which is full of ego, abuse and hostility - where kicking, bannings (and seemingly K-lines) are not only common, but tolerated and seem desirable. It was caused by administrators and leaders who became too "busy" (and as a result heavily unproductive) to have time for Perl, which is what have been placing food on their table. It was caused by central sites whose source code and data were hidden from the public due to a lot of unknown bad reasons, and for absolutely no good ones. (We advocate openness after all).

But most of all it was caused by the many competent and active Perl contributors who did not take the initiative to replace the existing and sick infrastructure by a more independent and decentralised effort.

(I had written something relatively similar in the pre-Web-2.0 era as part of my "Usability of the Perl World for Newcomers Essay". Nevertheless perlbuzz.com deserves credit for presenting it better, making it public knowledge, and making it more obvious.)

But the situation is now getting better. There are more independent developers who have started to set up sites and help resources of their own. More people blog about their Perl experiences. I'm also going to continue giving my share of the contribution by setting up Perl-Speak (Please don't register yet - the E-mail does not work yet due to problems with my hosting), by contributing to the FAQs, documentation and wikis, and by helping with coding. People like me would also like to complement the inadequate, "official", and completely non-open *.perl.org infrastructure, which has been suffering from bitrot and neglect.

Lately, I've had the pleasure of talking to many people on IRC who have decided to learn Perl for various reasons, and some of them are young or even underage. Along with CPAN and perl5, Perl offers some clear advantages that no language has.

Again, Perl is also not about hype. Some people have privately raved about Perl and recommended it to their friends, but otherwise Perl was promoted quietly, bottom-up and mostly by word-of-mouth. It's hard to know if Perl will become the "hottest" language again. But I believe it will still become more popular, even among new programmers.

2008-07-21: Hardware Log: ATI HD 2600 Pro

Up until this Saturday, my Linux workstation had an old Nvidia GeForce 4 MX card, whose fan recently died. We decided to upgrade the card, and I told my father that an ATI card would be preferable over Nvidia, because AMD/ATI are more FOSS-friendly and released SPECs. So he bought an ATI HD 2600 Pro card. It's r600, but quite low-end (but still not as much as the GeForce 4.).

We delayed installing it, but this Saturday, after I had to reboot after the new kernel update, we got to it. The hardware replacement went surprisingly well, and then we booted the computer. Everything went well on startup, and we got the login prompt. Then startx worked right away, as my Mandriva Cooker installation detected the hardware change and set up all the drivers accordingly. Mandriva++ .

Not everything worked flawlessly. I had to edit the xorg.conf file to re-add the settings for the Xkb Hebrew keyboard, and to change the Bits Per Pixel from 16-bits to 24-bits. Then I was happy. I'm using the RadeonHD open-source drivers, which don't really do 3-D yet, but since there are SPECs for them, the situation is expected to improve.

I'm so glad I've now escaped from Hang-vidia-land, and have a card from a company that plays along with the open-source ideology. I hope to contribute further to the open-source drivers by reading the SPECs and writing code, and hopefully I would be able to understand what I need to do better than with Nouveau. In any case, since I'm not obsessed with "gamer" computer games and am not using the high-performance 3-D/2-D effects in much, I can survive just fine using the non-3-D enabled drivers. (Just as I used the "nv" driver before the upgrade.).

2008-08-02: Hacktivity Log: CMake and JavaScript XSLT

In an email to the fc-solve-discuss mailing list, I detailed recent work on Freecell Solver. I added an suite of automated tests, which in turn allowed me to perform the conversion of its build-and-configuration system from the GNU Autotools (Autoconf, Automake and Libtool a.k.a Autohell) to CMake. I ran into a few problems in the process of the conversion, but it took less than 2 days, which was probably much less than its Autoconfisication of Freecell Solver took, excluding the constant maintenance of its Autotools configuration and random breakages.

You can refer to the trunk for the present state of the CMake configuration and the conversion-to-cmake branch for the gradual CMakification process.

In any case, now the source archive is smaller, configuring it is faster, make runs faster, and I have a configuration GUI. In the process, I also added support for the Qt GUI to the Mandriva CMake. There are probably still some quirks in the config+build system, but all in all, I'm very happy.

I also ran into a problem that Firefox completely refused to display the https:// URLs of svn.berlios.de, without letting me add an exception. As it turned out, the certificates were probably changed, and I had to manually delete all *.berlios.de related certificates, and then add them with exceptions.

Next - yesterday I read about jdavidb's problem of finding differences in two versions of a Fedora RPM, which prompted me to find out about Fedora's packages' version control repository, which I did. Then I ran into a comment in that page about the Better-SCM comparison, which inspired me to do further work on it.

I decided to finally experiment with the Google implementation of XSLT in JavaScript in an attempt to customise the output of the XML in the client-side. I wanted to fetch the contents of the XML document and XSLT stylesheet from the server which involved doing AJAX using jQuery. It took me a long time to get right, because I didn't realise the jQuery $.get(...) callbacks were asynchronous and executed at their own time. I guess it's called "Asynchronous JavaScript And XML" (= AJAX) for a reason.

Then I got it running, but then ran into a Google JS XSLT limitation of "not implemented: key" (probably the XSLT "<xsl:key />" feature which I'm using). Still, I gained a useful experience working with AJAX, JavaScript and jQuery.

In regards to August Penguin - I don't have a lot to report, because I was too tired due to lack of sleep in the previous days and so left early, right after the opening sessions.

All that put aside, I should note that the Cooker version of Mandriva (which is its development/experimental version), which I'm using now is now relatively broken on this machine, and many applications get hang for noticable periods of time or even completely. Part of it may be due to the switch from KDE 3 to KDE 4.1.x, but maybe there's something wrong on this machine's configuration, which I cannot get to the bottom of.

2008-08-03: Review of KDE 4.1 on Mandriva Cooker

Since KDE 4.1.x was recently released and was made available in Cooker, I decided to give it for a test drive. So I created a new UNIX account logged in to the console and ran startx. KDE 4 started, looking like I was used to, and I started making it behave like I was used to.

Here are a few problems I encountered, with solutions or workarounds, based on my memory and my IRC logs:

  1. I had problems putting an icon of Konsole, the KDE terminal emulator, on the panel. The problem was that the KDE "K" menu was configured in Mandriva to be the "Classic Menu", where the right mouse button does not work. After switching it to the Kickoff style, I could right-click the menu item to add it to the panel.

    I don't understand why the right-mouse button does not work in the classic menu, or why I cannot drag&drop menu items to the panel.

  2. Right-clicking on an applet or application launcher in the panel does not give one an option to move it. The only way to do it is by clicking the GNOME-logo-like icon at the right of the panel and entering into panel-editing mode. There is a "Move" option in the context menu of panel icons in KDE 3, so I wonder why it's absent.

  3. There doesn't seem to be a system tray by default in the panel, and after I added it its display of the akregator icon is a bit awkward.

  4. There aren't any tooltips for any of the icons in the panel, so it's hard to know what they are all about (One needs to click them to find out.)

  5. I had problems finding the equivalent of the KDE Control Centre (kcontrol) in KDE 4. Apparently it's called System Settings in the menus, and is invoked using "sysetemssettings".

  6. I had problems making sure the taskbar only displays windows from the current virtual desktop. Apparently, one has to click on the taskbar in a place where "there's no application" (i.e: in the space between applications) and then choose "Task Manager Settings", where the option is present.

    This option is not present in the KDE 4 "system settings" because it's relevant only to an individual taskbar, of which there can be many. Still, I would like to see a button on the left of a taskbar invoking a context menu.

  7. The "Expandable" property of the "Details view" of Konqueror does not get preserved across invocations of Konqueror, despite the fact that I pressed "Apply" and "OK". This seems like a bug.

  8. Finally, on Mandriva, KDE 4.1 does not make use of KDE 3's data, so KDE 4.1 applications like KMail, Akregator, etc. are useless to me.

Of course, KDE 4.1 still seems likable, and I enjoy using it. But I feel like they need to work on better usability. I am not a newbie - I'm actually a computer geek and an experienced software developer - and I felt like I could not make sense of some of the new UI there.

I'll try filing bugs in the KDE bug-tracker about it next, and see where it leads me. Hopefully these usability problems will be fixed in KDE 4.2.

2008-08-09: Tip: Fixing rpm hangup as root on RPM-based Linux distributions

This is more of a memo-to-self than a tip, but it may prove useful to others.

When rpm (the package manager for Linux) as root gets stuck right at the start without doing anything, try the following:

  1. Kill all rpm-related processes.
  2. Move away /var/lib/rpm/__db.* to a temporary location.
  3. Run rpm --rebuilddb.

That's it! It already fixed my wedged RPM database twice, so I'm putting this advice here so I won't forget it.

2008-08-21: Recent Hacktivity Summary

Well, I added more tests to Module-Starter, and while I was writing the tests, I discovered some bugs in it. So now there's a patch with the tests and the bug fixes and another one with only the bug fixes. Both of them are unapplied.

Having been trying to learn Ruby recently, I decided to write a Conway's Game of Life implementation for it. It took me most of one evening to get it working and then part of the other day to make it more idiomatic with the help of the people on Freenode. It felt good in a way.

Then I decided to finally see why my Test-Run failed many tests recently. As it turns out, this was due to broken TAP-Parser compatibility: the skip message in skip_all field now accepts only "SKIP " and not something like "skipping: ". Fixing the Test-Run tests fixed it and I released new versions.

I also noticed that many of the Test-Run-CmdLine plugin had the following pattern:

sub _init
{
    my $self = shift;
    $self->NEXT::_init(@_);
    $self->add_to_backend_plugins("AlternateInterpreters");
}

So I overloaded the construction function, only to add a plugin. I decided to fix this by doing an accumulation of such plugin specifiers in every plugin, and so got only:

sub private_backend_plugins
{
    my $self = shift;

    return [qw(AlternateInterpreters)];
}

Which is much cleaner and less intrusive.

Next on my plate was Freecell Solver. I decided to work on make_pysol_freecell_board.py which is a Python script that generates the initial boards of PySol. The script was crudely written to begin with and it became even cruder as time went by. I decided to give it a serious face-lift. So what I did was:

You can find what I have so far in the repository. It's much better than what I started with two days ago. Writing such Python code now seems more fun than I recall it, and I actually enjoyed it.

In regards to SMOP, they convinced me to install Ruby-1.9.x under a prefix, which I did, but then it yelled at me for not having a Haskell cabal thingy. Turns out that the Pugs Makefile.PL installs it somewhere under the home-directory, which I didn't want to happen, because I want to keep it tidy. Again, this reminded me of Drivl.com recipe on how to make square corners, and I decided to give up on SMOP again.

And I should note that I was able resolve a long-standing problem I had with XML-LibXML/XML-LibXSLT on my Mandriva system, and now I simplified the XML-Grammar-Fortune build-system.

I also spent some time writing a backup system for some of the Israeli MediaWikis that I manage. This involved a bunch of Perl scripts.

So - Perl, Ruby and Python - all in a few days work. Cheers everybody.

2008-09-10: Tip: Handling mailto: Links in Firefox 3

Resources such as this one explain how to setup Firefox 2.0.x to use kmail to send email from "mailto:" links. However, compatibility with this scheme was broken in Firefox 3, as evident from Firefox bug #428658 - "network.protocol-handler.app.mailto" users need to edit/remove mimeTypes.rdf".

The solution I found is to access "Edit → Preferences → Applications → mailto" and select the appropriate application there. This seems to work for Firefox 3.

2008-09-14: Recent Hacktivity Summary

Another hacktivity summary - what can I say?

  1. WebGUI - I decided to try installing WebGUI again. After preparing SRPMs for all of its CPAN dependencies and installing them, I was able to tweak its and Apache's configuration files based on the instructions there to get it to run. However, then when I tried to configure it, I found that it go stuck on the colours configuration. So I temporarily gave up.

  2. Readline Replacement: I began working on Term-Eatline, which was supposed to be a Perl prototype of libeatline, which aimed to be an enhanced BSD-licensed GNU readline replacement. After starting writing it using curses, I ran into a "Curses Problem in Detecting Backspace". I found a solution in the CPAN Curses::UI module, and went on. Then I ran into another problem with detecting Alt+Key combinations, and decided to see how readline is doing it. As it turns out, it doesn't use Curses after all.

    I decided to see if I can now use tcsh's code to bootstrap eatline, and ran into a tcsh bug which refused to even let it run. I wrote a patch for this based on the discussion and some debugging effort, and posted it on the bug report.

    Afterwards, I contacted a tcsh representative about the bug, and after telling him my motivation for contacting him, he referred me to libeditline which is a BSD-style readline clone. Guess I now have something substantial to start from.

  3. Browser bugs: I ran into a few browser bugs on my current Mandriva setup. For once, Firefox freezes/stalls for many seconds when loading most pages (which seems to happen only on my system), and there are many delays in the font resizing.

    I'm disturbed by these problems a bit, and don't know what causes them.

  4. Gringotts: Rorschach contributed a minimise-to-a-tray-icon patch for Gringotts. The patch still has some issues, but he is working on it.

  5. MediaWiki: I decided to contribute to MediaWiki, and decided to start by running the test suite. It failed. As it turned out, the test suite is currently broken, and the MW's developers method of ensuring nothing is broken is by uploading it to the Wikipedia and testing it there.

    I decided to fix the test suite. After fixing a few minor breakages and submitting patches for them, I ran into a very puzzling bug. I decided to isolate it by gradually removing more and more unnecessary code, while still being able to reproduce it. I started doing that on the test suite, and found out that I needed to incrementally copy more code from the MediaWiki in order to trim it. After 174 such iterations (and intermediate versions occupying 7.9 MB of my hard disk), I was able to find what the problem was and write a patch. The problem was that the database handle was initialised in an old way.

    Now the test suite still fails especially with some tests in the t/inc/Parser.t tests, but I'm hopeful. I am disappointed that the MediaWiki guys don't take automated testing more seriously.

  6. CPANPLUS-Dist-Fedora: Someone contributed a patch cleaning up CPANPLUS-Dist-Fedora so I needed to test it on my Fedora VirtualBox VM. This turned out to be harder than I expected. When updating to the latest version, I ran into a missing ldap .so dependency. I found out I can eliminate this problem by removing gnupg2 and all of its dependencies including kdepim. So I was able to update it after all.

    Then I decided to use icewm instead of KDE, and wanted to set the wallpaper. And then it turned out that infuriatingly, xsetbg and xli were not available in Fedora anywhere for some reason.

    Eventually, I was able to test CPANPLUS-Dist-Fedora there. The patch added a change that caused loading the package to fail on all system without an "rpm" executable, so I released an update as 0.0.3, which also eliminated the use of two unnecessary modules.

  7. CPANPLUS Typo fixing: while working on CP-D-Fedora, I found out that CPANPLUS had a line with "it's" instead of "its". So I grepped the source for similar cases and submitted a patch that hopefully corrects all of them.

  8. I asked where the previous YAPCs were on yapc.org and was replied that it wasn't implemented yet and was referred ot the source. True to my mantra that "One shouldn't complain unless one is willing to implement it himself.", I wrote a patch to create such a page. Most of the foundation for it was already there.

  9. Bug in Cwd.pm: I received a strange failing report for one of my CPAN distributions with an error that I could absolutely not reproduce on my machine. As it turned out, the problem was that on that machine they used the pure-Perl version of Cwd.pm instead of the binary one that I used, and they differed in their behaviour.

    I wrote a patch, with a test case and a fix and submitted it.

  10. Finally, I prepared Acme-CPANAuthors-Israeli. The first version didn't list the dependencies in the Build.PL, but this is fixed in version 0.0101.

At the moment, I feel a bit disoriented and don't know what to do further. I think I'll help some more with Firefox bug triaging.

2008-09-16: Tip: Dumping the History of Google Code's (or any Remote) Subversion Repositories

Consult this FAQ on Google Code for how to use svnsync to clone a remote Subversion repository so you can have a copy of its history. I was told on IRC that it has the same end result as using svnadmin dump on the physical repository.

This post was supposed to be a rant on why I'm not hosting my projects at Google Code because they don't have a way to retrieve the dump/history of the Subversion making it a case of vendor lock-in. While this had been true in the past, I discovered it is no longer the case today, while looking for the relevant FAQ entry.

So kudos for Google and the Subversion people for fixing it.

2008-09-18: Shell Variable Injection

GuySoft writes in his blog about Unix shell commands to convert video. In the shell commands, he expands variablse like -srate 22050 $infile -o $outfile".flv", without putting them inside double-quotes. In this post, I'd like to demonstrate why this is so dangerous.

When the shell expands a regular variable outside quotes it breaks into words separated by whitespace and then evaluates every word separately. To use a variable as a single token, one must enclose it inside double-quotes: "$MY_VARIABLE".

Now, let's demonstrate what can happen when we don't do it properly. I'm in the home directory of a user called "shelltest on my machine". Let's see what we have there:

[shelltest@telaviv1 ~]$ ls
Desktop/          test-shell-injection.sh*  tmp/
Freecell-Solver/  test-shell-injection.sh~
[shelltest@telaviv1 ~]$ du -s Freecell-Solver/
86M     Freecell-Solver/
[shelltest@telaviv1 ~]$ cat test-shell-injection.sh
#!/bin/sh
mkdir -p $HOME/Backup
cp $1 $HOME/Backup/
rm $1
[shelltest@telaviv1 ~]$

We have the populated "Freecell-Solver" directory, and a shell script called "test-shell-injection.sh" that uses $1 (its first positional argument) unsafely.

Now let's abuse it:

[shelltest@telaviv1 ~]$ ./test-shell-injection.sh "-fr $HOME/Freecell-Solver"
[shelltest@telaviv1 ~]$ ls -l
total 8
drwxrwxr-x 3 shelltest shelltest 28 2008-09-18 14:14 Backup/
drwxr-xr-x 2 shelltest shelltest  6 2008-09-18 14:05 Desktop/
-rwxrwxr-x 1 shelltest shelltest 58 2008-09-18 14:11 test-shell-injection.sh*
-rw-rw-r-- 1 shelltest shelltest 60 2008-09-18 14:09 test-shell-injection.sh~
drwx------ 3 shelltest shelltest 28 2008-09-18 14:07 tmp/
[shelltest@telaviv1 ~]$

Voila! Freecell-Solver is gone. I could have also said ./test-shell-injection.sh "-fr $HOME" to delete the home directory. If the script does something like for I in * ; do ... # Something with $I .. done, then a malicious user can supply a file called "-fr $HOME" (where $HOME is the value of the home directory) in the input to delete the home directory.

In short: when writing shell scripts, always enclose variable expansion in double quotes, unless you are certain that they don't contain whitespace (which implies that you generated them yourselves), or you want to use them as a list (which also requires that you generated them yourselves).

2008-09-18: OSDClub Tel Aviv Meeting: Ori Idan about the Semantic Web

The Tel Aviv Open Source Developers Club (OSDClub) (formerly Telux) will hold a meeting on 21/September (next Sunday). Ori Idan will deliver a presentation about the Semantic Web.

Ori is the director of the Israeli branch of the World-Wide-Web Consortium (W3C), and is a very good presenter, so it is recommended to attend.

The meeting will take place at 18:30 in the Schreiber Maths and Computer Science building of Tel Aviv University, room 008. Attendance is free of charge and everyone are welcome.

2008-09-19: Impressions from Ruby

Ahoy mateys! Arrrrrrrrrr! Happy Talk Like a Pirate Day!. In the piratey spirit, I decided today to get my act together, and start working on a project I've intended to write. Since I've been learning the Ruby programming language lately, I decided to write it in it, to learn more about Ruby and to experiment with it. In this post I'd like to blog about some of the facts I discovered that surprised me in Ruby.

First of all, I discovered that Ruby regular expressions don't have Perl 5's "\G" and /g feature that can be used to incrementally parse a string using regular expressions. The closest thing I could find was using .sub! (with the empty string), which does an in-place subtitution of a regular expression

The second thing I discovered was that Ruby did not have ++ or -- operators - one has to use += 1 and -= 1. When Ruby tried to parse a line containing "++" it yelled at me with an obscure error, that caused me to need to incrementally comment-out code until I found out what the problem was.

Another fact that disappointed me was that the Ruby debugger (ruby -rdebug) does not seem to have conditional breakpoints (i.e: break at a certain line only if a condition is met).

Finally, when I tried to do .inject { BLOCK } ("inject" is Ruby's list reduce / fold method), without a starting element, I discovered that it used the first element as a starting point instead of 0 or nil or whatever. Doing .inject(0) { BLOCK } in my case fixed it. This fact is documented in the documentation.

Otherwise, Ruby seems likable and I've been feeling I'm productive in it. I've been using RSpec for writing my automated tests. It's a so-called "Behaviour Driven Development" library, which is the same as Test-driven development, but RSpec is still nice.

2008-09-27: Tip: Circular Symbolic Links in Fonts Directories Causing Slowdown

I noticed a slowdown recently on my Mandriva Linux Cooker system where X GUI applications would stall for a long time and gdb showed that it happened in fontconfig calls. It happened with all user accounts on the system, including new ones and was not fixed by upgrades. Furthermore, other people who used Cooker did not have it.

Well, today I finally found a solution. Apparently, some of my font directories, contained symbolic links to themselves, which confused fontconfig and caused it to scan them times and again.

To fix the problem, run strace -e open -o fontconfig.strace fc-cache -v, and see if it excessively scans more and more sub-directories under the directory paths. Then move away the offending symbolic links to say /usr/share/REMOVED-fonts.

Now konsole starts instantly, KDE 3 apps no longer become unresponsive when the file with the unicode character, firefox stalls less, and font zoom there is faster. I'm finally happy again.

2008-10-07: Autumn Cleaning

I like to keep my queue of incoming items under control. Recently I noticed several vectors that I did tend to, and decided to deal with them as well. After I dealt with them, I've been a little negligent, but now they're fully under control again:

  1. Home Email Inbox (KMail) - down to zero messages.

  2. GMail's Inbox - down to zero messages.

  3. GMail's folders - down to zero unread messages in all of them except in the Mandriva Cooker and Mandriva Bugs folders, which I don't need to keep at bay. I previously left many unread messages there, but now I decided to keep them at bay as well.

  4. Bookmarks in Firefox's Main Bookmarks Menu - down to zero items. I often bookmark them there to read, sort or otherwise deal with later.

  5. Bookmarks in Konqueror's Main Bookmarks Menu - down to zero items.

  6. Unread web feed in my web feed aggregator (Akregator) - down to zero items.

  7. Files in the root of the home directory - down to zero.

Did I miss anything? See this Perlcast Interview with Tom Limoncelli on "Time Management for System Administrators" for more information on keeping your incoming items at bay.

2008-10-10: Firefox Tip: Disabling the Drag & Drop Previews

In Firefox 3, when one drags and drops images or text, they are given as a large preview around the cursor. I had found it annoying and sought a way to disable it. This tip came to the rescue after a short Google search - just go to about:config and set the nglayout.enable_drag_images to false.

Cheers!

2008-10-11: Reply to fxn Regarding User Was Successfully Created

fxn writes in his Advogato blog about the fact that many web apps confirm that a user was successfully created:

Why so many webapps confirm to the user that some action was successfully done? Of course it was!

Look at your desktop applications, your editor does not bug you saying "File saved!" constantly, iCal is indeed almost completely silent. You warn the user when the disk has run out of space, right?

I think those messages come from the insecurity the developer feels about the amount of failure points between request and response. Perhaps some are just repeating the pattern seen elsewhere. But that's not the user's business, you warn when you fail.

I disagree with this. The web is a different user-interface medium than desktop applications where different rules apply. If you don't give confirmation on the web for successful actions, the user will wonder if they were successful or not. So you do need to do so.

For example when submitting this form, I am brought back to the same form page. So I don't know whether it was successful or not or if it got sent at all.

So a web application writer needs to write a confirmation that many successful operations took place, and to error if there was a problem.

2008-10-11: The "I Have Nothing to Hide" Fallacy

A common pattern we hear on the Internet in regard to privacy or security is "I have nothing to hide" - no one will want to target me because they'll gain nothing from doing so. Today let me tell you a story that shows why this is not true.

A few years ago, I was using the same easy-to-remember password (which was only 6-letters long) on most of the sites I had accounts on. One day, I received an email from Freshmeat.net asking me if the fact that I had changed the description in the record of Freecell Solver there to "Freecell Solver is a useless 100% ANSI program that automatically solves games of Freecell", indicated that it was not worthy of inclusion there. This surprised me because I naturally didn't modify it like that nor intended to.

After talking with the admins of Freshmeat, I realised that someone logged in to my account, and submitted the malicious update for inclusion. They ended up giving me his IP, which was in Israel's Netvision ISP (while I'm subscribed to a different ISP). Now, this change was pretty innocent, but naturally, now that he knew my shared password, the possibilities for him were endless. As a result, I went on a concentrated spree of changing that password to new, different ones in all the accounts I created on the Net with it. I made a smarter use of my password manager and eventually discovered the auto-remember-passwords feature of browsers such as Firefox and Konqueror, and solutions such as OpenID.

There's no good excuse to compromise on security. Do you have a bank account and access it online? If you're not careful enough, a malicious attacker installing spyware on your PC might empty it. So you say to yourself: "What does he have to gain from me? I only have $10,000 there.". Maybe you do, but if he empties hundred or thousands of accounts like that by writing a robot, he'll become rich, so he isn't likely to not to target you.

And some people are keen on doing random vandalism with your online presence, like the one I mentioned, who may have also been trolling my blogs. Therefore, make sure you're as safe as possible. This incident was all I needed to become more careful, and I hope you now realise that, as well.

2008-10-19: Three Tips: Perl with Vim, Subversion Problem and mailto: links in Firefox

First of all, my home-site's back and you're gonna be in trouble! Otherwise, this entry contains three technical tips that I collected recently:

  1. Perl and Vim - sometimes one encounters a case where he needs to handle many errors in his Perl script. I encountered a similar case, when I had to work on a script that was written without "use strict;" and "use warnings;" and shouted at me senselessly when I added them.

    To greatly help with that install perl-support, and look at its Perl → Run menu - there's "update, run script" (Ctrl+F9) and "update, check syntax) (Alt+F9) there which place the errors in the quickfix buffer and allow to quickly navigate them.

  2. Subversion and Apache - I recently encountered this error when trying to mkdir the first directory in an empty Subversion repository, that was served using Apache:

    shlomi:~$ svn mkdir -m "Creating TO-DEL trunk" http://localhost:8080/svn/TO-DEL/trunk
    svn: OPTIONS of '/svn/TO-DEL': 200 OK (http://localhost:8080)
    shlomi:~$
    

    This surprised me because my client was up-to-date and the service used the same libraries, and it worked with other repositories. As it turned out the problem was that the Location entry in Apache's httpd.conf looks like this:

    <Location /svn/TO-DEL/>
        DAV svn
        SVNPath /var/svn/TO-DEL
    </Location>
    

    The problem here is that /svn/TO-DEL/ in the "Location" tag has a trailing slash which confuses Subversion. Eliminating that issue solves the problem.

  3. Firefox and mailto - if "mailto:" and other non-"http:" links stopped working in Firefox, make sure that NoScript is upgraded to the latest version. There were problems with some recent versions that broke "mailto:", "ed2k:", etc.

2008-11-01: Why I Hate KDE 4

Here are the reasons why I hate KDE 4:

  1. It looks like shit. Seriously a huge step backwards from KDE 3. The colours are annoying. The icons are hard to distinguish and ugly. I have no idea how to make it look better.
  2. Plasma leaks memory (see this bug report), making me having to restart the desktop after a few days.
  3. The buttons order is not Windows-like. Yes, they reverted to the GNOME/Mac-o-Sucks order by default. I have no idea how to change it to the Windows order which I like.
  4. The New Konqueror/Dolphin File browsers are slow and dysfunctional. They are much slower to scroll than their KDE 3 equivalents, and typing the first letters of the filenames there to jump to it is dysfunctional.. I have to use Konqueror 3.5.x to enqueue files into Amarok.
  5. Amarok 2.x is annoying. It looks ugly, the file entries have a huge height, the context tab is gone, and it is generally dysfunctional.
  6. The Konsole tabs are now dysfunctional. There is no button to open a new tab, and the right click on them doesn't do anything. Both of these worked in the Konsole of KDE 3.5.x.
  7. No different wallpaper on each virtual desktop. See this. This is actually a regression from KDE 3.
  8. The New Kopete is Dysfunctional. Its icons are annoying and hard-to-distinguish, I'm playing whack-a-mole with the popups of the MSN plugin, it crashes on exit, the input box is left-aligned in Hebrew, and the Jabber chat is now unusable.

To sum up, KDE 4 is dysfunctional, ugly, buggy and a huge step backwards. I temporarily switched to KDE 3 because KDE 4 made me irritated and edgy.

2008-11-04: Tech Tip: Finding the Email of a CPAN Tester

I'm writing it here so I won't misplace it in a "Know where the ledge is..."-style. If you need to find the id of a CPAN tester based on his report, use the "Find a tester" form on cpantesters.org that allow you to enter the ID ( or the URL) and find the tester. It took me a while to find it today after I misplaced it and needed to contact a CPAN tester, and Google was no help.

2008-11-05: Tech Tip: Sorting RPMs by Installation Date

In order to sort the installed RPMs (RPM Package Managers) by installation date on an RPM-based Linux distribution, use the following command:

rpm -qa --qf '%{INSTALLTIME} %{NAME}\n' | sort -n | \
    perl -MPOSIX -lpe 'my ($t,$p)=split; $_=POSIX::strftime("%Y-%m-%d",localtime($t))." $p"'

You can change the format of the strftime format to something else if you want more than the date.

2008-11-22: Tech Tip: Changing the Dialog Button Order (GNOME-style vs. Windows/KDE style) for KDE 4 Applications

On my Linux system, I had a problem that the order of the dialog buttons on the dialogs of KDE 4 applications were GNOME-style (e.g: [Cancel] [OK]) instead of Windows/KDE-style (e.g: [OK] [Cancel]). The solution I found was to change the Widget style (in "systemsettings → Appearance → Style → Style → Widget Style") from Clearlooks to Plastique. However, no one seems to know for sure what determines the actual button order.

Refer to this bugs.kde.org bug report and this Mandriva bug report for more discussion.

2008-11-24: Tech Tip: Setting MSIE 7 as the Default Browser on Windows

I should note that in the case of this tip, I don't know why it works whereas everything else I tried didn't, but it worked for me. Your kilometrage may vary.

At my home, we have a Windows XP computer with Microsot Internet Explorer 7. After I installed Firefox, and made it the default browser for my account, it became the default for other accounts as well. Furthermore, setting Internet Explorer as the default in its configuration tab (under "Tools → Internet Options") did not work, as Firefox remained the default browser.

The way I resolved it (eventually) was by also enabling the checbox that instructs Internet Explorer to warn on startup if it detects that it's not the default browser. Then and only then, it became the default browser again.

I have no idea why it worked for me, but it did.

2008-11-24: Report on the Last OSDClub Tel Aviv Meeting

Yesterday, the Tel Aviv Open Source Developers' Club held the first meeting in the 2008 Welcome-to-Linux series for the "Mini-Intro" presentation. I arrived to the place early as usual, and there was no one in the class-room except for some students who came for the previous class. They asked me what is going to be there, though.

The attendees arrived shortly afterwards. Then some minutes before the presentation, I called Eddie, who was going to give the talk on the cell-phone and asked him where he was. He said he forgot all about the talk, and asked if I could find a different lecturer instead of him. Luckily, Ori was there and we went over the slides in preparation.

Ori gave a good presentation, in my opinion, and some people from the audience asked him questions, and I and other people gave some commentary. Then, some people arrived while the talk was in progress.

During the break some people asked me about the possibility on soliciting some presentations that they were interested in. One of them told me he came for a presentation about the command line, and was less interested in all the GUI stuff which he was already mostly familiar with. Another asked if we could give a presentation about SELinux, NIS or similar topics related to system administration, because he was studying for his RHCE. We'll see what we can do about these requests.

I forgot the camera at home, so I couldn't take pictures.

Then I ushered everyone out of the room, and those that were interested went to the café. I went with two other guys, who turned out to be Biology students - one of them talked on his cellphone all the way and while we waited for the others to come, but I could talk with the other guy. He asked me which programming language he should learn first and I told him the conclusions from my Introductory Progrmmaing Language essay. We also discussed distributions.

Then Galia and Kobi (who went to re-park Galia's car) found us, and said they had thought we already were sitted in the café. We found a table, but then decided it was too low to be effective, and so switched places. Then we ate, drank and talked. We exchanged many jokes about Apple and stuff like the iRack and "iLand", which is Apple's CEO's private island. (One of our guests made the latter one on the spot.)

In short, a lot of good time was had. Next presentation is next Sunday about the Installation Process. Hope to see you all there.

2008-11-06: The Seven Deadly Sins of Arc

Arc is Paul Graham's dialect of LISP, that aimed to finally create a usable dialect of Lisp. It was released some months ago, and I played with it and even contributed some code to the Anarki Arc effort. In this post, I'd like to note why I think it has proved to be mostly irrelevant and so quickly went into obscurity. People who wish to create their own new popular programming language would be able to use this list to learn what they shouldn't do.

The Seven Deadly Sins of Arc are:

  1. Not Many Exciting Features and Paradigms - Arc may be more brief than other Lisp dialects, and removed a lot of extraneous syntax, but it doesn't seem to sport many exciting features and paradigms. It seems like a Scheme-like-language only slightly better and slightly worse at the same time.

    Graham made a call for advice on his site, looking for ideas no how to improve Arc, but it seems very few of them made it into the final language.

  2. Standard State-of-the-Art Features are Missing - Arc has no built-in Object-Oriented Programing (OOP) system and it also lacks namespaces. Both of these features are part of the absolute minimum expected of any language today.

  3. No Version Control was Used for Arc - when Graham published arc0.tar (more about it later) on his site, it was just a tar. He didn't publicise the URL for a version control system that he used for work on Arc. And in this article he claims he has recently been using a mostly-disconnected computer to do his work, which indicates he's oblivious of the importance of using a remote version control repository for constant work.

    I don't have a lot of confidence in a program that's not kept under version control, and neither do most professional programmers.

  4. No Automated Tests Suite - Arc shipped without any automated tests. If an obscure feature was broken, then Graham could not find out. Nor did he have the specification-feature inherent in the automated test suite. Also the lack of Test-Driven-Development (TDD) for its development is distressing. Maybe that's what Graham calls exploratory programming

  5. Basic Usability is Missing: there was no way to do ./arc filename.arc to execute a file directly from the command line. I had to add it to Anarki Arc, myself.

  6. Weird Versioning - the first released Arc version was arc0.tar. The second was arc1.tar and the third was arc2.tar. Normally, one would expect that version 1.0 of an open-source project would be very mature, much less version 2.0, but in Arc's case they were incremental releases, released after a short time.

    So the versioning used by Graham is strange, not very conventional and confusing.

  7. Brevity Taken to Non-sensical Extremes - Arc tries to be as brief as possible, which is good, but it sometimes defies common sense and convention. For example the (not) function in Arc is (no) so the code reads (if (no a) (pr "a is false")) (is that even English?).

  8. Bonus! No line numbers in errors - that's right, when something goes wrong in an Arc progam, then you don't get a line number where it occured. How can one write anything but toy programs this way? Of course, given that executing files from the command line was not supported, it may not be very surprising.

In short, if you were hoping for a usable and modern dialect of Lisp, then Arc is not the answer, and won't be without a lot of work. I was told Clojure is much better than Arc, which I have little experience with, but may be better. (Although the fact that it targets the JVM is somewhat of a turnoff).

But Arc, as it currently stands, is not the answer. Graham had a lot of time to invest in polishing Arc before its release, but it's not polished at all. The initial flurry of interest in Arc seems to be largely gone now, and it seems unlikely that it will ever mature into something professional and usable. Arc gave me some ideas on how to create a more usable Lisp, but it otherwise lacked essential feature and is still lacking. I'm going to look elsewhere for a usable Lisp dialect.

2008-12-20: Tech Tip: Enabling Audio Notifications in KDE 3 Applications

For a long time, I didn't have any audio notifications in KDE 3 applications, including not in such applications as Kopete, where people keep sending me nudges. Today, I was finally able to fix it. Here's how I did it.

Run kcontrol, and go to "Sound & Multimedia → System Notifications". Now notice the "Player Settings" button on the bottom-right corner of the screen, and press it. Switch the option to "Use an external player" and enter "/usr/bin/mplayer", or a similar capable non-interactive audio player. Press the "OK button" and say "Apply" in the main KControl window.

Sound notifications should now work, or at least they do on Mandriva Linux Cooker, where the combination of PulseAudio, artsd, and other curses like that messed up KDE completely. Hope it works for you.

2008-12-26: Eventful Week

The week before this one (starting on 14-December) was eventful, with one event on each day. On Sunday, I had a Tel Aviv Linux Club/Welcome-to-Linux meeting. Ori Idan gave his now regular "Living in the Open-Source Community and How to Get Help" talk. There were about 10 people there (less than previous times we held the series) but it went well. After the meeting, three of us went to the local university café. At the end of the meeting Galia spent a lot of time showing off the "cool factor" of her Mac computer. There were some cool things there, but not enough to convince me that Apple wasn't Evil. One thing she showed was that you can take a photo using the built-in-camera and drag-and-drop it to an MSN conversation to send it. (As opposed to saving it to a file and then sending the file - the "Windows way".) This seems sensible enough to have been done in Linux and Windows already (didn't check, though), and on the other hand, I couldn't open the photo that she sent from the Mac on my Linux Machine.

In any case, the next day, Monday, I gave a Perl lesson on MSN Messenger to my Perl pupil. I find it hard to interact with her remotely without the proximity, and due to the fact I don't have access to her computer. Another thing that made it more difficult was the fact that I didn't have sound notifications on Kopete which I solved since then.

On Tuesday, I went to the Open University center near Tel Aviv University to meet my pupil for paying for the MSN lessons, and then went to the doctor to get presecriptions for medications.

On Wednesday, I had a job interview. It went pretty well, but I still don't know if I got the job.

On Thursday, I went to a meeting of the Herzeliyah Linux club. I got there early and spent some time chatting with people who came to see the presentation. I didn't even notice the talk started late, until the organiser and the lecturer arrived. The talk was about the various ways to boot a Linux system, and was nice. I returned home after the talk, because no one wanted to go to a café or restaurant.

I hoped I could rest during Friday, but instead ran into several bugs in the comptuer. Kopete of KDE 3 stopped working for some protocols, due to a missing qca1, which turned out to be gone for good from Mandriva Cooker (as part of their general depracation of KDE 3). I had to download the old .rpm from the Mandriva repository, and tweak it to build qca1. I should note that now KDE 4's Kopete is much more stable than it used to be, and is actually usable, so I'm using it.

Then after I tried Amarok 2.0, I discovered that its album cover in the middle pane, was too large, and that it won't play Module files. On Saturday, I discovered that they dropped support for out-of-process scripts, which means it'll be less trivial to port the Perl script I wrote to Amarok 2.x. If I would want to do so, I'll need to re-implement it in QtScript/JavaScript, which I heavily detest and which will require a lot of extra research.

Finally, I had problems building perl-XML-SAX-ExpatXS, which doesn't exist on Mandriva. On Saturday, I finally got it working, after looking at other XML-SAX packages on Mandriva. This reminds me that I still need to upload the new RPMs I created.

That's it, I guess. This week was less eventful, and I even got some coding done (more about it later hopefully).

2008-12-26: Recent Perl-Related Hacktivity

It's been a while since I programmed intensively until a few days ago, but that's changed recently. Here are some things I worked on recently:

  1. I worked on my Perl/XS tutorial-of-sorts. You can find it in its repository. The first thing I did was re-organise the repository so there won't be a several sub-dir in the trunk for each XS function I added to the module, which I use to incrementally introduce the person to XS. Then I added a function, which I encountered some problems with, and resolved them.

  2. I worked on Config::IniFiles, and uploaded a new verstion to CPAN. The good people at the modules@perl.org mailing list resolved the problems we had with its maintainenace status, and the upload got indexed. There are more fixes in the trunk as of now.

  3. I worked on File-Find-Object-Rule which is my spin-off of File-Find-Rule that uses File-Find-Object instead of the core (and philosophically problematic) File::Find module. I'm breaking the File-Find-Rule API because it relies too much on File::Find'isms, so I would have to rename it.

    While I was working on File-Find-Object-Rule, I found a bug in File-Find-Object which I fixed, and released a new version to the CPAN. It will be sometime before File-Find-Object-Rule is ready for release, because while the tests now pass, it still requires a lot of work.

2009-01-09: Problems With SVK

Many months ago, I discovered several huge problems with SVK (a distributed version control system based on Subversion) on my system: it no longer propagates the local changesets to the remote repositories, and I can no longer mirror a new remote repository. However, there was no way I could get any help with it. The chief SVK maintainer, CLKAO has been extremely unresponsive. I could almost never reach him on IRC or on email, and when he does respond and I respond to him, he doesn't respond back. I met someone on IRC who seemed somewhat knowledgable, but he could not offer anything better than a workaround where I'd lose my entire local history.

I offered a bounty of 50 USD to the first person who will fix this problem, but no one took the offer. In the svk-users mailing list, many (most?) questions go answered and unresolved. A new version of SVK (2.2.1) was recently released, and I tried to prepare an RPM for it on my Mandriva Cooker system, hoping it will fix the problem, but the tests failed there.. Fearing something is broken, I decided against proceeding. The SVK bugs queue contains 61 (!!) open bugs that weren't addressed or closed.

Best Practical, the company that hired CLKAO to further his work on SVK, are offering commercial SVK Suppot starting at 5,000 USD per year. (Which I cannot afford) The original reason they hired CLKAO was because they were using SVK and wanted to promote it and sponsor its development. However, it now seems that they just decided to capitalise on it, and kill the pro-bono/community channels of support.

In short, I'm stuck with a lot of data on my hard disk that I cannot propagate to the outside. I'll be appreciative of any way to fix SVK so I can propagate it to the outside, or a recommendation of a different tool that can do it for me based on the SVK local Subversion repository. In the meanwhile, my advice to you is that until the attitude of the core SVK developers changes, and the bugs in SVK are fixed, then you should stay away from SVK, and not depend on it.

If you're looking for a way to decentralise a remote Subversion repository, then you can look at git-svn , bzr-svn , and hgsubversion . But you should not depend on SVK.

2009-01-10: Reports on Recent Open-Source Meetings

I attended four open-source-related meetings this week. On Sunday, Gabor Szabo, gave a presentation about Padre, the Perl IDE, which he described as a "Notepad with a big ego". The presentation was originally titled "wxPerl programming and Padre, the Perl IDE", but for better or for worse Gabor did not talk about wxPerl at all. The presentation was mostly focused about what Padre can do now, Goals for next releases, and a call for volunteers. I found the most impressive part to be the fact that it had some rudimentary Perl 5-syntax-aware refactorings, like lexically change a variable name.

10-20 people came to the meeting, but many of them left before it was done. (don't know/remember why), and only 4 of us went to the cafe afterwards. Still, a lot of fun was had.

For another review of the meeting see xsawyerx.

Next on Wednesday, I attended the Perl-Haifa meeting. I took the train to Haifa, and after I arrived at Matam, I met a regular Haifux attendee on the way, whom I haven't seen in a long time and we talked about stuff. Not many people came to the meeting, but one or two ended up late.

Uri Bruck presented about Perl in Software Art. He presented a lot of unorthodox art which was software based. I daresay the projects he presented there were more strange than I expected it to be. One problem we had was with the intermittent Wireless LAN connection on his computer, but that wasn't too bad.

After that, Shmuel Fomberg gave a presentation about Data-ParseBinary. The story of the module is interesting. I originally attended a presentation about PyConstruct given by one of its developers on a Python-IL meeting. Then, when I helped Alan Haggai Alavi with the (Perl/CPAN) Archive::Zip TPF grant, I suggested that as part of the grant, he'll write a PyConstruct-based parser for .zip files, to help in analysing different archives and see where Archive::Zip goes wrong. Now Shmuel saw it mentioned in the grant, and liked the idea, so he ported it to Perl as Data-ParseBinary. And then, Data-ParseBinary was good enough that Alan Haggai decided to use it as the parser for the .zip format for his Archive::Zip grant.

Before the presentation, I talked with Shmuel, and thanked him on behalf of Alan. He said that he knew of Alan's use of Data-ParseBinary based on a web-search and that he was his only known "customer". Even Shmuel himself did not make an active use of Data-ParseBinary in production. (But he said he's planning to soon).

In any case, in the talk, Shmuel gave a relatively comprehensive overview of the module's features, and how to use it for many use-cases. One attendee kept asking him questions about why he did things this way instead of the other, and Shmuel answered with the right way to do things. He said that Data-ParseBinary's interface was originally based on Construct's, and so many of the problems the attendee found with it were derived from there. We also had some fun correcting the pronunciation of certain English words.

Data-ParseBinary seems very cool and useful in case you need to analyse or modify binary formats.

On Thursday, I went to a Herzelinux (Linux in Herzeliyah) meeting. Eitan Isaacson gave a presentation about accessibility in GUI software in general and about Accersiser, which is a tool he wrote in order to help test the accessibility of programs. Accersiser basically lived at the lowest layer of the accessibility API and acted as an accessibility aid that shows what is available there. The presentation was interesting, but from my impression, it seems that Accersiser is too crowded with information so one cannot see the forest from the trees.

He mentioned KDE/Qt applications, and said that they don't yet interface with the accessibility framework used by GNOME, Java and Mozilla because it is based on CORBA, and they don't want to have to depend on it. They said they're planning to add a DBUS-based transport, which will hopefully enable Qt and KDE to interface with it as well.

After the Herzelinux talk, someone drove me to the Israeli Ruby meeting at a pub, nicknamed "Beer on Rails". Since I ate a lot of refreshments at Herzelinux, I only ate some spicy fries there (which were very good). I engaged in a few talks there, but was soon left out because I sat in the corner between two conversations that I could not hear well. One talk that I remember was that someone told me he tried to use Vim instead of Emacs, but found it frustrating that while in Insert mode he could not move to the previous or next characters without pressing Escape. I told him to use the cursor keys, but he said he'd rather not because they are too far away. Then I told him that he can bind certain key bindings to do something else while in Insert mode, but he said "Yes, but that would no longer be vi.".

I'm personally not too bothered by this, because I have no problem using the arrow keys for certain tasks, and because I find that when working on editors on other systems, I only do minimal tasks until I can set up an environment that I like like I am used to. But I can relate to such problems.

In any case, I eventually got too tired, paid, and took the bus home. I hope the next meetings will be in a different format, and somewhat earlier during the day.

So it's been a busy week, and a lot of fun was had. Bye for now.

2009-01-14: Shavin' Another Second

I've been working on File-Find-Object for quite some time now. File-Find-Object provides an alternative to File::Find, which is a built-in module in the perl 5 distribution, with several major philosophical limitations. What both modules do is traverse a directory tree and allow the programmer to do something with all the results, similar to the Unix shell's "find" command.

My obsession with working on File-Find-Object started in this perl5-porters thread, where I also claimed that File::Find was probably IO-Bound, and so making it iterative and capable of instantiation (which is partially what File-Find-Object is all about), will not harm its performance much. As it turns out, I was not entirely right.

About two weeks ago, I decided to finally benchmark File-Find-Object. The original motivation for this was a small change I did and perceived as an optimisation in which I reduced the number of hard-disk-reading system calls that it used to a minimum. I wrote a few shell scripts that timed the old version of F-F-O versus the new one on my ~/progs directory (which contains many sub-directories), after it was in turn traversed by the shell "find" command.

Both version took somewhat over 2 minutes to complete the scan, which seemed pretty good to me. However, when I benchmarked a similar script using File::Find, I found out that it finished the scan in under 5 seconds, making it 24 times faster than File-Find-Object. I didn't want F-F-O to be so much worse, so I decided to work on optimising it.

My methodology in optimising File-Find-Object was to profile it using Devel::NYTProf (the Perl profiler with the best reputation, and one which I can now recommend as well), find problematic spots, try to optimise them, and then benchmark the old code against the new to see if there were any improvements, and if so - commit.

Among the notable changes I performed were:

  1. The first thing I noticed was that a lot of time was spent in Class::Accessor code. To remedy this, I switched to Class-XSAccessor, which resulted in saving 40 seconds of runtime. The other changes I made were less substantial individually in terms of run-time saving.

  2. Added a flag to $self (the object instance) with whether or not the stack of the directories through which the traversal is in progress is full. It was used inside several dynamically-generated functions where I delegate to top and non-top versions.

    To save some extra speed, the flag is checked using exists which should be a bit faster than checking for the truth of the value.

  3. Saved 2.5 seconds of run-time by implementing a commonly used function (that was reported to consume a lot of time) in a direct manner, instead of the top/non-top delegation, and using some judicious optimisation using the "||" operator.

  4. Saved many seconds on eliminating calls to _copy methods that flat-copy array references (and that are also generated dynamically). Often, the copying was not needed at all.

  5. Eliminated excessive calls to File::Spec's functions by caching the results.

  6. For each node, File-Find-Object performs two actions in a specific order depending on the "depth" instance-wide parameter. Previously, handling them involved weird returns of arrays, and indexes-mangling (which I admit that I originally implemented in my endless refactoring). This was changed to a simple array of method names, from which values are extracted.

  7. Re-arranged the order of operations to prevent an if (_is_top()) { ... } conditional, that was done in every iteration.

  8. Originally, the main object served as the controlling object for the top directory, and dir_stack->[0] was the directory below it. So when checking for fatherhood of a directory, one had to return the main object for the father of dir_stack->[0]. In a series of commits, I moved it to be a separate object, as well as occupying the first step in the directory stack.

    As a result, I was able to cache the "_current" object inside an accessor, because doing so in for the main object before would have resulted in a circular reference, which would have caused a memory leak.

Thanks to all these optimisations, File-Find-Object now runs at about 30-40 seconds.

Yesterday I also benchmarked the first version of File-Find-Object before I took over its maintenance, and started refactoring it, and it runs at 17 seconds for the same test.

So what are the Conclusions?

  1. Accessors in Perl may incur a large overhead if Class::XSAccessor (or similar) are not used.
  2. Extracting methods excessively can incur a heavy penalty on performance of Perl programs.
  3. Traversing a directory in Perl can be very CPU-bound.
  4. Many small optimisations can together yield a huge benefit. Bill Raymond says in this message to the fc-solve-discuss mailing list that "I achieved my fast times by multitudes of 1% reductions". I can attest to it here.

As indicated by the benchmarks, the road to a faster File-Find-Object is not over, and I still have some ideas for improvements in mind. But it's still much better than it used to be, and I can be proud of it.

The title of this post was inspired by Ido Trivizki's "Shavin' another bit" presentation given at YAPC::Israel::2004, and which I attended and enjoyed.

2009-02-05: Tech Tip: Resolving JDBC+MySQL exception.

I recently tried to build a Java-based project that is using MySQL. I got the following error:

BUILD FAILED
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 1 ms ago.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2104)
        at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:729)
        at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:46)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:283)
        at org.apache.tools.ant.taskdefs.JDBCTask.getConnection(JDBCTask.java:319)
        at org.apache.tools.ant.taskdefs.SQLExec.execute(SQLExec.java:429)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
        at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:357)
        at org.apache.tools.ant.Target.performTasks(Target.java:385)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
        at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
        at org.apache.tools.ant.Main.runBuild(Main.java:758)
        at org.apache.tools.ant.Main.startAnt(Main.java:217)
        at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
        at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:519)
        at java.net.Socket.connect(Socket.java:469)
        at java.net.Socket.(Socket.java:366)
        at java.net.Socket.(Socket.java:209)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
        at com.mysql.jdbc.MysqlIO.(MysqlIO.java:276)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2027)
        ... 27 more

After many experimentations, I discovered the problem was that MySQL on my local machine listened only on a UNIX-domain socket and not on TCP, which is what JDBC tried to connect with. After commenting out the "skip-networking" option from /etc/my.cnf (which is the location of the MySQL configuration file on my system) and restarted mysqld, everything worked.

2009-12-24: Web Application Frameworks

Ido Kanner writes in his blog (in Hebrew) about web-application framework. I am replying here, because his blog does not provide a Preview plugin for comments (due to limitations of wordpress.com). Aside from the alternatives he mentioned for Perl, I am aware of the following:

  1. CGI::Application - very basic functionality, but also very flexible.
  2. CGI-Application-Plus - backwards compatible with CGI-Application, with some improvements.
  3. CGI-Builder - inspired by CGI-Application, but not backwards compatible. (by the same author as CGI-Application-Plus).
  4. CGI-Prototype - another lightweight framework based on prototype-based OO (like Self, JavaScript, Io, etc.)
  5. Catalyst - a web framework that supports many different view/model backends and has many plugins on CPAN. Seems to be the most popular.
  6. Jifty - should be more RoR'ish than Catalyst, and also have a concept of continuations. Only supports HTML::Mason and Jifty::DBI.
  7. Gantry - don't know a lot about it.
  8. AxKit - Perl and XML-based for Apache
  9. WebGUI - a Perl-based framework mixed with a content management system. (I had problems getting it up and running on my Mandriva Cooker system, after installing it from source, and it also made my Apache instances consume much more memory, so I disabled it.)
  10. Mojolicious - started as a fork of Catalyst. Don't know too much about it, but you can find more from the site.

I also had some experience with the PHP-based Symfony (review in the link).

Ido also mentions some JavaScript based "frameworks" like jQuery, or Mootools, but naturally they are not an alternative to server-side frameworks, but rather complementary.

Moreover, sometimes Content-Management-Systems can be used in a similar fashion to web-development frameworks, and there are many of those too.

2009-02-05: Tech Tip: Getting Rid of Visual Artifacts in KDE 4

I recently (or not so recently) discovered that the KDE 4 desktop on my Mandriva Linux Cooker system displayed some bad visual artifacts when loaded as the default desktop. It didn't happen on fresh UNIX accounts. As I discovered, even moving away the entire home directory (/home/shlomi) to a new place, and creating an empty directory did not resolve the problem. Nor did, copying the files from a good account.

I finally discovered that the problem was in /var/tmp/kdecache-shlomi ("shlomi" is my username), which after exiting from KDE 4 and removing solved the problem for good. Of course, all of this took many hours of trying to incrementally move away files and see if it solves the problem. I cannot guarantee it is the only cause of problems similar to that one, but it helped me.

2009-02-10: Oppose the Patent Encumbered IETF TLS Authorisation Patent

Glyn Moody reports on an FSF call-for-action for rejecting a patent-encumbered standard. It needs to be done today, and this is what I wrote to ietf@ietf.org:

Please don't accept the patent-encumbered standard for TLS
authorisation. Doing so will allow a single entity to control the use
of the standard and will weaken the authority of the IETF.

Please follow the instructions in the links and send them an email. You also need to approve of the confirmation email.

2009-02-11: "Windows Refund Howto" on the Tel Aviv Open Source Club

Hi all!

The Tel Aviv Open Source Club will host a talk by Zvi Devir about "Windows Refund HOWTO" - how to get a refund for a copy of Microsoft Windows that ships as part of a new computer - on Sunday, 15-February-2009 (next Sunday).

The meeting will take place at Tel Aviv University, at the Schreiber Maths&CS building, room 008 on 18:30. So mark your calendars. For more information can be found the club's homepage and on the meeting's wiki page.

With any other questions or problems, feel free to contact me.

2009-02-13: New Essay: "Optimising Code for Speed"

I published a new essay about optimising code for speed on my site, which you are welcome to read. Quoting from it:

We've all seen the fact that while computers got faster, software has often become slower to run unless the hardware is upgraded. The so-called "Gates' Law" claims that commercial programs decrease in speed by half every 18 months, due to various reasons. It is well known that the various versions of the DOS operating system ran adequately on a PC XT's and 286's and that a Intel 386 was a "lean and mean DOS machine" as a certain journalist claimed back then. On the other hand, Microsoft Windows 3.0 and Microsoft Windows 3.1 already required a fast 486 computer to be ran comfortably, while Windows 95 was barely usable there and needed a Pentium computer. Windows XP already ran slowly on a Pentium machine and required a high end Pentium III or Pentium 4 computer. Windows Vista requires even more hardware resources than Windows XP, up to the point that many computers in use today cannot run it comfortably.

Now, while software simulations that run directly against the CPU and memory (and possibly hard-disk) are still running much faster than before, the responsiveness of the system itself does not seem to improve much.

I used Wikibooks to work on this article. Except for a few minor changes from other people, I was the only contributor. One thing that disappointed me was that wiki2xml (a MediaWiki extension that can convert the text to DocBook/XML and other formats) blissfully ignored all internal or wikipedia links ([[…]]), rendering many links in the article useless. This required that the article be extracted from the wiki by cleaning up the HTML of the page.

2009-02-16: מדוע אתה מפתח את זה בחינם?

ארתיום שואל "מדוע אתה מפתח את זה בחינם?" ואני החלטתי לענות לו כאן (ייתכן שאתרגם זאת לאנגלית יותר מאוחר). מכיוון שיזמתי ו/או תרמתי לפרוייקטים רבים הרי שלא אוכל לכסות את כולם ברשומה זאת, ואתמקד רק בבולטים שבהם.

התרומה הראשונה שלי של תוכן פומבי וזאת שהשקעתי בה את כמות הזמן הרבה ביותר היא אתר הבית שלי. אם לדייק, הרי שאני מרוויח ממנו מעט כסף משום ששמתי שם פרסומות. בכל מקרה, נכון לעכשיו, שני הצ'קים של כ-100 דולר כל אחד שקיבלתי מגוגל לא מתקרבים אפילו לכסות על כמות ההשקעה שהשקעתי באתר, ואף שאני אשמח שבשלב מסוים אצליח להתקיים מהפרסום באתר בלבד, הרי שנכון לעכשיו, זה לא קורה. בכל מקרה - מדוע אני עובד עליו? הסיבה הראשונה היא שאני נהנה לעבוד עליו ואני לומד מכך הרבה. הסיבה השנייה היא שאתר עשיר כל-כך מקנה לי מוניטין רב והרבה אנשים לומדים עליי כך ומתרשמים מיכולותיי. הסיבה השלישית היא שאני לפעמים מקבל פידבק (בדרך-כלל אוהד) מאנשים על דברים שפרסמתי באתר, או מתפתחים דיונים מעניינים בעקבותיהם. והסיבה הרביעית היא שאני לומד מכך הרבה.

בנוגע ל-Freecell Solver הרי שזה התחיל מנסיון שעשיתי כדי לראות אם מספר רעיונות שהיו לי כיצד לפתור את פריסל אכן יעבדו בפועל. הם עבדו עד כדי אפסילון גדול מאוד והייתי כל-כך מרוצה מהעובדה הזאת, שהחלטתי לפרסם גרסה ראשונה (0.2.0) של התוכנית באתר שלי בטכניון וכתבתי הודעה עליה ב-Freshmeat.net. המשכתי לעבוד על הפותר מתוך עניין, מכיוון שקיבלתי עליו משוב רב מאנשים שניסו להשתמש בו, מכיוון שלמדתי ממנו הרבה, ומכיוון שאני יודע שהוא שימושי לאנשים רבים וגם אני עושה בו שימוש. בנוסף Freecell Solver סיפק לי השראה רבה וחומר רב לדיון במאמרים ובמסות שלי כך שגם בזה הוא עזר לי.

ישנם גם המודולים שלי ב-CPAN, שזה קצת חטא להכליל את כולם בקבוצה אחת כי הם משמשים למטרות שונות ומגוונות. את רובם כתבתי בעצמי (עם מעט תרומות מאנשים אחרים), וחלק קטן מהם אימצתי מאנשים אחרים. מדוע אני עובד עליהם? כי אני אוהב לכתוב פרל, כי אני מוצא את עצמי מעורב בכל הקהילה השלמה והתוססת של פרל בארץ ובעולם, וגם מזה למדתי בעבר. חלק גדול מהמודולים היו כאלה שהייתי צריך לשימושי האישי והחלטתי לשחרר לציבור, כך שממילא הייתי צריך לכתוב אותם.

תרמתי בעבר גם לפרוייקטים כמו סאבוורז'ן, ו-גימפ. הסיבה שתרמתי לסאבוורז'ן היא שהתחלתי לעשות בה שימוש, אחרי שלדעתי האלטרנטיבות שהיו לה אז (Arch, Aegis וכו) היו מסובכות, מבלבלות ומעצבנות ואחרי שהיה לי נסיון רע עם BitKeeper הלא חופשי. כאשר עבדתי על סאבוורז'ן התרשמתי לטובה מהיחס שלהם לתורמים, שזה דבר שפרוייקטים רבים של קוד פתוח יכולים ללמוד ממנו. הפסקתי לתרום לסאבוורז'ן מכיוון שנוכחתי שהם היו במצב כזה טוב שהם לא זקוקים כל-כך לעזרתי, וגם נוכחתי שסאבוורז'ן לא גורם לי לבעיות מיותרות, אז החלטתי לעזוב בשקט מבלי שאיש ירגיש. במידה שתהיינה לי בעיות איתו בעתיד אני לא שולל שאשקיע זמן בלתקנן, אבל כרגע אין לי כוונה לחזור להיות תורם פעיל.

לגימפ תרמתי מפני שהיא לדעתי תוכנה מדהימה, ורציתי לקדם אותה. הטלאי הראשון שלי עבר מספר גלגולים ושכתובים עד שהתקבל בסופו של דבר, אבל זה היה לגיטימי מבחינתי. אחר-כך כתבתי טלאי פצפון ל-gimpressionist ומבט קצר על הקוד שלו הביא אותי להיווכח שמצבו בכי רע, והשקעתי זמן רב בלנקות אותו. מה שמשך אותי היה הרצון לקדם את התוכנה.

אז זהו בעצם. אם אתם רוצים להצטרף לשרשור אז בשמחה אקרא את מה שתכתבו.

2009-02-16: Should We Include a Better Alternative to File-Find in the perl5 Core?

chromatic asked for some link/follow-ups love to his "Modern Perl Books" blog, so here's a post inspired by one of his Modern Perl posts where he complains about how inane the File-Find interface is. I personally share chromatic's sentiments about File-Find, and have for many years. After posting to perl5-porters about its shortcomings, it was agreed that fixing them in File-Find may hurt its performance, and so was not acceptable without forking the code or rewriting it. At the end of the thread, I mentioned that I found File-Find-Object on CPAN, and that I intend to contribute to it and improve it. That's what I did, and I eventually took over its maintenance from the originator.

File-Find-Object solves all the major shortcomings of File-Find that I'm aware of. But this post is not specifically about File-Find-Object. If someone will come up with a better alternative to File-Find than File-Find-Object is, then I'll support the new code.

The point is that I think that, as a statement of intent, and of general disgust from File::Find, a different module with a better interface and a saner and more modern philosophy should be shipped as part of the perl 5 core. It's not the first time that a core module in perl 5 was "complemented" with a better alternative: CPANPLUS.pm was added to the core in addition to CPAN.pm , Module::Build was added as an alternative to ExtUtils::MakeMaker, and recently Test::Harness was augmented with TAP::Parser/TAP::Harness/Test::Harness-3 . I don't see why File::Find should be any different considering the fact it has several philosophical usability problems which cannot be easily overcome (it cannot be instantiated, cannot be easily interrupted in the middle, and does not have an iterative interface).

I don't suggest we change the File::Find API. That will wreck havoc. But I think we should include another module and put a big and visible notice at the beginning of File::Find saying "File::Find is deprecated and should no longer be used! Use $better_file_find , that ships with perl as of perl-5.12.0 or available from CPAN for earlier versions, instead."

What do you think?

2009-02-21: How I Recovered Data from a Hosed SVK Repository

Many months ago, I noticed that SVK, a version control system based on Subversion, stopped propagating changesets to the remote Subversion repositories. Furthermore, I was unable to mirror other remote repositories. I wrote about it, asking for help:

All of this has been left unsolved until yesterday when I decided that I had enough and wanted to finally resolve this problem. My first idea was to try to write a script that will propagate the changesets using the Subversion APIs, so I looked at the SVK code trying to see where exactly it happens. I found the SVK code hard-to-follow, so I first contemplated fixing what caused it to report an error, but was also unable to.

Then I joined the Freenode IRC network and went to the #svk channel. There was only one person there who tried to help me, and he was intermittent in helping. Then I joined #perl and casually mentioned that I thought SVK sucked (just to lose some steam), which prompted mst to try to help me. After interrogating me he suggested that in order to recover the changesets from SVK, I should follow the following steps:

  1. Find out what is the last SVK revision which still propagated changesets to the Subversion repository. This can be found using the commit message in the svn log.
  2. Revert (using a selective svnadmin dump and then an svnadmin create ; and svnadmin load) to the older version of ~/.svk/local that corresponds with the revision number of the commit. (While naturally keeping a copy of ~/.svk/local/ and its dump in a safe place.)
  3. See that a simple change in the svk checkout (like svk mkdir to-delete-`date +%s`) will propagate to the remote repository.
  4. Revert that change using the Subversion client directly.
  5. Revert to the working-but-old copy of the SVK repository, and apply a filtered stream of the dumps (using svndumpfilter) that will exclude changes from the //mirror part of the SVK repository.
  6. After all that is done, checkout the trunk and do svk push from there.

Of course, the devil is in the details, and it took me a lot of experimentation until I got to something working. Facts I've discovered:

  1. svndumpfilter needs a dump that was generated without the --deltas flag (and possibly without --incremental either. Otherwise it will complain.
  2. I couldn't seem to get svndumpfilter exclude mirror to work and instead ended up using svndumpfilter include local
  3. The --drop-empty-revs and --renumber-revs flags to svndumpfilter which seemed like a good idea confused svnadmin load, and everything worked after I removed them.
  4. Using tar -czf and tar -xf can really speed up restoring older versions of a Subversion repository because svnadmin load is pretty slow.
  5. I ended up taking a range of repositories from the pristine revision to "HEAD" and then manually editing out the first revision. Maybe it was unnecessary.

After all that, "svk push" worked and propagated 800 changesets to my homepage's trunk in 8:20 hours (must be a record). Since I only have an ADSL connection with 25 KBps upstream, and the Subversion service is located in Taiwan and I'm living in Israel, it took a while and I left it overnight to run.

Today, after I woke up, I saw that the operation ended up successfully. I was able to build an up-to-date version of my homesite from the Subversion sources, and am mostly ready to stop using SVK.

Again, I'd like to thank mst for suggesting this ultra-cool idea for recovering my data from SVK. It took me the whole of one day, but now I'm finally free, and my data is out there and safe. I'm going to use plain-old-Subversion to work against it, until and assuming I'll get the hang of hg/bzr/git/whatever. But at least I no longer have to use a mostly-unusable SVK.

Hackfully and happily yours, Shlomi Fish.

2009-03-02: CPAN Hacktivity Report

After my last post about File::Find alternatives, chromatic responded, and said that it was opinion that the core should not be bloated further and that he would prefer if File::Find did not exist in the core in the first place. I brought the relevant links on perl5-porters, and David Cantrell wrote a patch to add a warning to File::Find, while disabling it from all the places in the core that use this module. As other people noted, we should practice what we preach, and so if we decide that File::Find is deprecated, then we should include a different alternative in the core, and adapt all the existing code to make use of it instead. Someone even suggested implementing File::Find in terms of the new module.

Eventually, I realised that the best solution would be to include a note in the File::Find documentation about its limitations, and refer the interested people to the better alternatives on CPAN.

In any case, since my "Shavin' Another Second" post, I implemented several more small optimisations (and general code cleanups) to File-Find-Object, and it now takes about 27 seconds to run in my test run. One thing that surprised me was the fact that after a relatively large optimisation and overhaul to the code, all the tests pass immediately after the first clean run. Usually, the code required more persuasion to make it work.

I also began File-Find-Object-Rule, which is the File-Find-Object-enabled-fork of File-Find-Rule, which provides a more convenient interface above File::Find. The first release of F-F-O-R had missing dependencies, and so consistently failed all CPAN testers' reports. Version 0.0101 was better, but its tests failed on MS-Windows, due to the fact that the File-Find-Rule tests excepted UNIXish paths (which File::Find happily provided, but File-Find-Object doesn't). This was fixed in File-Find-Object-Rule version 0.0200.

Version 0.0200 is still a mostly "smallest-effort" port of File-Find-Rule, so now comes the interesting part of truly making use of File-Find-Object's advantages, which may involve extending the latter API. I also would like to create a documented API for File-Find-Object-Rule, so the code won't be tightly coupled with File-Find-Object.

Speaking of File-Find-Rule, you may find this recent discussion on use.perl.org interesting.

Except for that I also did some work on XML::RSS, like integrating some patches that someone else sent me, doing QA, and answering an email someone sent about one of its limitations (which I'd like to fix.).

I also found a bug in ack, and submitted a patch fixing it.

Finally, after my home-site's remote repository good condition was restored, I decided to try to build it from scratch and correct the process that has gathered a lot of rot since I last did it. I discovered a lot of problems with installing the necessary CPAN modules that I wrote in part for the homepage, but I'm waiting on a reply to this question about Module-Build on its mailing list, before I can correct the build.

2009-03-11: A New "Perl 6 Tricks and Treats" Newsletter by Gabor Szabo

Gabor Szabo announced a new newletter he started for "Perl 6 Tricks and Treats". I am subscribed to his "Test Automation Tips" newsletter, and am very happy from the quality of his submissions to it, so I can recommend Gabor's writing first hand.

Please subscribe if you're interested in Perl 6, and please help publicise the announcement of this newsletter by sending a message to your local Perl mongers group, other FOSS-related groups or on your blog.

2009-03-23: Tech Tip: Installing CPAN Distributions without .pm Files

I recently needed to install cpan_bot from the CPAN (the Comprehensive Perl Archive Network). However, it does not contain any .pm's (Perl Module files) - only a script. All my attempts to install it failed until I tried this:

perl -MCPAN -e 'CPAN::install("ZOFFIX/cpan_bot-0.11.tar.gz")'

So you need to use the author's ID followed by a slash and the tarball's filename.

2009-03-26: Fixing the Amarok 2.x Playback of Module Files

This is a behind-the-scenes story of a patch I worked on. The problem I encountered was that Amarok ( a KDE-based music player) version 2.x could not play Module files, such as ".mod", ".s3m", ".xm" or ".it". The symptom was that dropping them in the playlist did not get them enqueued. Talking about it on IRC, I got a lead that led me to investiage the following code in Amarok's src/EngineController.cpp:

    // Filter the available mime types to only include audio and video,
    // as amarok does not intend to play photos
    static QStringList mimeTable =
        Phonon::BackendCapabilities::availableMimeTypes().filter(
            "audio/", Qt::CaseInsensitive ) +
        Phonon::BackendCapabilities::availableMimeTypes().filter(
            "video/", Qt::CaseInsensitive );

    const QString mimeType = item.mimetype();
    const bool valid = mimeTable.contains( mimeType, Qt::CaseInsensitive );

So when I ran it using the gdb debugger, while using kde-devel-gdb (a set of gdb macros to facilitate debugging KDE applications), I dumped "mimeTable", and, after grepping it, found that it indeed did not contain the MIME types associated with module files. Then I investigated why I could still select and play module files in KDE 4's Dragon Player, and realised that while it used the mime types list as a file-chooser dialog, it did not enforce a file that was not associated with any of them from being played.

Having realised that, I set out to see why the list of mime types was not complete. I first tried to see what is the list that phonon returns as is. So I wrote (or more actually copy and pasted) the following code:

#include <iostream>

#include <Phonon/VideoWidget>
#include <Phonon/BackendCapabilities>

#include <KAboutData>
#include <KApplication>
#include <KCmdLineArgs>
#include <KLocalizedString>


static KAboutData aboutData( "test-phonon", 0,
        ki18n("Dragon Player"), "0.1.5",
        ki18n("A video player that has a usability focus"),
        KAboutData::License_GPL_V2,
        ki18n(
        "Copyright 2006, Max Howell\nCopyright 2007, Ian Monroe"
        ),
        ki18n(
        "IRC:\nirc.freenode.net #dragonplayer\n\n"
        "Feedback:\nimonroe@dragonplayer.org"),
        "http://dragonplayer.org" );

int main(int argc, char * argv[])
{
    aboutData.setOrganizationDomain( "mpris.org" ); //for DBus
    aboutData.addCredit( ki18n("David Edmundson"),
    ki18n("Improvements and polish") );
    aboutData.addCredit( ki18n("Matthias Kretz"),
    ki18n("Creator of Phonon") );
    aboutData.addCredit( ki18n("Eugene Trounev"),
    ki18n("Dragon Player icon") );
    aboutData.addCredit( ki18n("Mike Diehl"),
    ki18n("Handbook") );
    aboutData.addCredit( ki18n("The Kaffeine Developers"),
    ki18n("Great reference code") );
    aboutData.addCredit( ki18n("Greenleaf"),
    ki18n("Yatta happened to be the only video "
    "on my laptop to test with. :)") );

    KCmdLineArgs::init( argc, argv, &aboutData );

    KCmdLineOptions options;
    options.add("+[URL]", ki18n( "Play 'URL'" ));
    options.add("play-dvd", ki18n( "Play DVD Video" ));
    options.add("debug",
    ki18n( "Show Additional Debug Output" ));
    KCmdLineArgs::addCmdLineOptions( options );

    KApplication application;

    QStringList mimeTypes = Phonon::BackendCapabilities::availableMimeTypes();

    QStringListIterator mimeIter(mimeTypes);
    while (mimeIter.hasNext())
    {
        std::cout << mimeIter.next().toLocal8Bit().constData()
            << std::endl;
    }
    return 0;

It has a lot of KDE verbiage, but I hope you get the idea. The most important lines are at the end. And indeed it did not print the associated mime types.

Reading a bit about Phonon, I tried to see what the backend that it was using was. I believed it was Xine but wanted to be sure. Eventually, what I did was to read /proc/`pgrep amarok`/maps which contains a list of all the shared libraries that it used. A bit afterwards, I discovered that the Phonon backend was configurable from "Settings → Configure Amarok → Playback → Sound System Configuration" and indeed had Xine as the most prioritised backend. After switching it to gstreamer, I realised that Amarok could now play files ending with .mod, but not with say .xm or other Module files' formats, so it indicated that the problem was with Xine.

Looking at the code of the Xine Phonon backend (after some investigation), I found the following:

QStringList Backend::availableMimeTypes() const
{
    if (m_supportedMimeTypes.isEmpty())
    {
        char *mimeTypes_c = xine_get_mime_types(m_xine);
        QString mimeTypes(mimeTypes_c);
        free(mimeTypes_c);
        QStringList lstMimeTypes =
            mimeTypes.split(";", QString::SkipEmptyParts);
        .
        .
        .
    }
}

So it seemed that Phonon returns its results based on xine_get_mime_types(). Therefore, I wrote the following test program to see what xine returns:

#include <xine.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char * argv[])
{
    xine_t * xine_h;
    char * mime_types;

    xine_h = xine_new();
    xine_init(xine_h);
    mime_types = xine_get_mime_types(xine_h);
    printf("%s\n", mime_types);
    free(mime_types);
    xine_exit(xine_h);

    return 0;
}

And it didn't return the appropriate mime types for the MOD files, either. So the problem was with xine-lib.

Next I decided to look at xine-lib and investigate. Searching for the xine_get_mime_types() functions yielded a multiplexer that collected the mime types of the various plugins. So I looked at src/demuxers/demux_mod.c and found the following:

static const char *get_mimetypes (demux_class_t *this_gen) {
    return NULL;
}

So no wonder no Module files-related mime types are returned. After fixing it by writing a patch that just added the mime types as one-big comma-delimited string (as seen in other demuxers), and rebuilding xine-lib, both of my test programs returned the appropriate mime types, and I was able to play files in Amarok.

I submitted the patch to the xine-devel mailing list after subscribing to it. I have yet to receive a reply, to say nothing of getting my patch applied, but at least I now know the problem has been fixed and I (and others) will be able to play Module files in Amarok to my heart's content.

2009-03-31: Tel Aviv FOSS Meeting on 5-April-2009: Intro to Perl 6 - Part II

The Tel Aviv Open Source Club will host the second part of a series of talks by Gabor Szabo about "Introduction to Perl 6" - on 5-April-2009.

The meeting will take place at Tel Aviv University, at the Schreiber Maths&CS building, room 008 on 18:30. So mark your calendars. More information can be found on the wiki page for the meeting.

Attendance is free, it is not necessary to RSVP and everyone are welcome.

With any other problems, feel free to contact me.

Abstract

Introduction to Perl 6.

A series of presentations on learning and using Perl 6 from the ground up to the special features.

Many would think that Perl 6 is just a new version of Perl and that it might only be interesting for Perl programmers. However, Perl 6 is in fact a compiled language running on a virtual machine that embraces many new concepts not found in most programming languages.

The presentations will be equally interesting for Perl, Java and C# programmers.

During the series of talks we will start by learning the basics of the language and will get to various high level concepts. For now we have planned 2 sessions but if we need more time we'll schedule more meetings.

Note:

After the talk we will go to the café at the main entrance where we can continue the discussion. If people bring portable computers, we can get the off the ground on the spot. VirtualBox images will be provided with everything that is needed for playing with Perl 6 set up inside. So you may opt to bring a computer with VirtualBox installed.

2009-04-01: New Book Coming Soon

Mastering cat

It's official! I'm going to publish a new book soon, and I placed a new interview with me about it. Read it so you'll understand why "Mastering cat" would be an essential addition to your collection of technical books.

2009-04-02: The PONIE Subversion Repository

While cleaning up an old hard-disk, I found a Subversion working copy of PONIE - the Perl 5 port to Parrot, and found out the URL of its long-lost Subversion repository. Now I see it's the first hit on a Google search for "ponie repository", but still I wondered what it was for a long time.

2009-04-05: Escaping Special Characters in MySQL Regular Expressions

MySQL sports the REGEXP to enable search using a regular expression pattern. However, it does not contain a built-in function (such as Perl's quotemeta) to escape the special characters in such patterns in order to prevent injecting malicious regular expression code.

Some time ago, I needed to search for a string within word-boundaries in MySQL in a PHP site I help maintain. Here's the solution I came up with for constructing this search:

function enbackslash($matches)
{
    return '\\' . $matches[1];
}

$safe_word = $mysqli->real_escape_string(
    preg_replace_callback('/([^\\w\\s])/', "enbackslash", $word)
    );

$sql .= "text regexp '[[:<:]]${safe_word}[[:>:]]'";

Basically what happens is that I prefix a backslash to any non-alphanumeric and non-whitespace character, and then escape the string again using the standard RDBMS-specific escaping function.

I would appreciate any ideas for improving upon this code, but it seems to work for me.

2009-04-14: Recent Hacktivity

This is another log of my (= Shlomi Fish) hacktivity.

  1. xine-lib patch got applied I mentioned my xine-lib patch for fixing the MOD/S3M/XM/IT/etc. mime types, and after pinging the mailing list discovered that it has been applied, and later on corrected.

    After one of the recent Mandriva updates, I can now listen to Module files on Amarok 2.x comfortably. Thanks to the Amarok developers for applying and correcting my patch.

  2. Building the KDE 4.x trunk - I decided to try the KDE 4.x-trunk to help with the wallpaper-per-virtual-desktop feature that was incorporated there. It took me quite a long time to build it, because I did it wrong and the KDE build process did not warn me about it. Here's the executive summary of how to do it right: you need to build the following modules from trunk (in order): kdesupport, kdelibs, kdepimlibs, and kdebase, using a script such as the following:

    #!/bin/bash
    k4d="/opt/kde4-trunk"
    k4sd="$k4d/share"
    k4bd="$k4d/bin"
    PATH="$k4bd:$PATH"
    test ! -e build && mkdir build
    cd build
    rm -f CMakeCache.txt
    export CMAKE_PREFIX_PATH="$k4d:$CMAKE_PREFIX_PATH"
    cmake -DCMAKE_INSTALL_PREFIX="$k4d" \
        -D"PYTHON_SITE_PACKAGES_DIR:PATH"="$k4d/lib" \
        -DCMAKE_BUILD_TYPE=debugfull \
        ..
    

    In kdebase, you may see that "make install" tries to install some python files under /usr. I workarounded it by editing the appropriate .cmake files and deleting the installation instructions for them, but there's a patchset in the works for that.

  3. KDE trunk bug - In any case, I noticed a few bugs in kde4-trunk (as expected) and reported a bug about the picture names in the wallpaper selection. I spent the rest of the day searching the code and running it under the debugger, in order to try to fix the bug, and was eventually able to supply a patchset for that (in the bug report). The patches were not committed yet.

  4. First-Come First-Served Readers/Writers Lock - A Chinese programmer contacted me by email and reported the fact that the First-Come First-Served Readers/Writers Lock did not implement the _destroy() function, which was documented there. He was right, and it was funny because I haven't touched this code for many years (2002 according to Freshmeat), and didn't expect it to contain any problems. Apparently, they were one of the first to want to use it in production.

    So I went to work. I converted the build system to CMake, because GNU Autohell gave me a weird error (again), that I couldn't care to fix. I fixed this problem, and also fixed some more issues with the code and API. Then I uploaded a new version and published it on Freshmeat.

    I guess Murphy was right by saying that every computer programme contains one more bug than has been discovered so far.

  5. CPAN - In the XML::RSS realm, I applied a patch that Simon Wistow sent me for a test file, and corrected it for style, and then uploaded a new version (1.44) to the CPAN on 1-April-2009. I also corrected some bugs in HTML-Links-Localize and WWW-Search-MSN.

  6. Freecell Solver - I investigated a build problem of recent Freecell Solvers with CMake-2.6.2, and discovered that it was a bug in CMake-2.6.2, which was corrected in 2.6.3. I also applied some cleanups to the code, and extracted some incredibly complicated C-preprocessor-macros into functions, and afterwards wrote a makefile for targetting LLVM bitcode, and implemented a working "profile" CMAKE_BUILD_TYPE.

  7. GUI for a Ruby Library - as an exercise in learning the Ruby programming language, I started writing a library, and then I needed a GUI to visualise it. My first thought was using wxRuby, but Mandriva didn't have a package for it. After contemplating on using gtk+, I decided against it, and started prepared an .src.rpm for wxRuby, while trying to adapt an existing wxPerl GUI to use the Ruby library using Inline::Ruby. I discovered that the CPAN version was incredibly broken and completely non-functional on recent perl's, but looking at the rt.cpan.org bug-report, I found a an up-to-date version on gitorious, which I was able to use.

    So I started adapting the wxPerl library to use the Ruby code. It worked very nicely at first, but then I ran into a strange crash. I reproduced this with a simple non-GUI Perl Inline::Ruby script that used my code, while the equivalent Ruby script ran flawlessly. So I reported this as a bug in the gitorious Inline::Ruby. Meanwhile, wxRuby failed to build, and I have some bugs to report.

    I realised I needed to write the GUI in Ruby, and so, decided to use QtRuby. I started from the final tutorial example, and now have a mostly working code. All of this wasted a lot of time that could otherwise be invested on the library itself. But at least I uncovered some bugs.

2009-04-20: The Linmagazine Events Calendar Has Become Unreliable

This is a small warning for open-source enthusiasts in Israel not to rely on the Linmagazine Events Calendar. It's no longer actively maintained, and my two submissions of Tel Aviv Open Source Club meetings were not published. For one of them, I sent a request to the editor to accept my event via Drupal's messaging mechanism, and he didn't.

For alternatives you may consider the Whatsup.org.il post calendar ( available on the front page) and hopefully we'll be able to arrange an aggregated Google Calendar/iCalendar for FOSS events in Israel soon. But if you've relied on Linmagazine.co.il's calendar alone - you shouldn't.

2009-04-20: My Review of The Perl Review, Winter 2008 Issue

For all those who are interested, I have written a review of the Winter 2008 issue of the journal "The Perl Review" on the Israeli Perl Mongers wiki. I was given a copy of this issue as a present by szabgab in one of the recent Tel Aviv Open Source club meetings, and felt that I should review it.

2009-04-25: No Wiki Spam Lately

I'm administrating several instances of MediaWiki. Up to several months ago, it had been the case that every few days, there was a spam attack on one or more of them, which required my attention. Lately however, they haven't been spammed at all. So I wonder why.

The anti-spam measures we employ are:

  1. We do not allow anonymous commits, and instead require registration. Nevertheless, the registration does not require an E-mail-handshake, which means it can be implemented more easily by pure-HTTP bots.

  2. We utilise the MediaWiki SpamBlacklist plugin, which blacklists edits based on URLs matching certain regular expressions. We use both the Wikimedia-maintained blacklist and our own blacklist, where we add URLs that we were spammed with.

    Lately, our custom blacklist has not been updated, since we didn't encounter any spam.

  3. All external URLs in the wikis are using rel="nofollow" to make sure they do not gain page rank. This did not prevent wiki spam, but it makes sure that spammers don't benefit from spamming.

Possible explanations to why there was no wiki spam lately is that the wikimedia-blacklist has been more promptly maintained and prevented most spam, or that spam-bots check for the rel="nofollow" attribute and in case they find it, do not spam. That, or something changed in the configuration of the wikis, which I wasn't made aware of.

I'm happy that I have less work to do to combat the spam, but am worried that I may become negligent. But I guess I should enjoy it while it lasts.

2009-04-25: Tech Tip: using iotop to monitor the disk activity.

Today I discovered iotop ( on Freshmeat). It's a (Linux-only) tool to find which processes are doing the most hard-disk activity at present. Useful for the cases where your hard-disks' LED is active and you don't know why.

The 0.2.1 version (which is the latest) is available in the Mandriva Cooker repositories, and it was a single urpmi command away from installation. I assume installing it on Debian, Ubuntu, etc. should be just as easy.

I learned about iotop in this blog post (in Hebrew) by Ilan Shavit, which I read today. Thanks Ilan, for the tip!

2009-04-30: Several Vim Tips

I have collected several vim tips in the past weeks, so here they are:

  1. One can use the :sp(lit) command to split a window into two. :new splits and starts editing a new buffer. Finally, both of these commands followed by a filename stats editing the specified file in the new viewport.

  2. This tip is long overdue. You can make Vim behave more like a Windows-like editor by adding source $VIMRUNTIME/mswin.vim to your .vimrc. Reportedly, it causes many problems and is not recommended, but I still like to use it, afer all these years.

  3. One can match any char including newline using \_.. The "\_" construct works for other character classes, and allows you to match all of them as well as newline.

    It took me a long time to find that, back when I did, so I'm documenting it here.

  4. After one indents or unindents several lines using "<", ">", "<<", ">>", etc. and the indentation level needs to be promoted or demoted further, one can use the "." (= repeat) command to repeat the shifting again (and again) until it is right.

    I discovered this trick by accident, but immediately found it useful.

Hope all Israelis had a nice Independence Day. Happy Vimming!

2009-05-10: Recent Hacktivity

I worked a lot on Freecell Solver, doing refactoring and optimisations. On my Birthday (5-May, Tuesday) I tried to do several optimisations, and most of them did not work, so it was a bit frustrating. But some of them did work, and at least I know not to repeat the other ones.

After using it for Freecell Solver, I had decided to adopt the Statistics-Descriptive CPAN module, and eventually got the necessary permissions. I uploaded version 2.7 which featured some bug fixes, cleanups and a conversion to Module-Build. This release still failed some CPAN Testers tests, mainly on 64-bit architectures, due to rounding problems, so I uploaded a new release - 2.8 - yesterday, with more robust tests and a few other improvements.

I worked a bit on the Perl Beginners' Site, fixing some broken links, adding more external links, and fixing some other problems. While working on it, I found a validation problem caused by HTML-Widgets-NavMenu, which I corrected on its trunk, but did not upload to CPAN yet.

One day, I did not have a lot to do, so I decided to convert Test-Run from the old and deprecated NEXT.pm to MRO::Compat. I ran into a few problems because my inheritance graph was incompatible with the C3 guidelines, but mst helped me (again). Thanks! So I ended up uploading Test-Run along with all of its plugins with the converted code.

I also worked on a solution for this KMail bug that a friend referred me to. It took me a lot of experimentation to get a Perl script to be able to properly filter the Postfix outgoing e-mail, and after everything seemed to work, my ISP's email-server rejected the message. Either I'm doing something wrong, or I need to use a different outgoing SMTP server.

Happy Hacking and happy birthday to Alan Haggai Alavi!

2009-05-12: פגישת קוד-פתוח-תל-אביב: יישומי הקוד הפתוח של פרינג

מועדון הקוד הפתוח התל-אביבי (תלוקס) ייפגש שוב כדי לשמוע את הרצאתם של אלכס נרסט ומיכל גבע אודות יישומי הקוד הפתוח של פרינג.

ההרצאה תערך ביום ראשון, 31 במאי 2009, בשעה 18:30, בחדר 008 של בניין שרייבר (למתמטיקה ומדעי המחשב) באוניברסיטת תל אביב. פרטים נוספים ניתן למצוא באתר ובויקי. הנוכחות בהרצאה היא חינמית ולא נדרשת הרשמה מראש.

בהרצאה זו נכסה את הדור הבא של פיתוח יישומים לטלפונים ניידים. פרינג היא יישום לטלפונים ניידים וקהילה עם מיליוני משתמשים ברחבי העולם שמאפשרת למפתחים לבנות יישומים ניידים באמצעות יכולות פיתוח הווב שלהם (כמו PHP, Java וכו).

אנו תמיד מחפשים מרצים שיתנדבו לתת הרצאות בנושאים שונים הקשורים לקוד-פתוח ולמחשבים. במידה שאתם מעוניינים לתת הרצאה, או יש לכם הצעה להרצאה שמעניינת אתכם, נשמח לשמוע ממכם.

2009-05-14: Source for the "E Text Editor" Was Released

Apparently, the source for E Text Editor, a multi-platform Textmate clone was released under a not-entirely-open-source licence. This is good news despite the fact that it is not a fully-open-source licene, because now Textmate-like functionality is available for Linux and other platforms, and not only on the Mac.

I'm still sticking with Vim, though, which is fully cross-platform, open-source and even GPL-compatible.

2009-05-28: Tech Tip: Kopete: Recalling Previous Messages

When using Kopete, a unified instant-messaging client for the KDE desktop environment, one can recall previous messages using Ctrl+Up and Ctrl+Down. A friend, who is a KMess developer, told me it was also a KMess feature.

This wasn't very discoverable, and I discovered it only by accident. But it seems useful.

2009-05-31: Tech Tip: Perl Debugger: Preserving the History Between Invocations

One thing that always bothered me about the Perl debugger is the fact that it didn't rememeber / preserve the GNU readline command history from previous sessions like a shell does for example. Today I discovered how to enable the preservation of the command history from previous sessions of the debugger. To do so, add something like the following the the file "~/.perldb" (under your home directory):

&parse_options("HistFile=$ENV{HOME}/.perldb-history HistSize=5000");

I explictly set the history size to a large value, but this is optional. You can also set the path to the history file to a different value.

Now when you start perl -d it will recall all the latest values from previous invocations.

2009-06-01: Recent Hacktivity

Before I start with the entry itself, I should note that someone copied my "Recent Hacktivity" meme. So I am going to tell my team intimidating lawyers to send him a "Cease and Desist" letter for Copyrights/trademarks/patents/trade-secrets/whatever violation. (Just kidding)

OK, I did a lot of work on Perl-Begin. I went over all the pages and updated all the out-of-date information I could find, and then also implemented a decent "Testimonials" sidebar instead of the half-assed "These companies/projects are using" Perl that there was there earlier. Next, I plan to have to create a tools/IDEs/etc. section.

I also did some work on the Better-SCM site. Most of it was done by going over the emails in my relevant folder and implementing their suggestions.

I did a lot of work on Freecell Solver. I implemented Simple Simon support in Games-Solitaire-Verify, and verified all the FCS-generated Simple Simons solutions up to PySol deal No. 5,000. I also prepared updated Simple Simon statistics. I also did some Perl and PDL work on creating a meta-scan that minimises the average solutions' lengths generated by the solver. This worked out pretty well. Finally the code in the Freecell Solver trunk is now worth over a million U.S. Dollars (at least according to SLOCCount by David A. Wheeler).

I did some extensive work on refactoring and cleaning up the Statistics-Descriptive CPAN module, and released version 3.0000 to the CPAN. Next I'd like to create a perltidy theme to correspond with my personal style, so I'll be able to reformat the code there properly. At the moment, a lot of it is still using 2-whitespace-wide indentation levels, which is annoying.

Next I tended to this akregator bug, where the "Copy Link Location" did not copy to the middle-mouse button paste. Fixing it was pretty easy because there was a line doing it commented-out. The patch was accepted and is now in both KDE-4.2 and KDE-trunk.

I helped sort out the Perl-related mailing lists page at the Perl 5 Wiki. The effort was started by szabgab, and I did most of the sorting. In the process, I learned about many interesting mailing lists, and of many almost dead ones. After I was through, I did not know what to do with the rest of my life.

I ran into this bug with Amarok 2.x on my system, which I fixed, by removing some weird junk from the end of /etc/fstab. The fact that I could not use Amarok irritated me a lot, and I was glad that I resolved it.

I released a new version of the Config-IniFiles CPAN module with some fixes and a t/pod.t and t/pod-coverage.t files to prevent POD problems.

Finally, one evening after an intensive biking session, I worked on forward-porting my Amarok per-song-volume script to Amarok 2.x. Now, the original Perl script could not be used because running arbitrary UNIX process scripts is no longer supported by Amarok 2.x, which now requires all scripts to be written in JavaScript using QtScript. Porting it took all evening, but I had it working eventually. I still resent the Amarok developers for only allowing JavaScript there now, which I don't hold in high esteem.

2009-06-14: New Version of HTML-Widgets-NavMenu

Just for the fun of it, I decided to join the Iron Man Blogging Challenge, so here's my first post.

I recently released a new version (1.0300) of HTML-Widgets-NavMenu, my CPAN module for generating navigation menus. This release fixes a bug in the "HeaderRole" type of navigation menus (which I believe is not widely used) and also cleans up the internals. Instead of my ($self, %args) = @_;, there's now my ($self, $args) = @_;, which is cleaner and should also be a little faster.

You can see HTML-Widgets-NavMenu in action on my homepage. By going to various pages, you can see the navigation menu expands or unexpands in certain places, and the breadcrumbs' trail and next/previous/up links updated.

The story of how I created this module is interesting. It started with a lot of ugly and crufty Website Meta Language logic that evolved to generate my homesite's navigation menu. I didn't really like it so I look for an alternative. I looked for one on the CPAN and found two modules that didn't seem too inviting. Then when I mentioned my quest on Perl-Israel, a fellow Perler said he created a hack based on his module for generating JavaScript menus to do just that and uploaded it to CPAN. That module seemed to have the most user-friendly interface, so I decided to use it. However, I patched it, inherited from one of its classes, and created a lot of code of my own.

Eventually, I ended up with only original code, and so I wrapped things up, licensed it under the MIT/X11 licence and uploaded it to CPAN. Since then, it was improved to some extent, including adding 100% Automated Test coverage at one point (which didn't prevent the HeaderRole bug).

So far I received two five star reviews of the module on CPAN Ratings, and also heard from other people using the bug tracker and on E-mail. As a result, I believe this module is moderately popular, and fills a nice niche.

If you're looking for a module to generate navigation menus for your site, then please take a look at it. And if you have ideas for improvements, I'll be glad to hear from you.

2009-06-21: Tel Aviv FOSS Meeting on 28-June-2009: Programming Red Flags

The Tel Aviv Open Source Club will host a talk by Yaron Meiry ("Sawyer") about "Programming Red Flags") on 28-June-2009.

The meeting will take place at Tel Aviv University, at the Schreiber Maths&CS building, room 008 on 18:30. So mark your calendars. More information can be found on the wiki page for the meeting.

Attendance is free, it is not necessary to RSVP and everyone are welcome.

With any other problems, feel free to contact me.

Abstract

Some programming habits should be avoided at all costs; some should be considered with care and some are just divine. Red flags tell us what to try and avoid and what to inspect more closely. They are a sign of code that might become problematic or straightforwardly dangerous. The talk will be Perl-oriented (with Perl examples) but the programming practices are generic and can be applied to most programming languages, so if you program in Ruby, Python, PHP or other languages, this talks to you as well.

Hopefully by the time we're done, you'll be able to write cleaner, more maintainable, less error-prone, and better code. If not, your money back guarantee!

2009-06-28: Introducing CPANHQ and a Response to Limbic Region's "Improving search.cpan.org (Advanced Search)"

Limbic Region recently wrote a post on his use Perl journal titled "Improving search.cpan.org (Advanced Search)". The post concludes by saying that "Feel free to add your own [ideas] - I know someone who may actually be motivated to implement some of them as an alternate to search.cpan.org is reading :-)". Since I talked with L~R about CPANHQ, a seacrch.capn.org alternative in question, that I have been contributing to, I can assume meant me.

Before I reply to L~R's points, let me introduce CPANHQ. As documented in the first and only wiki page it has so far (which I wrote two days ago), "CPANHQ aims to be a community-driven, meta-data-enhanced alternative to such sites as http://search.cpan.org/ and http://kobesearch.cpan.org/. Currently, the functionality is very basic, but we have great ambitions."

I learned about CPANHQ by overhearing a conversation on Freenode's #perl channel. I joined the effort, and made several enhancements to the Catalyst and DBIx-Class-based code, which was a useful learning experience. The functionality is still very basic, and there's a lot of work to do, so any help would be appreciated. Just fork the github repository ( or my repository, which I tend to keep somewhat more up-to-date ), and play with the code.

Now back to L~R's suggestions. At first I was a bit sceptical of them and they seemed like an overkill, and not really useful. When I'm using search.cpan.org to search for a distribution, either I know the name of the module that I'm looking for and just perform a search to be directed to the location of the exact page, or I usually find the order of the search sufficient for my needs.

However, on a second reading, it seems like what Limbic suggests (like being able to sort results by Kwalitee, creation date, number of downloads, last update date, rating, etc., providing a code search, or recommendations) actually has some merit and should be considered. One thing I'm sure about is that the default CPAN search should remain simple, while only giving an option for an advances search using a different form, where it might be possible to formulate a more complex query using a specialised language. Otherwise, if we complicate the existing search, it could prove confusing to many users.

L~R's suggestions remind me of my CPAN Module-Rank thought experiment, that aimed to define an automatically calculated metric for the relevance of CPAN modules.

L~R later referred me to a post to Perlmonks.org titled "Advanced CPAN Search?" where someone asks a similar question.

To sum up, please consider monitoring or even contributing to CPANHQ if you want to improve the future of finding stuff and interacting with CPAN. It seems that some people have issue with the current CPAN search state-of-the-art, and would like to see it improved. We'd be happy to hear what you think.

2009-07-01: Tech Tip: Finding CPAN Distributions that only have a Build.PL

A few times in the past, I wanted to find perl 5 CPAN distributions that only had a Module-Build-based Build.PL file and not a fallback Makefile.PL file. Yesterday, after some trial and error, I was able to formulate Yahoo Search query, to do just that.

Here it is - the shortened URL with a hyperlink to the full one:

http://xrl.us/bezbkx - Yahoo Search to find Build.PL only distributions

It works by looking for specific phrases in the /dist pages, looking for "Build.PL" and specifically excluding "Makefile.PL". I hereby place this URL and whatever associated techniques under CC0 / Public Domain. Enjoy!

2009-07-07: A Perlbuzz Feature by Me about CPANHQ

W00t! Andy Lester published a feature I wrote about CPANHQ on Perlbuzz. This is in continuation to my previous blog entry about the topic, but much more focused.

This publicity seemed to have work, because we've been visited by two new Perl people on #cpanhq, who tore my hacky Catalyst code to pieces. Anyway, check the article out, and if you like it, upvote it on reddit.

2009-07-12: הזמנה להשתתפות בכנס אוגוסט פינגווין 2009

כמידי שנה, קהילת התוכנה החופשית וקוד המקור הפתוח בארץ בחסות עמותת המקור יקיימו את כנס אוגוסט פינגווין. השנה יתקיים הכנס ביום שישי, ה-7 באוגוסט (2009), החל מהשעה-08:30 בבוקר בבניין אבנר של מכון ויצמן. הכניסה לכנס עולה 30 שקלים כאשר חברי וידידי עמותת המקור ואיגוד האינטרנט הישראלי לא צריכים לשלם.

אז מה יהיה בכנס?

2009-07-17: What you can do with File-Find-Object (that you can't with File::Find)

I've written about File-Find-Object before, but I've intended to write an entry demonstrating its philosophical advantages over the core File::Find module. Today, I'd like to get to it.

As opposed to File::Find, File-Find-Object is:

  1. Has an iterative interface, and is capable of being interrupted in the middle.
  2. Can be instantiated and be used to traverse an arbitrary number of directory trees in one process.
  3. Can return result objects instead of just plain paths.

I'd like to demonstrate some of these advantages now.

Case Study #1: Looking for a Needle in a Haystack

Let's suppose you have a huge directory tree containing many directories and files, and you're looking for only one result (or a few ones). Once you found that result you wish to stop. This question was raised in this Stack Overflow post.

So how can you do it with File::Find? Not very easily. Either you can throw an exception:


sub processFile() {
   if ($_ =~ /target/) {
      die { type => "file-was-found", path => $File::Find::name };
   }
}

eval {
    find (\&processFile, $mydir);
};

if ( $@ ) {
    my $result = $@;
    if ( (ref($result) eq "HASH") &&
         ($result->{type} eq "file-found")
       )
    {
        my $path = $result->{path};
        # Do something with $path.
    }
    elsif ( $result ) {
        die $result;
    }
}
else {
   # be sad
}

This is incredibly inelegant, and abuses the Perl exception system for propagating values instead of errors. But there's even a worse way, using $File::Find::prune:

#! /usr/bin/perl -w

use strict;
use File::Find;

my @hits = ();
my $hit_lim = shift || 20;

find(
    sub {
        if( scalar @hits >= $hit_lim ) {
            $File::Find::prune = 1;
            return;
        }
        elsif( -d $_ ) {
            return;
        }
        push @hits, $File::Find::name;
    },
    shift || '.'
);

$, = "\n";
print @hits, "\n";

Here, we prune all the levels from the results up to the root to get out of the loop.

So how can you do it with File-Find-Object? In a very straightforward manner:

#!/usr/bin/perl

use strict;
use warnings;

use File::Find::Object;

sub find_needle
{
    my $base = shift;

    my $finder = File::Find::Object->new({}, $base);

    while (defined(my $r = $finder->next()))
    {
        if ($r =~ /target/)
        {
            return $r;
        }
    }

    return;
}

my $found = find_needle(shift(@ARGV));

if (defined($found))
{
    print "$found\n";
}
else
{
    die "Could not find target.";
}

The find_needle() function is the important thing here, and one can see it doesn't use any exceptions, excessive prunes or anything like that. It just harnesses the iterative interface of File-Find-Object. And it works too:

shlomi:~$ perl f-f-o-find-needle.pl ~/progs/
/home/shlomi/progs/Rpms/BUILD/ExtUtils-MakeMaker-6.52/t/dir_target.t
shlomi:~$

Case Study #2: Recursive Diff

Evil Djinni from Disney's Aladdin

Let's suppose an evil djinni has removed the -r flag from your diff program, making you unable to recursively find the differences between files in two directory tree. As a result, you now need to write a recursive-diff program in Perl that will run diff -u on the two copies of each equivalent path in the two directorie.

Since File::Find cannot be instantiated two times at once, then when using it, we will need to collect all the results from both directories, and then traverse them in memory. But with File-Find-Object there is a better way:

#!/usr/bin/perl

use strict;
use warnings;

use File::Find::Object;
use List::MoreUtils qw(all);

my @indexes = (0,1);
my @paths;
for my $idx (@indexes)
{
    push @paths, shift(@ARGV);
}

my @finders = map { File::Find::Object->new({}, $_ ) } @paths;

my @results;

my @fns;

sub fetch
{
    my $idx = shift;

    if ($results[$idx] = $finders[$idx]->next_obj())
    {
        $fns[$idx] = join("/", @{$results[$idx]->full_components()});
    }

    return;
}

sub only_in
{
    my $idx = shift;

    printf("Only in %s: %s\n", $paths[$idx], $fns[$idx]);
    fetch($idx);

    return;
}

for my $idx (@indexes)
{
    fetch($idx);
}

COMPARE:
while (all { $_ } @results)
{
    my $skip = 0;
    foreach my $idx (@indexes)
    {
        if (!$results[$idx]->is_file())
        {
            fetch($idx);
            $skip = 1;
        }
    }
    if ($skip)
    {
        next COMPARE;
    }

    if ($fns[0] lt $fns[1])
    {
        only_in(0);
    }
    elsif ($fns[1] lt $fns[0])
    {
        only_in(1);
    }
    else
    {
        system("diff", "-u", map {$_->path() } @results);
        foreach my $idx (@indexes)
        {
            fetch($idx);
        }
    }
}

foreach my $idx (@indexes)
{
    while($results[$idx])
    {
        only_in($idx);
    }
}

( As a bonus, we do not need to sort the results explicitly at any stage, because File-Find-Object sorts them for us. )

This program did not take me a long time to write, it works pretty well, and does populate a long list of results of one or both directories.

Conclusion

If you use File-Find-Object instead of File::Find, your code may be cleaner, your logic less convulted, and you may actually be able to achieve things that are not possible with the latter. I hope I whet your appetite here and convinced you to give File-Find-Object a try.

So what does the future holds? I recently ported File-Find-Rule to File-Find-Object and called the result File-Find-Object-Rule . As a result, "->start" and "->match" are now truly iterative, and I believe you can iterate with them on several objects at once. As I discovered by porting File-Find-Object-Rule-MMagic, I unfortunately cannot maintain full backwards compatibility with the plugin API of File-Find-Rule, because the latter exposes some of behaviour of File::Find (in a leaky abstraction fashion).

I'm planning on porting more File-Find-Rule plugins to File-Find-Object-Rule, and would appreciate any help. I also would like to look at the directory tree traversal APIs of other languages to see if they contain any interesting techniques.

2009-07-20: IRC Channel for Perl in Israel - #israel.pm on irc.perl.org

This Hebrew announcement will be followed by an English one.

פתחנו ערוץ חדש ב-IRC (Internet Relay Chat) עבור משתמשי וחובבי פרל בישראל - #israel.pm ברשת irc.perl.org. אתם מוזמנים להצטרף אלינו ולשוחח שם.

We have started a new Internet Relay Chat channel for Perl in Israel - #israel.pm on the irc.perl.org network. You are invited to come there and chat with us.

2009-07-25: Preparing a Screencast Under Linux

I wrote about the screencast of PySolFC solving Freecell automatically on the fc-solve blog. As I promised there, I wanted to tell the story of how I ended up with a ready screencast. The process was not completely free of problems, but proved to be quite straightforward.

Googling for screencast and linux yielded this blog post at liunxhaxor.net. The first program I tried from there was Istanbul, but it proved to be too crashy, and could never save the video to the final file. Wink is non-open-source so I prefered not to use it, and I had a bad prior experience with vnc2swf. Then I noticed it also mentioned recordMyDesktop, which is a command line program that worked very well and was able to produce a working video. (recordMyDesktop has two GUI front-ends, but I did not bother to learn how to use them, but they may work for you.)

After I had the recording of the screen, I needed to edit it, because one could still see the cursor starting from the window where I ran it, and then me Ctrl+Cing it. And I also wanted to add some background music. So I looked for a good video editor. I had little luck with either cinelerra or Kino, but someone on IRC recommended avidemux, and it proved easy enough and bug-free enough to use. Nevertheless, it refused to import the Ogg Video (Theora) file that I got from recordMyDesktop, but I was able to convert it to AVI format first with ffmpeg.

Using avidemux, I was able to delete the beginning and end of the file, as well as add some background music (from Adam Singer's album "Lifeforce" - available under a Creative Commons licence).

Afterwards I had a ready video file, which I uploaded to YouTube. While I waited for the file to upload, I wrote some blurb for the description. So far the video received 67 views and did not receive any comments. I still found learning how to make a screencast highly enjoyable, and look forward to preparing future screencasts. I hope I've given you some insights into how to prepare it.

Happy screencasting and watching!

2009-07-27: Updates to Perl-Begin.org

After the last news item, a lot of work was invested into Perl-Begin.org, the Perl Beginners' Site, making it even better than it used to be. Here's a summary of the changes:

We hope you enjoy the new Perl Beginners' Site and please recommend it to your friends. All content on Perl-Begin is made available under the Creative Commons Attribution Licence which allows almost unlimited re-use.

2009-08-03: More updates to Perl-Begin.org

I thought the Perl Beginners' Site was perfect after the last update, but boy I was wrong. A quick review and critique of the site by a certain prominent Perl developer proved me wrong, and afterwards I had more stuff that I found lacking. So here's what has changed.

We're now mirroring the public domain "Perl for Newbies" tutorial as well as the GFDLed book "Impatient Perl" by Greg London. Their presence on the site was done in order to preserve Perl-Begin's common look and feel, and to make sure people would be able to access them without having to go to a different site, which may be blocked by over-zealous web-filters. We hope to mirror other freely distributable material that may prove useful to beginners on Perl-Begin in the future.

The "call-for-action" notice at the beginning was changed to "Learn Perl Now!" and now appears on all the pages.

We also fixed several look and style glitches.

There are also several corrections to text and links.

Stack Overflow was added to the web-forums page.

There are now mentionings of two new topical books - "Perl & LWP" and "Programming the Perl DBI".

Finally, the "About this site" page was updated and made more modern.

We hope you enjoy the new site. If you like it, please recommend it to your friends - if you don't - let us know and we'll see what we can do.

2009-08-10: Perl readdir segfault.

I was working on the 5th part of my Perl for Newbies tutorials when I wrote some example Perl code like the following:

my $dir_path = shift;

opendir my $dir_handle, $dir_path
    or die "Cannot open $dir_path - $!."

my @files = readdir($dir_path);

closedir($dir_handle);

(Can you spot the bug?)

Much to my surprise, this code segfaulted when I gave it "." as an argument. I realised that the problem was that I was reading from the path variable instead of from the directory handle, but it still shouldn't have segfaulted.

I asked a few people on IRC about it, and some of them were able to reproduce it, but only with perl-5.10.0 on Linux. Someone was able to golf it down to perl -e 'readdir($x=".")', which segfaults too.

So I opened a ticket for this bug, and Rafael (RGS) fixed it in blead, but without a regression test case. So I submitted a patch to add it, and also pushed it to my github clone of the perl repository. I hope it get applied.

On a different note, I'd like to note that I have completed the transition of the automated test suite of Quad-Pres from Bash to Perl 5 and Test::More TAP-emitting tests. Now they are much less verbose, and possibly run faster (though it might be just a feeling).

Now I'd like to continue its switch from GNU Autohell to something more decent. I still don't know whether I should use CMake or Module-Build as I depend on the wml executable from Website Meta Language. Any insights would be appreciated.

2009-08-10: Tech Tip: listing all files without "." and ".."

Today I had to deal with some pesky dot-files again, and again ran into the problem that ls -a listed them along with "." and ".." which were useless at best. A common idiom in filtering them out is using ls -a | grep -v '^..\?$' (or something similar using egrep or grep -P if it's available). But being too lazy to write it, I did a man ls and discovered a better way, right at the first page of the manual.

One can simply use ls -A or ls --almost-all to filter out the implied "." and "..". At least, this works with GNU ls - don't know about BSD or the various proprietery UNIXes.

So for example, ls -a may give you:

.  ..  FCFS_RWLock_Scheme_RLE.txt  FCFS_RWLock_Scheme.txt  .svn

While ls -A may give:

FCFS_RWLock_Scheme_RLE.txt  FCFS_RWLock_Scheme.txt  .svn

Enjoy! I hope to write about August Penguin soon, so stay tuned.

2009-08-17: Source of the perl.org Sites Are Now Available

Thanks to Matt S. Trout (mst) and his awesome "power of asking", the sources of the perl.org domains are now available. I've wanted to merge sofe of my work on Perl-Begin to learn.perl.org and www.perl.org , and now my job will be much easier.

So thanks to Matt, and the www.perl.org maintainers who made the sources available.

2009-08-20: XSLT Facts - find out how truly evil XSLT is.

We were chatting on ##programming when someone came and asked for help with using XSLT (Extensible Stylesheet Language Transformations - an XML-related standard) from within Perl. We helped him, but meanwhile someone else prompted us to start an "XSLT facts" marathon similar to the Chuck Norris Facts and other similar collection of facts. While it can be classified as geek humour, most of the uttered facts are more universal than geek culture. (I am rindolf)

Just for the record, I don't really hate XSLT, or think it's evil or particularly scary, but I found generating such facts fun. This bit was in fact converted to HTML using an XSLT transformation I wrote.

2009-08-23: Open Source Licences Wars

I published a new essay on my site titled "FOSS Licences Wars" which "gives an overview of the various options in picking up a licence for one's Free and Open Source Software (FOSS) project, and tries to give some guidelines choosing one". Reading from the article:

When Joel Spolsky ("Joel on Software") wrote his notorious blog post "Language Wars", many people asked whether he has "jumped the shark" and that his blog will go downhill from there. I too have read the post, and agreed, that while it had a few good points, it was too based on "feeling rather than knowing". Joel later on posted many good articles and shorter entries on his blog, but many people still recalled it as a very low-point in the blog.

Like Joel, I have a home-site and several blogs, where I post articles and essays about my thoughts, and this time I've decided to risk something similar to what Joel has done on an equally flamatory topic: licences of open-source software. I'm going to introduce the various options, explain a little about their motivation and then give some advice according to my own personal opinion.

Comments are welcome here or on one of the Internet forums where the story was covered. This essay is made available under the CC-by licence, which allows full re-distribution, and re-use as long as credit is given. Nevertheless, a donation or sharing derived works under similar licences, would be welcome.

2009-08-27: A Valid Use of Ampersand-func?

I was talking on #perl one day, when someone joined and asked:

foldr (\(a,b) -> (a:).(b:)) [] $ uncurry zip \
    ("Hv o edyu IPtdy","aeyura orSC oa?")
    <- What would this be in Perl? (It outputs
    "Have you read your SICP today?")

In case you don't know your Haskell, what it does is interleave the two strings and have 1 letter from here and the other one from here. In Perl, strings are not lists or sequences, so we understood we'd need to split them first to do something like that. Furthermore, we recalled the "zip" function from List-MoreUtils that may prove to be useful.

Eventually, someone came with this Perl 5 solution:

perl -MList::MoreUtils=zip -e 'my @former = split //, "Hv o edyu IPtdy";
  my @latter = split //, "aeyura orSC oa?";
  print join "", zip @former, @latter'

Assigning to array variables was necessary because zip has a prototype. Then I figured out that we can avoid that by over-riding the protoype using "&zip", and proposed this solution:

perl -MList::MoreUtils=zip -e 'print join "", &zip(map { [split//, $_] }
    "Hv o edyu IPtdy","aeyura orSC oa?")'

Which was much shorter and clearer but used "&zip". What happens is that we pass it the arrays directly as references. I was able to golf it somewhat into:

perl -MList::MoreUtils=zip -E 'say&zip(map[split//],"Hv o edyu IPtdy",
"aeyura orSC oa?")'

So we got what appears to be a valid use of "&func" to over-ride a pesky prototype. Now, Perl 6 hackers are invited to contribute a Perl 6 solution.

2009-08-31: MooseX-Getopt Hacking

It seems I didn't announce it here, but Test-Run version 0.0120 was converted to Moose, the postmodern object system for Perl 5. After I uploaded all 0.0120-related modules, I was looking for more Moosification, and decided to try and convert runprove to MooseX-Getopt, while still passing all tests.

After a lot of playing with the has attributes to get them I ran into what appeared to be a problem which I could not overcome: I could not configure it not to use Getopt-Long-Descriptive, in case it found it was installed. I didn't want to use it, because I wanted my help display.

After talking about it on #moose, I was told that it is indeed a limitation. I decided to fix it and did it by adding a new parameter. However, t0m told me that I should do it by extracting the non-GLD dependent functionality into a separate role, and doing it. I did it, and he had some further comments which I incorporated into my MooseX-Getopt repository. My enhancements are not in the CPAN module yet, but they are going to be soon.

And I live you with a #perl discussion about Chuck Norris:

 <Su-Shee>  2010 is planned more or less as a refactoring year.
 <DrForr_>  A whole *year*? What physical plane of existence do you reside
            on?
 <Su-Shee>  DrForr_: there's also bugs to fix and systems to care for and
            things like that. it's not that we're locked into the closet
            and a year later a new, shiny product is released. ;)
 <DrForr_>  Closet optional.
 <Su-Shee>  I'm not a wonderwoman refactoring half a million lines of perl
            in a week, sorry. :)
 <rindolf>  Su-Shee: Chuck Norris refactors 10 millions lines of perl
            before lunch.
 <Su-Shee>  rindolf: hm. that's the reason.. I'm not as hairy as chuck
            norris and I don't have a beard...
 <rindolf>  Su-Shee: ah.
 <rindolf>  Su-Shee: Chuck Norris also wrote a complete Perl 6
            implementation.
 <Su-Shee>  rindolf: I heard, he already wrote Perl 7.
 <rindolf>  Su-Shee: yes.
         *  rindolf wants to be as awesome as Chuck when he grows up.
 <Su-Shee>  rindolf: I envy you. I'll never be as awesome without a beard.
 <rindolf>  Su-Shee: heh.
 <rindolf>  Su-Shee: it doesn't matter if you're rigid on the outside as
            long as you're rigid on the inside.
 <Su-Shee>  chuck norris doesn't make mistakes.
 <rindolf>  Su-Shee: Chuck Norris corrects God.
 <Su-Shee>  rindolf: I'll apply as his secretary.
 <rindolf>  Su-Shee: as Chuck's?
 <Su-Shee>  rindolf: yes.
 <rindolf>  Su-Shee: OK.
 <rindolf>  Chuck Norris doesn't code. When he sits next to a computer, it
            just does whatever he wants.
 <Su-Shee>  I'll tell my boss tomorrow. Chuck is who he wants.
 <rindolf>  Su-Shee: Chuck Norris is his own boss.
 <rindolf>  Su-Shee: if you hire him, he'll tell your boss what to do.
 <Su-Shee>  good point.

2009-09-01: What Would you like in a CPAN Distro Manager?

I uploaded Dist-Man (short for Dist-Manager) version 0.0.1 to CPAN, yesterday. It is based on Module-Starter , and its documentation still has some remnants from it, but I'm planning to take it into new and exciting directions. The source code is maintained in a Mercurial repository at Berlios.de (called "web-cpan" after the meta-project for most of my CPAN modules - I'm only given one Mercurial/hg repository per-project there), and there's also a bitbucket mirror. Feel free to clone it, fork it, branch it, spin-it-off, or whatever the cool kids are doing now with their distributed version control repositories. I can also give commit bits. The reason I'm using Mercurial is because I'd like to play with it a little more, and because git still often throws me off when using it and still seems counter-intuitive. Like an old email correspondent of mine said "You gotta move, or you stone.".

In any case, I'd like to share some plans and ideas I have in mind for a distribution manager similar to Module-Starter and ask you for your input. Note that my plans take an entirely different approach from Dist-Zilla. While Dist-Zilla generates your code from templates, Dist-Man intends to deal with the code as is, and modify it in place, assuming a change is needed.

So without further ado, here's what I'd like to see, or at least seems like a good idea:

  1. Provide scaffolding for more lib/*.pm t/*.t , etc. files. You can run something like pl-dist-man new --name=lib/MyPackage/NewModule.pm and it will create lib/MyPackage/NewModule with all the POD and some sample code ready to go, possibly with some tests under t/, etc.

  2. POD at the end. - that's what Perl Best Practices recommends, and what Perl-Critic checks for by default. It's not my personal style, but I heard someone complain to me about it in private.

  3. Keep a machine-readable configuration file with the settings - keep a configuration file in YAML/JSON/Config-General/etc. format in the distribution that will allow Dist-Man (and other programs) to determine configuration options about the distribution. So, for example, it can use the licence field to determine what to put in the licences of new modules, programs and test scripts. And we can have Build.PL, Makefile.PL read this file and feeding it to Module-Build or Module-Install or whatever.

  4. Hermetically changing configuration parameters in the code files - you can change the licence in all the files in one step. Or bump the version number in all files.

  5. Support for more software licences - perhaps with some integration with Software-License.

  6. Make judicious use of Template-Toolkit - I'd like to make the generation of the Perl code be based on Template Toolkit so it can be easier to maintain. The base Module-Starter code is at the moment modular, but seems to crying for help from a good templating system.

  7. Good integration with the underlying version control system - when you do pl-dist-man add lib/MyPackage/File.pm it will add it to the MANIFEST and the underlying version control system, and you can move files or directories around, hermetically rename namespaces, etc.

  8. A plug-in to add a diversity statement - see what Kirrily Robert wrote about it here. This plug-in will allow the developers to add it automatically to all of their modules. I don't rule out coming up with such a plug-in for Module-Starter too, but I just wanted to mention it here.

While I have given Dist-Man as an example, this discussion should not be specific to any particular implementation. So what would you like to see in a manager of CPAN distributions?

Note: Yes, I'm aware that Dist-Man is a fork of Module-Starter, and I tried my best to avoid it, but it seems I had no option. The reason for the fork were the fact that there was a large round-trip delay when applying my patches (usually several months), the fact that I could not get either hgsubversion or git-svn to handle importing the history from its Subversion repository well (see this hgsubversion issue entry), and the refusal of the Module-Starter maintainer to provide me with a commit access, even after I promised that I will only work on a separate branch, and not commit to the trunk directly (at least not without his permission). I let him know about how I feel, and of my intention to fork the project, and he implied he'd rather see me fork the code instead of complying with one of my needs.

I feel bad about forking it but I feel like I had no choice, and as Eric Raymond (ESR) says in the Magic Cauldron:

(It bears repeating here that nobody in the hacker community wants projects to split into competing development lines; indeed (as I observed in Homesteading the Noosphere) there is very strong social pressure against forking, for good reasons. Nobody wants to be on a picket line, in court, or in a firefight either. But the right to fork is like the right to strike, the right to sue, or the right to bear arms—you don't want to have to exercise any of these rights, but it's a signal of serious danger when anyone tries to take them away.)

So I'm not particularly interested in discussing it further. Please keep the discussion on the topic of what you'd like to see in a manager of CPAN distribution, be it Module-Starter, Dist-Man, Dist-Zilla or whatever.

2009-09-02: TelFOSS Meeting on 6-September-2009: Advanced GDB

The Tel Aviv Open Source Club will host the talk "Advanced GDB" by David Khosid on Sunday, 6-September-2009 (the upcoming Sunday), at 18:30 in Tel Aviv University, the Schreiber Maths&CS building, room 008.

Attendance is free, it is not necessary to RSVP and everyone are welcome.

With any other problems, feel free to contact me.

Abstract

In general, I will be sharing my discovery with you: When the debugging of modern software is required, basic GDB techniques are insufficient, but new techniques can be created from the nearly 160 commands available in GDB. "Modern software" refers to multi-threading, using STL and other libraries, IPC, signals and exception mechanisms. In this lecture, I will explain techniques for debugging large, modern software written in C++.

The presentation will be accompanied by vivid examples for all the topics.

Covered topics:

1. Preparing a program for debugging; starting the debug session

2. Controlling GDB: getting help, navigating, using breakpoints and watchpoints

3. Extending GDB through user-defined commands

4. Analyzing STL containers and algorithms

5. Dealing with multi-threading, C++ exceptions

6. Managing signals

7. Automating repetitive tasks on the almost unchanging code base

8. Remote debugging

2009-09-05: French Translation of the "Encouraging Women in FOSS" Howto Needed

Abstract: I'm calling for a volunteer effort to translate the "HOWTO Encourage Women in Linux" document by Val Henson and others.

Background: I've recently been communicating and collaborating with a young (16 years-old) French open-source enthusiast. His English is pretty bad, but I can communicate with him with some difficulty. He's using Linux, and has two computers in his room, but his younger 8 years-old sister has none in hers ("Unfortunately" as he said). This reminded me of this section from the "encouraging" howto where it is said that:

The most striking example of a subtle bias against computing for women is that, in the U.S. at least, the family computer is more likely to be kept in a boy's room than in a girl's room. Margolis and Fisher give several telling examples of this trend and its effects on pages 22-24 of Unlocking the Clubhouse.

Now, in Israel, the common practice among most modestly capable families is to have a separate computer for every member of the family (whether male or female), so I don't see why his family should be backwards here, given that they can afford it, and that nowadays a powerful PC computer can be bought for 400 USD (or less) and that can run circles around my original PC XT machine, on which I've learnt programming and played many games.

The problem is that this kid's parents don't speak English, and so I need a French translation for that. While some of the Linux documentation project's howtos have been translated, I couldn't find any French translation of the "Encouraging women in FOSS" howto. I suggest setting up a page for collaborating on the translation for it on a public wiki, and that everyone contributes their share at their spare time.

Translations to other languages will also be welcome, and I can give my share of the work for a translation to Hebrew (though I feel it won't be necessary because the vast majority of Hebrew speakers have decent if not good English). If you're up for the task, please mention the wiki page with your respective language below.

Hazal said that "Whoever saved a single soul - has saved the world entire.".

Note: while I have had some critique and reservations regarding the HOWTO, I still think it makes a good read, and should be taken into consideration.

2009-09-11: Perl-Begin Updates: CSS, Page about Hashes, and More

Here are the new changes to the Perl Beginners site since last time.

2009-09-24: Hamakor Prize Winners for 2009

Hamakor is the Israeli Society for Free and Open Source Software. Among the activities of the society, it gives the "Hamakor Prize" for contribution to open source in Israel, whose winners are determined by a vote by the society's friends and members. This year, the prize was given for several categories.

Better late than never, we wish to announce the winners in the Hamakor Prize for 2009, in the various categories:

Congratulations to the winners and the other candidates, who didn't win this year, but whose contributions are still worthy of appreciation.

2009-09-27: Yak-shaving Perl Hacktivity

I recently asked the module-authors mailing list about which parser generator to use for Text-Qantor. Eventually I opted to use Damian Conway's new Regexp-Grammars as recommended by Ovid. So I started converting the code from the broken Parse-Yapp grammar to it. I had some tests and ended up getting it to work after some trial and error and debugging.

Now, Qantor used to convert from its plain text representation directly to XSL-FO, but I preferred to convert it to an intermediate XML format instead. So I worked on it and finished. However, then I needed to translate the XML to XSL-FO, so I wanted to write an XSL transformation, and then realised I've actually done such stuff a lot before: XML-Grammar-Screenplay, XML-Grammar-Fortune, XML-CompareML, etc. As a result, I decided to refactor all the common functionality into a common XML-GrammarBase distribution.

Shouldn't be hard, right? Well, not so fast. When trying to create the skeleton of the distribution using a pl-dist-man setup … command, (see Dist-Man), it died with an error - so I needed to fix it first. The fix was not hard, but I needed to extract some common test functionality into a ./t/lib module. However, the Test-Count used some Test-Count variables that now moved to the file. So I needed to implement an include functionality. (Yak Shaving)++

Implementing this functionality in Test-Count was time consuming, and I did it by changing many files. You can now do # TEST:source "$^CURRENT_DIRNAME/lib/MyMoreTests.pm";, which includes the file.

Then I went back to Dist-Man re-counted the text and fixed it, and now I can work on XML-GrammarBase again. This is what is called yak shaving, but this time it yielded many benefits to my other projects.

2009-10-03: Perlsphere Implemented Using Perlanet

You may be familiar with Perlsphere, which is a Perl-related feed aggregator, which is more open for membership than "Planet Perl". I've encountered some problems with its web feed, including some entries that displayed the HTML markup, and the fact that web feed entries are not prefixed with the name of the Blogger or Blog. As a result, I decided to experiment with implementing the same functionality as Perlsphere, only using Perlanet, which is a lightweight PlanetPlanet-like web feed aggregator written in Perl.

So I present perlsphere-using-perlanet. There are links to the OPML file, the Atom feed, and the source repository at the top. So far it seems to work pretty well, and it may be a viable replacement for the Plagger-based Perlsphere in the future. Note: the URL and everything about it are temporary, and I don't update the feed peridiocally at the moment (but I can set up a cronjob). The OPML file may be out-of-date too.. Anyway, comments and bug reports are welcome.

I should note that I've locally applied the patch in this XML-Feed bug report which also affected some of my CPAN modules, and you may need it to get it to work.

2009-10-05: Pre-Announcement for the "Welcome to FOSS" Series of TelFOSS

In previous years, the Tel Aviv Open Source Club has conducted the "Welcome to Linux" series along with other Israeli clubs, towards the beginning of the scholastic year. This year the series will be in a different format of Welcome to the World of Open Source Software (also see the wiki page). We intend to introduce free and open source software to the users in general, and among it many open source programs that can be ran on both Linux and Windows, as well as introduce the ideology and philosophy of the open-source code and their technological and social advantages.

Right now, we're looking for someone to give the demonstration of the cool open-source programs, which is the first presentation which will take place a few weeks after Sukkoth. Ori Idan has already volunteered to give the presentation about the philosophy of open source software. After these two presentations, we intend to give several secondary talks. Among them would be "Free Development Tools for Linux and Windows" (a general overview of them, and not only those that are intended only for C or for Java), as well as "Don't be afraid of the command-line" where we will show how nifty the command-line can be. So we're also looking for volunteers to present these talks.

Whoever wants to pitch in and contribute may contact Shlomi Fish, the coordinator of the series and the club and subscribe to the organisation mailing list (which doesn't have a lot of traffic at the moment).

Be free and Happy Sukkoth.

2009-10-08: XML-RSS-1.46 - now with dropped support for perl-5.6.x

I recently released XML-RSS version 1.46, with a few changes:

The latter caused an incomplete test coverage, and various test failures in perl-5.6.x due to a bugs with XML-Parser there. I think it's time for the Perl world to move away from ancient versions of Perl. perl-5.8.0 was released in 2002, 7 years ago, which is plenty of time to upgrade to it.

On a side note in this hacktivity log, I'd like to note that I've done some work on XML-Grammar-Screenplay and converted the XML grammar to its own namespace. I was able to successfully doing it after temporarily giving up on several other XML-related processsing tasks.

Cheers!

2009-10-13: Social Post about Perl: Giving a Remarkable Online Help

Remarkable Customer Support as it applies to voluntary projects

One of my favourite Joel on Software articles is "Seven steps to remarkable customer service". While the article is oriented towards software shops it has universal implications. Quoting from the article:

Here are seven things we learned about providing remarkable customer service. I’m using the word remarkable literally—the goal is to provide customer service so good that people remark.

Today I'd like to write about providing remarkable online help on the various online forums of Perl. Online help is similar to customer support, only we don't charge for it, and it is done by volunteers. Nevertheless, we do have an interest to provide good online help, because otherwise we will get fewer people who use our project. They are still our users/customers even if they don't pay us.

So let's start.

How not to lose a single soul.

One day (which happened to be my birthday) I chatted on Freenode #perl when someone with the nickname of sas123 joined and said he couldn't understand references. We naturally referred him to perlreftut, but even after reading it, he said he did not understand. Some of the channel regulars said that perlreftut was written well enough that he must have been able to understand. However, I recalled a a Hackers-IL thread called "Smoking out pointer-impaired minds?" where we discussed why understanding pointers or references is hard, and why many people will have trouble understanding pointers at first.

As a result, I invited him to #perlcafe, and we held a long discussion about what Perl references were. And he eventually was able to understand them. Such extraordinary willingness to help is something that people will remark upon and recommend to their friends. I do not think this discussion should have stayed on #perl because it would have reduced the Signal-to-Noise ratio and prevent other people from getting help (and it did take a long time). However, I think the #perl participants should have told sas123 that they will help him on a different channel right away instead of telling him that he's beyond hope.

Communities around FOSS projects should try to avoid losing even one newcomer. That's because people who've been disappointed by bad social treatment will rant about it to their friends, vent about it in their blogs, or worse. And naturally, several individual people add up to a lot pretty quickly.

I still recall an incident, where someone joined #vim, said he's been using GNU Emacs or XEmacs for many years. Now, because the people on #emacs have insulted him and treated him badly shortly before, he decided to switch wholesale to the alternative - Vim. When I suggested that he could use a Vim add-on which could make it behave more like Emacs (in order to ease the transition) he did not want to use it saying that he'd rather do the conversion properly.

So the Emacs people lost a long-time user, who opted to painfully re-train himself to use something completely different, losing years of habit and customisations. As members of the Perl community (for example) it is much more likely that we will lose a new member who can easily choose Python, PHP, Ruby, Lua or whatever. So we need to provide a remarkable online help to keep them.

It's not new.

Other people inside and outside Perl have been echoing similar concerns. See for example what Andy Lester wrote about "Don't optimize for yourself in communities" on Perlbuzz, and the Kirrily Robert's "Geek Etiquette" blog.

A tougher nut to crack.

A few months ago #perl on Freenode was frequented by a certain newbie who needed to use Perl for her site about role-playing games. She needed too much hand-holding and often made many steps backwards and as a result one of the ops has eventually banned her. One day, when I was on Freenode, she private messaged me and explained the problem, and I invited her to #perlcafe or #perl-cats to ask her questions and get help with her code there.

Now we help her at #perlcafe . She still requires a lot of hand-holding, often won't immediately understand why the advice we give her is good, and is often too much utilitarian and not focused enough in her quest for Perl knowledge. But she still seems to make some progress, and we now consider #perlcafe as an "incubator" for her (with all the Science Fiction implications) that will make her ready for public consumption on #perl.

Should her training stay on #perl? Naturally not. But, on the other hand, she still should not have been banned, and instead a few people should have volunteered to help her on a different channel.

How to do it.

The old adage of "Be conservative in what you do, be liberal in what you accept from others" applies to human communications as well. On help forums, we can expect many people who are tactless, non-native speakers of English, who may incorporate many paradigms from their native tongues or from popular Anglophone culture, with relatively bad English, in a bad mood, not very knowledgeable or clueful, with personality defects, and/or who have not read the FAQ or the needed documentation. On the other hand, we should always be friendly, polite, with the best English that we can emit, and being as helpful, accessible and non-laconic as we humanly can.

As annoying "clueless" newbies can be, they still can and often do improve and they are still human. And being hostile to them can have a snowball effect. By being courteous and helpful to newcomers, we can also further educate them on how to behave in our own or other on-line communities which will in turn make the long-term experience better.

As a pragmatist, I dislike what I call the "grand disciplinary" approaches, which say that people should have a lot of discipline to maintain a certain state. These things almost never work. I much prefer the more practical approaches of doing small and one time things like changing the channel's topic. Or alterantively telling yourself that you should remember to do it a few times, which ends up turning it into a habit.

Building a friendly and helpful environment is not that hard, and we can make a better one by several incremental steps. And it is worth it.

Some final notes.

A final note: I may have focused on Perl and Freenode's #perl here, I only did it because Perl is my area of expertise and because I'm an active member of Freenode's #perl ("The poor of your city come first", etc.). However, I think that #perl on Freenode is generally a great resource for beginners and experts, and I've seen much nastier IRC channels and Internet forums in general, including on Freenode. So don't take me the wrong way - all I'm saying is that there's still room for improvement.

Thanks to Andy Lester, Trashlord and Rick for giving some input and corrections about early versions of this article.

2009-10-22: Code/Markup Injection and Its Prevention

I'm running out of time for a post to Planet Perl Iron Man, so I'm going to prepare something quick, but hopefully enlightening. I'll use Perl as a demonstration language, but the practices I'm going to cover tend to be more universal.

Starting from the early UNIXes and before, operating systems represented code and its data as sequences of units of a certain number of bits called bytes. Starting from Unix, which was designed to run on such machines as the 16-bit PDP-11 and later on the 32-bit VAX family of computers, this unit has generally become 8-bits. Today, there are many 8-bit based text encodings (and many more binary encodings for binary data), and the interested reader is referred to Joel Spolsky's introduction on the subject and Juerd Waalboer's perlunitut or your language's equivalent document.

In any case, let's suppose we have a string where we want to embed a variable containing a string. In Perl we can do:

my $total = "Hello " . $adjective " World!";

Or more simply:

my $total = "Hello $adjective World!";

So if we put "beautiful" in $adjective, we'll get "Hello beautiful World!" in $total and if we put "cruel" there we'll get "Hello cruel World!" there.

So far so good, if it's a plain string written in plaintext. However, what if it's in a more well-formed format? Let's say HTML:

# Untested
my $input = get_input_from_user_somehow();
print <<"EOF";
<p>
$input
</p>
EOF

The alert reader will notice that $input was inserted as is into the HTML output. And since we didn't check if it contains special characters or escaped its special characters, a malicious user can insert arbitrary HTML code and even JavaScript code there. This in turn can wreck havoc upon the users of the page.

This form of HTML injection is called a a cross-site-scripting attack (XSS). If present in web applications or web-sites, it may allow malicious crackers to set up traps to the unwary, and possibly gain access to sensitive information on the site, such as the passwords of users or administrators. And you did notice how easy it was to write code that exhibited this problem, right?

Here are some other forms of code or markup injection:

  1. Shell Command Injection - I've discussed it briefly in a different post about "shell variable injection" in Bash, but it also exists in Perl. Imagine doing system("ls $dir"); or as some newcomers are tempted to do `ls $dir`, which the latter still has some legitimate uses. Now I as a malicious user can put in the $dir variable some malicious shell code which will wreck havoc on the system of the user that is running the script.

  2. SQL injection allows a user to inject malevolent SQL code that can do untold damages in the database. It is very common in web applications and many other applications that use SQL code. If you do something like "SELECT id FROM users WHERE name='$name'" then by putting single-quotes in the name, and using SQL syntax one can insert arbitrary SQL there and do a lot of damage. There was also a very nice xkcd comic about it.

  3. Perl Code injection - let's suppose we want to construct an optimised anonymous function ( sub { ... } ) on the fly. We can build its code and then use the string eval - eval "". A lot of Perl programmers think it should be avoided at all costs, but metaprogramming has some legitimate uses. Moreover, this can happen in other cases, like when we construct a Perl program (or a program of any other language on the fly and execute it).

    In any case, if we insert a variable into the eval "" which was input from the user without being escaped or validated, we can have an arbitrary code execution.

  4. Regular expressions' code injection - imagine you want to see if a string is contained in a list of strings. One naïve way would be to concatenate the strings using a separator that is unlikely to be contained in them (such as \0) and then match this gigantic string using $haystack =~ m{$needle}. However, if $needle contains special regex characters, then the operation can take a lot of time to match or worse - yield an incorrect result. One way to avoid that is to use perldoc -f quotemeta or its \Q and \E regular expression escapes - $haystack =~ m{\Q$needle\E}g. In this particular case, it is also probably better to use a hash, but naturally this was just one example where we'd like to embed some arbitrary (but plain) text inside a regular expression.

These are the prominent examples I can think of now, but they are not the only ones. Your program is in danger whenever it accepts text input from the user and passes it directly to an output format that has some grammar and syntax that can be influenced by this string.

So how to mitigate such code injection problems? There are many ways - sometimes providing alternatives and sometimes complementing each other:

  1. Make sure you have enough discipline to escape the input before it is passed to the output venue. Write automated tests for that.
  2. If you still want to allow some user input, then make sure that you analyse it to make sure it doesn't contain any malicious code that can abuse the system. For example, you may wish to restrict input only to certain HTML tags and attributes.
  3. Taint your data using unsafe typing or "kinding" and make sure that it can only be output after either being escaped or being untainted. Joel on Software recommends making the wrong code look wrong, which while desirable and important, is probably less preferable than the wrong code to behave wrong and abort with a huge "You suck!" error or something. This may not be very possible given certain limitations of the programming language, but it is a better ideal.
  4. Use auto-escaping features of your environment such as SQL place-holders (e.g: "SELECT * FROM mytable WHERE id = ?"), and the list argument of "perldoc -f system" (e.g: system { $cmd[0] } @cmd).
  5. Perform frequent code reviews, black box tests, and encourage hackers to find problems in your code.
  6. Use complementary security measures that make sure that even if a problem occurs, its damage is mitigated. As examples, you can try running the script under an underprivileged operating system user, or as a database user that lacks certain database privileges.

There are probably several measures that I'm forgetting, so feel free to add them as comments or trackbacks. In any case, be careful when writing code that may cause code or markup injection because the consequences may be dire.

2009-10-24: "Criticisms of Perl" page under work on the English Wikipedia

Due to the fact that the "Criticism" section was removed from the English wikipedia article about "Perl", I've decided to create a "Criticisms of Perl" page on the English Wikipedia under "User:Shlomif" (which is my user-account). Nevertheless, the page can be edited by other editors, and interested people can comment on the discussion page.

Reading from the page, it aims to be a coverage of the criticisms raised for the Perl programming language which would be comprehensive, and written from a Neutral Point of View, but still mostly concise.. The article itself still lacks many citations, and adding them would be appreciated. Also, many important criticisms should be better covered. Be bold and edit.

2009-10-30: Follow-up to chromatic's "On Answers to Smart Questions"

On the Modern Perl Books blog, chromatic wrote a post titled "From Novice to Adept: On Answers to Smart Questions" where he mentions my comment to a previous feature of his there. I am flattered that chromatic decided to expand upon my comment. Quoting my comment:

I agree with you about it. However, idioms are nice and dandy, but naturally, when going on IRC, one can often open the Pandora box of the colour of the bike shed argument. For example should it be:


Person->new(
   first => "Sophie",
   'last' => "Cohen",
   birth_year => 1977,
);

Or:

Person->new({
    first => "Sophie",
    'last' => "Cohen",
    birth_year => 1977,
});

(with a hash ref)

I tend to think the second option is better because it's probably a bit faster (not that it matters a lot), and because it will warn about an "odd number of hash elements" earlier, but there's a lot of code on CPAN out there that uses the first option, because it involves somewhat less syntax. In one of my modules, I supported only the second option, and a programmer who was after my T-shirt offer, implemented convulted code to have it either with a single hash-ref or flattened into @_, which I had to reject.

Naturally, this is just the tip of the iceberg. Brace indentation, placement of HTML opening and closing tags, whether HTML should be nicely indented (even if generated by a server-side language), or instead occupy as little space as possible, whether you should sub-class sub new {... } or sub _init {....} etc. etc. are all a matter of much debate, and they are pretty much a bike shed colour's argument (while they often do have some points for and against).

I think it's important for a beginner (in Perl or in any other language) to distinguish between such minor matters and non-idiomatic code, and that it is even more important for an expert to see past his own preferences for the bike shed's colour when helping newcomers.

chromatic discusses it in the page above, and gives some advice for newbies who want to avoid getting confused from such discussions on personal style and conventions. Often, matters such as those of indentation and whitespace or placement of braces have some good and valid arguments either way, but they are still a bike shed colour argument, and as such should be either enforced globally by the project or workplace, or alternatively a programmer can do what they prefer on their personal project.

2009-11-01: Why you should Upgrade to New Perl Versions?

Gabor Szabo wrote about "Why bother upgrading perl?" and announced it on the Israeli Perl Mongers mailing list. guy keren replied like so:

i have a different question: why do you care?

there are reasons for people to use the older software for any software X you'll mention - you operating systems, office suites, programming languages/compilers, external libraries etc. etc. etc.

what is the point of pushing people? this sounds like the method of commercial companies - they want to sell the upgrades. what do you want to sell - course-grades? [Editor's note: Gabor is giving commercial training for Perl and other fields.]

the basic idea of open source is to give the power to the users - including the power to stay with an older version of software if and when they see fit.

if there was an assured (QA-wise) upgrade process that never broke things and was trusted by people - people might have considered these upgrades with a more favore. as long as this process has its cons - you'll have people that will simply refuse to upgrade until they have no choice - and sometimes even then.

Before I republish my reply to guy here, I should note that the "Why do you care?" attitude is not very beneficial. Essentially, guy is implying that as human beings, we should "mind our own business", be selfish and/or self-centered and not care what happens to other people. However, it's good that we care about other people, because we are part of a community, and because helping people we know can bring us a lot of pleasure. As much as I support Ayn Rand's "Objectivism" and Rational egoism, I still think that we derive a lot of short-term and long-term pleasure from helping people we know, and that it is a healthy part of acting out of one's best self-interest. Selfishness (meaning "having a complete disregard for others"), is a form of undesirable behaviour, and should not be encouraged.

Now let's move on to my reply:

I think you're missing the point. Gabor is not trying to sell courses or services for the newer versions of Perl (which as you note are free-of- charge). Also, companies who want people to upgrade their software usually have a better motivation than selling upgrades.

I blogged about it in my blog post - "The Broken Window Fallacy". Quoting from there:

I talked with a few people on IRC about how I hated the fact that most people are still using Windows which is quickly getting inflicted with malware, or Internet Explorer, which is lagging behind Firefox and other browsers in its support for web standards, and makes the job of the web designer much harder. So they told me that it is actually a good thing because that way someone has to fight the malware (a very uphill battle), and they have more work as web developers.

The problem is that tech workers can never run out of things to do. In the time they have to adapt their web-sites for MSIE, they can better spend on things like having better functionality, more web-sites, more pages. The time people spend making sure their Windows systems are free of malware, is better spent happpily using Linux and getting some actual, productive work done.

Many Perl contractors , consultants, workers and people who volunteer to help people on online forums need to deal with old Perl versions or "Cannot use modules from CPAN" constantly. Yes, it provides work, but these people would and can rather do something else. See the "Parable of the broken Window" on Wikipedia.

Naturally, this is not only true for Perl, but for all software technologies. I wouldn't want to support Red Hat Linux 9.0 (or older versions) or a 1996 vintage FreeBSD (which I happily used at the time), or Windows 95 or older. And I've recently stopped caring about supporting Microsoft Internet Explore (MSIE) 6 on my sites (which are not commercial, or else it may have been more of an issue).

We are not making any money directly from such people who upgrade their Perls or whatever to newer versions (unless they hire us to handle the upgrades for them, which is a bit far-fetched.). But in the long run, we are happier because we have an easier time supporting their systems and not having to comply with their irrational whims.


As a result, we have a genuine, egoist, interest to make sure our customers use recent versions of their programs and are not stuck on old versions. This is because it prevents us from doing exciting and cutting-edge work during the time that is spent in providing support for the old programs. And as I noted in the original post, supporting old programs (e.g: adapting the sites for Internet Explorer version 6), is causing a lot of frustrations and bad feelings, which are also undesirable and have a bad net effect on our self-esteem and happiness.

I hope you agree.

2009-11-06: New CPAN Package: XML-Grammar-Fiction

Earlier today, I uploaded version 0.0.1 of XML-Grammar-Fiction to CPAN. It is a fork of my own XML-Grammar-Screenplay, which was heavily modified in the recent days, with quite a lot of coding on my part (I felt encouraged by all this coding). The code is still pretty hairy and only the basic functionality was implemented, and there may be many bugs, but it passes all tests on my system, and I've already used it for one of my stories, so I'm happy.

So what is it about? XML-Grammar-Fiction provides the so-called Fiction-Text, a lightweight markup language of sorts to write prose (such as short stories, humorous bits, novellas or novels) in, which in turn gets converted to Fiction-XML, which is a dedicated XML grammar for representing fiction, and which can also be used directly. Fiction-XML in turns can be converted to DocBook/XML or directly to XHTML.

I needed it because I found it troublesome to work on one of my stories in Hebrew ("The Pope Died on Sunday") directly as DocBook/XML, using either gvim or the KDE 4 Kate Editor, due to the problem with the XML tags being in Latin whereas most of the text being in Hebrew, so I sought a more clean text-based format. As a result, XML-Grammar-Fiction was born, just in time for NaNoWriMo (= (Inter-)National November Writing Month), which I'm planning to spend some time investing on my stories and screenplays (for which I already have XML-Grammar-Screenplay).

So how does a sample Fiction-Text story look like? Here's the markup for one straight out of the t/data directory of the XML-Grammar-Fiction CPAN distribution.

<body id="index">

<title>David vs. Goliath - Part I</title>

<s id="top">

<title>The Top Section</title>

David and Goliath were standing by each other.

David said unto Goliath: "I will shoot you. I <b>swear</b> I will"

<s id="goliath">

<title>Goliath's Response</title>

Goliath was not amused.

He said to David: "Oh, really. <i>David</i>, the red-headed!".

</s>

</s>

</body>

You can see the mixture of some XML-like tags and plain paragraphs.

Naturally everything is still subject to change, and I suppose only tech-savvy users will be able to install it and use it now, but I plan to work on it some more in the near future.

Happy writing!

2009-11-11: Recent Hacktivity Log (11-November-2009)

For a while before, I felt like I was under-productive and that I had no incentive to write a new recent hacktivity log. This has changed some weeks ago, after a sprout of productivity where I did many small things. So here's another boring recent hacktivity log, which you are free to skip.

In the Perl-Begin front, I've added Matt S. Trout's essay "But I can't use CPAN!" to the CPAN page of perl-begin.org. Thanks Matt (mst)!

After that, I finally tended to the fact that the header image was too narrow and generally resulted in a lot of padding in direct green on the side. I took a photo I took and placed on Flickr (under the Public Domain) and tweaked it using the GIMP.

I've also done many CSS tweaks and the people on #css were a lot of help. I discovered that a co-maintainer added some JavaScript to dynamically modify the CSS and which I've mis-tweaked, and had to fix it, and convert it to jQuery. The people on #css said it was not necesssary at all and could be done with pure-CSS, but I didn't like their solutions. It also sparked a small discussion about our use of Sass, and whether CSS should be generated at all.

I've made some progress with the 5th part of Perl for Newbies, and have started to cover the authoring of a CPAN-like module using Module-Starter and Module-Build. I've been feeling quite unmotivated to work on it, but I hope this feeling will pass.

I've discovered the XML-Grammar-Fortune tests started failing, and realised it was because XML-LibXSLT started outputting the XML with indentation. I had to workaround here by trimming leading whitespace from both the expected and the received outputs. Le sigh.

Later on in the past two days, I decided to make the XML sources of the fortune cookie files be served with a client-side XSLT stylesheet so they'll be nicely displayed as XHTML in the browser. This turned out to be hard because the XSLT I used internally in XML-Grammar-Fortune was not accepted by either Firefox or Opera, and their errors left a lot to be desired. Eventually, I was able to understand the Opera errors (which were more specific than Firefox's) and while referencing the XSLT specification, was able to fix the brokenness and get what I wanted to work.

I've also did some work on some of my screenplays and stories, among them the ideas page of the Blue Rabbit's logs, and some added text to "Star Trek: We the Living Dead".

Also in regards to my homepage, I fixed the style of the breadcrumbs' trail at the top, and simplified what I need to do to over-ride some stuff in the Website Meta Language headers.

As I noted in a previous entry, I also released XML-Grammar-Fiction. I have some plans of creating a binary package for MS-Windows with tools useful for writers based on the XML-Grammar-* work, but this will require a lot of polishing of the code. Meanwhile, it would be useful for people who are more tech-savvy.

I've done some work on the Catalogue of (program) optimisations over at the Algorithms wikia. I should note that it is my initiative based on my call for that in my Optimizing Code for Speed essay, which I mostly wrote last November (although published somewhat later).

By encouragement from someone, I did some work on my script to collect mailing lists from the Perl 5 Wiki, and on normalising the format of the Perl 5 Wiki entries.

Recently, in the past day or two, all KDE applications broke on Mandriva Cooker (Cooker is the Mandriva bleeding edge distribution) due to a Qt problem. As a result, I had to work on my Debian "testing" partition. I finally ran "apt-get update ; apt-get dist-upgrade" there again, and wanted to work on my homepage. In order to do that, I need some CPAN packages. I first tried to prepare and install a package of CPANPLUS-Dist-Deb using dh-make-perl, but the tests failed, so I did not want to risk installing it. Furthermore, the "deb" line in the POD only had an entry for Debian "unstable" - not for Debian "testing" which is what I have. Then I tried to install local-lib from the Debian repositories, but it wasn't there, so I followed the bootstrapping instructions on the POD, and was able to install it on my account and install some modules using it - enough to build my homepage.

2009-11-12: הכרזה: מפגש פרל ב-17 בנובמבר ומפגש "מה היא תוכנת קוד-פתוח?"

הדבקה זו מהווה הכרזה ופרסום של אירועים קרובים של קוד-פתוח בארץ.

מפגש פרל ברחובות

ביום שלישי, ה-17 בנובמבר 2009, יערך המפגש הראשון של Rehovot.pm במסגרת חידוש מסורת מפגשי שוחרי-הפרל הישראלים. ראו גם את דף התיאום בוויקי.

מיקום וזמן: מכון ווייצמן ברחובות, בנין לווין (מס' 30) חדר 101 בשעות 18:00-22:00

על הפרק:

הדוברים במפגש יהיו הדר לוי אהרוני, אבישלום שליט וגאבור סבו. ההשתתפות חינם. אם אתם מתכוונים לבוא נא להודיע לגאבור למטרות הערכת כמות המשתתפים.

ברוכים הבאים לתוכנה חופשית: "מה היא תוכנה חופשית ופתוחת-קוד

ביום ראשון בתאריך 22/11/2009 בשעה 18:00 תתקיים הרצאה של מועדון הקוד הפתוח התל-אביבי במסגרת סדרת "ברוכים הבאים לעולם של תוכנות קוד פתוח" בנושא "מה זאת תוכנה חופשית ופתוחת-קוד ולמה זה טוב?". את ההרצאה יעביר אורי עידן והיא תתקיים במסגרת סדרת ברוכים הבאים ללינוקס בתלוקס ותתקיים באוניברסיטת תל-אביב, בחדר 007 במסדרון הבניין למדעים מדוייקים.

אנו נכסה את האידאולוגיה והפילוסופיה מאחורי תוכנות חופשיות ופתוחות קוד, נסביר מה מייחד את לינוקס ומערכות הפעלה פתוחות אחרות ממערכות הפעלה סגורות וכן את היתרונות הגלומים בהן בתור היותן חופשיות.

התחלנו להכין כרזה (פוסטר) לצרכי פרסום הסדרה שניתן להדפיס ולתלות.

2009-11-15: Perl.org facelift , a new learn.perl.org Tutorials Page, and an IRC bit

In case you haven't heard the central *.perl.org sites got a facelift and are looking much better now. Moreover, learn.perl.org finally has a page with links to some high-quality and recommended online tutorials, which is in turn based on the corresponding page from perl-begin.org. I'd like to thank Matt S. Trout and apeiron for giving some commentary and/or revisions of the initial patch.

And finally, a funny IRC conversation that took place on Freenode's #perl the other day. Enjoy!

2009-11-18: דו"ח על פגישת הפרלאים הישראלים אתמול

טוב, אתמול הלכתי לפגישה של שוחרי הפרל הישראליים, בנושא MATLAB , PDL (Perl Data Language) ומה שביניהן. למי שלא מכיר, מדובר בחבילות לחישובים נומריים, המספקות טיפול מהיר ונוח בוקטורים, מטריצות וטנזורים מסוגים אחרים (שיכולים להגיע לגדלים ניכרים), וביצוע פעולות על כל האיברים בטנזורים, כמו כפולה בקבוע, כפולה איבר איבר, כפל מטריצות, חיבור, חיסור, חילוק וכו. מאטלאב כאמור התרחבה ונותנת יכולות רבות ומגוונות על סמך הבסיס הזה, אבל זה הרעיון הכללי. PDL הינה תשתית בעלת תחום-עניין דומה לפרל.

בכל מקרה, התחלתי להתכונן לפגישה אתמול - הדפסתי את המפה, התרחצתי וחפפתי שער, בדקתי את זמני רכבת ישראל (האתר החדש דרך אגב עובד מצוין לפי מה שבדקתי בפיירפוקס על לינוקס - כל הכבוד!), ארזתי תיק, אכלתי ארוחת צהריים ויצאתי לדרכי. ברכבת פגשתי גברת שירדה באותה תחנה כמוני (רחובות) והיא הציעה לי להסיע אותי לפתח המכון. נסעתי איתה ובדרך דיברנו על החיים של שנינו. זכור לי הרבה מהשיחה, אבל היא לא קשורה במישרין לנושא. בכל מקרה, ירדתי בעבר השני של הכביש וראיתי קרוב שתי חומוסיות. מכיוון שהייתי רעב והיה זמן רב, החלטתי לקנות מנה פלאפל ב"פלאפל וייצמן" ואכלתי אותה בדרכי (פלאפל מצוין, דרך אגב). נכנסתי וקצת הסתבכתי בכניסה עם האבטחה, מכיוון שלא זכרתי את שמו של חיים פרילוסקי שהיה המארח שלנו (להבא אדפיס את הדף עם ההכרזה.), אבל בסוף הסתדרתי. כשהגעתי לבניין (בניין לוין, מס' 30 במפה) עדיין היה זמן רב וחדר המחשבים בו נפגשנו לא היה פתוח, אז שתיתי ממטהר המים, וסתם הסתובבתי וחשבתי על כל מיני דברים.

בסוף אבישלום שביט (שהיה אמור להרצות) בא ויכולתי לדבר איתו ומעט אחריו בא לאון, שלמד בטכניון ועכשיו משרת בחיל האויר, והוא איש של פייתון בעיקר. בהתחלה חששנו שלא יבואו אנשים רבים, אבל בסוף החדר היה כמעט מלא. חיים פרילוסקי לא היה יכול לפתוח לנו את החדר מכיוון שהרגיש לא טוב, אז הוא שלח מאסטרנט בשם ערן במקומו, שפתח לנו את החדר ושאל אותנו מספר שאלות.

מכיוון שהמרצה הראשונה עדיין לא הגיעה החלטתי לפתוח את ההרצאה ב-book report על הספר remix מאת לורנס לסיג, שם דיברתי על הרשמים שהיו לי ממה שקראתי מהספר. המרצה אז הגיעה והייתי צריך להפסיק.

בחלק הראשון של ההרצאות עשינו סבב הכרויות בו כל אחד הסביר מעט על עצמו ומה הוא עושה בפרל, מאטלאב ו/או PDL. רשמתי קצת על האנשים בדף בויקי. אז התחילו ההרצאות: הדר לוי אהרוני הסבירה קצת על מה זה מאטלאב והיתרונות שבו והראיתה קצת שימוש. אבישלום שביט הראה באמצע ההרצאה מספר יתרונות של מאטלאב. דיברנו על כך שרק המאטלאב הבסיסי עולה כ-5,000 דולר למחשב, ושתוספים רבים עולים עוד כסף, כך שכל הפונקציונליות יכולה להגיע עד כ-25,000 דולר. אולם סטודנטים מקבלים אותו בחינם. יש מהנדסים רבים שמתרגלים כל כך לעבוד במאטלאב ומשתמשים בו למשימות רבות כמו ניהול קבצים ועיבוד טקסט להן הוא לא כל כך מותאם.

נאמר שם שאם רוצים לנתח קובץ טקסט המכיל בתוכו מספרים (parsing) הרי שבפרל זה יהיה בד"כ מהיר בכפי 10 מאשר אם משתמשים במאטלאב לשם כך. למאטלאב יש יכולות מדהימות של אפשרות יצירת גרפים והיסטוגרמות, אפשרות לחולל קוד למעבדי אותות דיגטליים (Digital signal processors) ולקמפל קוד מאטלאב לקוד בסי, יצירת מנשק משתמש בנוחות, אפשרות לכתוב הרחבות למאטלאב ב-C, ודברים נוספים אחרים. כמו-כן סיפרו שמערכת העזרה של Matlab היא מקיפה מאוד והושקעו בה זמן ומחשבה רבים וניתן ללמוד נושאים רבים במתמטיקה רק ממנה.

בהפסקה, גאבור הוציא את המאפים שקנה (szabgab++) ואכלנו לתאבון, ושוחחנו. יצא לי לדבר עם אמיר שכרגע עושה תואר שני בבלשנות באוניברסיטה העברית, וסיפר כמה השימוש בפרל עזר לו בתואר, ושהוא יכול בעזרתה לנתח ספרים שלמים ולהוציא מהם את מה שהוא צריך, והוא לא יודע איך סטודנטים אחרים לבלשנות שלא יודעים תכנות, מסתדרים. לטענתו, קורס מבוא לפרל (או לשפה דומה) צריך להיות קורס חובה בסמסטר הראשון של הלימודים. גם דיברנו על הוויקיפדיה ועל "הוויקימילון", שלטענתו (ולדעתי) אינו מספיק מובנה ושהפורמט של וויקי פחות מתאים עבורו. אז הזכרתי לו שאני משתמש בוויקיפדיה העברית בתור מילון אנגלי↔עברי והוא אמר שגם הוא עושה בו שימוש ושהוא כתב תסריט בשביל לעזור בכך שהוא מחפש עבורו מישהו שיעזור לו בלתחזק אותו.

גם ראיתי שם שוב את רינה משוחרי הפרל הירושלמים שלא פגשתי כבר זמן רב, ודיברתי איתה קצת. אני מקווה שהם יוכלו בקרוב להמשיך בפגישות שלהם בירושליים (אם כי לי קצת קשה לבוא לשם מפאת המרחק והגישה הדפוקה לשם).

בכל מקרה, אחרי ההפסקה נמשכה ההדגמה של מאטלאב עם כל מיני הדגמות מגניבות, ואחר-כך נאמר מה צריך כדי ליצור אלטרנטיבה טובה למאטלאב: סביבת עבודה טובה, תאימות טובה (לא חלקית שכל הזמן (נתקעים), ודברים שאפשר לשפר מעבר למאטלאב. בכל מקרה, הודגמו שם מספר דברים חביבים במאטלאב כמו סגורים (closures) - למשל ב-octave:

octave:4] f1 = @(x) @(y) { x+y }
f1 =

@(x) @(y) {x + y};


octave:5] f2 = f1(6)
f2 =

@(y) {x + y}

octave:6] f2(100)
ans =

{
  [1,1] =  106
}

octave:7] f2(50)
ans =

{
  [1,1] =  56
}

בכל מקרה, לאחר מכן גאבור עלה על הבמה, והדגים את ה-REPL (ראשי תיבות ל-Read-Eval-Print-Loop) של PDL בשם perldl ודברים מגניבים אחדים בו (שככל הנראה רחוקי םלהגיע לרמה של MATLAB אבל עדיין חביבים למדי), כמו עזרה אינטראקטיבית ודמואים ובכללן תצוגות תלת-מימדיות. כמו-כן הסביר על כך שהחזון שלהם ל-Padre הוא שיהיה לו תוסף ל-PDL שיאפשר פיתוח יותר נוח ודומה יותר למאטלאב. כך למשל הגרסה הבאה של Strawberry Perl תכלול את Padre ואת PDL בפנים. הוא סיפר שכשנתן מצגת על Padre בכנס אחד באירופה הוא נתן רשימה ארוכה של תכונות שלו ואמר "הן לא ממומשות עדיין. עכשיו אתם צריכים לממש אותן." ואכן חלקן הגדול מומש מאז.

אז היה כבר אחרי 20:00 והתכוונו להתחיל את ה-hacking session, אבל אני עזבתי עם עוד תל-אביבי אחד בשם דותן כדי לתפוס את הרכבת האחרונה. בדרך דיברנו על דברים רבים כמו: CGI-Application, CGI-Prototype, The Catalyst Web Framework, JavaScript, jQuery, KDE 4.x, XFCE, תרפיה קוגניטיבית-התנהגותית ועוד נושאים.

אז סך הכל - היה נחמד מאוד ונהניתי מאוד. זה היה שינוי מרענן בשגרה עבורי. עכשיו אם תסלחו לי, יש לי קוד פרל לכתוב. אני מקווה לראות אתכם בפגישה של מועדון הקוד הפתוח התל-אביבי ביום ראשון.

2009-11-22: Numeric Plans for Tests and Test-Count

Adam Kennedy (Alias) writes in use.perl.org about "the unfortunate demise of the plan.". I've read his post and all of the comments that were posted there up to now. I agree with Adam that a numeric test plan is usually preferable to a done_testing() one, because in one of experiences with a failing test script (in a DBI-related module), I noticed that it generated a different number of tests depending on the version of one of its dependencies. So only a subset of tests run in one of the cases, and this was enough for me to realise that I should make sure that an exact number of tests run.

Eventually, I came up with Test-Count, which allows one to keep track of the number of tests inside a Test::More script using a domain-specific-language embedded in comments next to the tests. This way, one can have more faith that they have run the correct number of tests.

Personally, I found that some people take some time getting used to writing the Test-Count annotations, but for me it's been a great boon. I do not rule out that done_testing() has some uses or may be better in some clear-cut cases, but since many of my tests are data-driven or use loops or helper functions, then I always prefer to have a numeric plan and to keep it up-to-date using Test-Count.

At the moment, Test-Count is tailored for my own needs and many important features there may be missing. However, bug reports, feature requests and especially patches are more than welcome.

2009-11-27: Changing the Seldon Plan

In Isaac Asimov's book "Second Foundation", the second foundation are in charge of the Seldon Plan, the grand "psychohistory" plan for the advancement of the galaxy, and one thing that is mentioned there is that it should by no means be considered holy and that changing it according to the circumstances is expected and even necessary. This mentality has an affect on us as open-source programmers, who should do our best to remedy bad open-source code as we find it.

Recently, in the Perl world there has been a continuous trend to shift from Ancient Perl to Modern Perl, and often people ask for our help on Perl 5 code they found (possibly on CPAN - possibly elsewhere) with "Ancient Perl" paradigms. That doesn't necessarily mean that Perl 5 code is very old, because often people who wrote it didn't know any better, and there are plenty of Perl 5 (or even Perl 4) tutorials or books still floating around the Net teaching bad practices. As a result, we normally tell them to either abandon or convert the code completely for a better trusted and more modern CPAN module, or alternatively to modernise it.

So far, so good. However, some people who've asked for us help said something like "I do not consider myself capable of rewriting his module and [it] has demonstrated robust stability over the years [so] I have little reason to.". So essentially, all the comments that we have given on it were dismissed due to apparent lack of competency on the part of the code's end-user, and that Ancient Perl code will still linger in use. The link I gave is not the only place where I saw it - I also saw it in at least another place (though I think it was on IRC).

I think that this spirit stands in opposition to the spirit of open-source and possibly even the Hacker Ethos. By all means, if we consider all the millions of lines of open-source out there as the Seldon Plan which powers all the open-source programs in use, and empowers us and end-users, then we should not feel detrimental or afraid to change the code that we use everyday. And people should not deploy code that they are not confident enough in changing, modernising and adapting to their needs.

One cannot usually expect code to remain the same forever. As time progresses, we can expect there to be code rot, features that needs to be added, tests that need to be added, or as is often the case in Perl 5 and other languages, paradigms that are considered to be bad and that should be changed to newer and safer paradigms. If you're lucky, the original developer or a co-maintainer can do that for you, but sometimes an end-user needs to stand up and volunteer to do that.

The open source nature of "Free and Open Source Software" permits everybody to create derivative works of the software and distribute them. So we don't have legal reservations from improving them and contributing our improvements to the general public, and should not have any moral or ethical ones, either. "Hacker sees bug. Hacker doesn't want bug. Hacker fixes bug.".

2009-11-30: Random Stuff: Word Count Vim Tip, Mirroring CPAN Using rsync, and Mozbot

This is a "random stuff" entry because I'm too lazy to write separate entries for each of the topics.

First of all the vim tip: you can do word count in Vim, by selecting a text using "v" and then typing "g" and then "Ctrl+g". This will display statistics on the text including word count. I always assumed it existed in a form but didn't had to use it until recently when I wrote a scene in a screenplay and wanted to see how long it was. This tip can also be found at ":help word-count".

The second tip is that in order to mirror a mini-CPAN that contains only the most recent releases using rsync instead of minicpan's HTTP or FTP download, one can use mst's mirror.sh in conjuction with the list of CPAN rsync mirrors. I needed to modify mirror.sh slightly in order to mkdir the "work" directory which it requires. I changed:

cd $LOCAL

to:

cd $LOCAL

if ! test -e "work" ; then
    mkdir "work"
fi

Finally, a little "recent hacktivity log": at one point someone on irc.mozilla.org asked me if I could help them with their IRC bot called firebot and yesterday I decided to take a look at its core called "Mozbot", and maybe try to improve it. I took a look and detected many vestiges of "Ancient Perl" there, and other anti-patterns such as long subroutines or use of Net::IRC and decided to work on improving it.

After reading the "Using Mercurial locally with CVS" document on the Mozilla site (first hit in a Google search for "hg cvs") and setting up a "mozbot-shlomif" Mercurial repository to work on it, I submitted my first patch and have done more changes in the mozbot-shlomif repository. The code is not perfect, but at least it has "use strict;", the "-w" flag and the "-T" flag, so it could be much worse. I'd still like to move away from Net::IRC to POE-Component-IRC and maybe even try to port the entire Mozbot to buu's buubot. But it will take some time.

2009-12-07: Studying Moose Top-Down

This is a post about Moose the post-modern Object-Oriented Programming system for Perl 5. Until a few days ago, I've learnt most of what I knew about Moose by reading some presentations about it and experimenting. However, a few days ago, I've decided that maybe there was a more-Moosey way of doing the following pattern that I've been doing in my classes:

# Super-base class.
sub new
{
    my $class = shift;

    my $self = {};
    bless $self, $class;

    $self->_init(@_);

    return $self;
}

# In derived classes:
sub _init
{
    my $self = shift;
    my $args = shift;

    $self->_init($args);
    # Initialise from $args.

    return;
}

Turns out there was - you can use the BUILD method to provide initialisation for each part of the class hierarchy (it gets calls in a walk-method style - not only for the most qualified sub-class). Only in my case in my en-Moosification of XML-Grammar-Fiction, I only needed a 'default' callback to the class members, and to make them 'lazy' by default.

This made me conclude that I have to learn Moose top-down to become more familiar with it. So I started reading the chapters of Moose::Manual and have already learned that I could encapsulate push() and shift() out of an array ref using its delegation, which also proved useful in XML-Grammar-Fiction. After I'm done with the manual, I'm planning to go over the Moose cookbook.

Hopefully, this will give me a lot of ways to avoid repetitive code in my CPAN modules. Of course, now, I'm facing the classic dilemma of whether to make something a base class or a role… decisions, decisions.

2009-12-11: Recent Hacktivity

Happy Chanukah, everyone! Lately, I've had little energy for blogging, and have been coding a lot. I suppose that's a good thing, but I'd like to keep up with my Planet Perl Iron Man statistics. So here we go - another recent hacktivity report.

As I said, I've went over the Moose::Manual and learned many nice new tricks to make my Moose-using Perl modules more Moosey. I've implemented some of them in XML-Grammar-Fiction and also on Test-Run, which is a much more massive codebase.

While I worked on Test-Run I noticed that some of the Test-Run-CmdLine tests failed, and realised the problem was a MooseX-Getopt bug which I was able to supply a test for, and which involved a very short fix. While I was in the neighbourhood, I also contributed a test for handling the Getopt::Long no_ignore_case which I used as well, and the results were released in MooseX-Getopt-0.26 by Tomas Doran.

I also ran into an undesirable behaviour with MooseX-StrictConstructor which I was able to workaround with some help from the #moose people: when using it inside a class which has a base class, one needs to do BEGIN { extends ("BaseClass"); } instead of just extends("BaseClass");. I didn't understand why, but it just works.

I'm now reading the chapters from the Moose Cookbook, which will take me some time.

After that, I finally did the work I promised on the list of Perl mailing lists. There's still a lot to fill in, but the YAML/DBI foundation is in place.

I worked on a Ruby program which I originally started writing in order to learn Ruby. I've added some tests, and refactored it. I've got bitten by a case where I overrided a variable called yx as a parameter to a closure and it got clobbered afterwards. This is one reason why I prefer Perl 5's (and Scheme's) explicit lexical scoping using "my", which provides better protection against such mishaps.

I continued to work on Mozbot, but my original patch was not applied yet, so I don't know if there's any interest in further work on the internals of the robot.

I've made some progress with reading the book xUnit Test Patterns: Refactoring Test Code. One of the insights I gained so far was that the failed tests should point what became broken without too much debugging. The book is off to a slow start with many overviews and introductions, which is kinda annoying, though.

Finally, I attended a meeting of PHP-Israel where there were a lot of Bourekas, and were someone gave a presentation about PHP Unit and writing unit tests. There was a discussion of whether we should only test the external behaviour of a class, or also its protected/private methods, and if the latter - how (there was some way to do it in PHP). One thing I noticed was that some of the PHPers used Eclipse for their development, and there was some discussion about how to get it to behave properly on Linux.

A few of the PHPers also thought that they would love to use JavaScript on the server. It's funny, but as a Perl 5 programmer (with experience in many other languages) I look at JavaScript and see many horrible aspects, and missing features. However, the PHPers seem to think it's better than PHP. Maybe it's the Blub Paradox.

2009-12-16: #perl and ##programming fortune collections

I've split a separate file for fortune cookies made out of Freenode #perl conversations, and another one for those taken from ##programming. The motivation for them was to make my "shlomif" collection smaller and more managable, but at least it's now in the same place. I should note that you can find the XML version of the file by changing URL's the to .xml and the text version by changing it to a blank one. (Memo to self: link to them from the HTML page). The main page of the fortunes contains more information.

Enjoy!

2009-12-19: GNU Make Tip - Speeding Up By Using the -r / --no-built-rules Flag

By default, when GNU make runs makefiles it keeps looking for many default targets like ",v" and a lot of other junk. If your makefile has no default targets (especially if it's a makefile for a non-C/C++ project), you can speed things up considerably by saying "make -r $TARGET" or "make --no-built-rules $TARGET" instead of the default target.

Here is a comparison of its performance on my homepage, after it was completely built:

shlomi[homepage]:$trunk$ time make
make: Nothing to be done for `all'.
2.03user 0.07system 0:02.21elapsed 94%CPU (0avgtext+0avgdata 19344maxresident)k
0inputs+0outputs (0major+2025minor)pagefaults 0swaps
shlomi[homepage]:$trunk$ time make -r
make: Nothing to be done for `all'.
0.13user 0.03system 0:00.18elapsed 89%CPU (0avgtext+0avgdata 12256maxresident)k
0inputs+0outputs (0major+1582minor)pagefaults 0swaps
shlomi[homepage]:$trunk$

Very significant!

In order to avoid specifying it times and again you can make it the default by setting the MAKEFLAGS environment variable (which isn't documented too well) to "-r".

How I discovered the tip? Well, I occasionally used "make -d" for debugging my makefiles, and noticed it was trying to apply a lot of implicit targets. Having understood that they may be very time consuming, I ran "man make" to see if there was a way to avoid running them. And lo and behold, they made running make on my homepage much faster.

2009-12-21: Why You Should Not Use Python .pyc Files for Hiding Source Code

A few months back, I've spoken to a Python programmer (who used to program in Perl) and among the many things we discussed, he said that one feature he appreciated about CPython was that you could take the Python .pyc files, which contains compiled Python bytecode, and ship them without the .py source files in order to hide the source code from people he's distributing it to, so it will be harder for them to reverse-engineer and study the code.

I told him that with a symbolic language such as Python, such .pyc files do not provide adequate "protection" as they contain the names of identifiers and other detailed information on the code. At one point, he even thought that they are compiled C code, but I told him CPython can work pretty well on machines without any kind of C compiler.

On Freenode's #python, people seemed to have agree with me (I am rindolf):

<rindolf>  Another question: can I depend on compiled python
bytecode (.pyc) for "hiding" code? Doesn't it still contain all the
identifiers verbatim?
<lvh>  rindolf: No. You cannot hide code, stop trying.
<kniht> rindolf: why are you trying to hide code?
<DeadPanda>  rindolf, check the new IEEE Security and Privacy (if you can),
you can't hide Python code
<DeadPanda>     rindolf, unfortunately, there's nothing you can do.
Otoh, you can probably do the bare minimum and make your boss happy.
<kniht> rindolf: because these final users are liars and cheats? if that's the
business plan, not sure what I can say

Python knows the identifiers of the variables at run-time. For example:

shlomi:~$ cat exec-test.py
#!/usr/bin/env python

import sys

a = "I am a"
b = "I'm b"

exec(sys.stdin.readline())
shlomi:~$ python exec-test.py
print a
I am a
shlomi:~$ python exec-test.py
print b
I'm b
$ python exec-test.py
print a.upper(), b
I AM A I'm b
shlomi:~$

Even if you're not using exec(), eval() or friends, python still has to accomodate for them being potentially used and as a result keeps this information in the bytecode. My partner said he doesn't use eval and friends because they are "a bad programming practice" and as a result thought he was safe. However, that's not the case.

He told his clients that Python bytecode was only marginally worse than Java and .NET bytecode which "are used to protect the code of highly sensitive IDF and US millitary applications - bytecode is sufficient protection." However, according to this Slashdot thread "java is [a] cake to reverse engineer".

So it's not adequate protection, and Python is even substantially less than that.

I next suggested he may opt to use obfuscators to obfuscate his code, and he said that:

They can't sue me, they'll have to sue whoever reverse engineered the code - I believe it's illegal to reverse engineer commercial compiled bytecode - it isn't however to reverse engineer obfuscated code far as I know -- it's not a technical issue, it's a legal/business one.

I don't understand the distinction between obfuscated code and bytecode in this case. And like I told him "Any sufficiently advanced obfuscation is indistinguishable from bytecode.".

Talking with a different Python programmer, he told me that .pyc were considered adequate "protection" for them, despite the fact there are several .pyc disassemblers and decompilers present.

In short, depending on .pyc's for protecting your source code, only provides fig-leaf protection. So it's not a substantial advantage of CPython over perl 5, which doesn't cache the bytecode in ".pmc" files by default. (even though it has a bytecode that it uses internally in memory).

As a postscript I should note that sometimes after that my conversation partner said that to me:

BTW, I looked more info python bytcode and you're right - you can get something almost identical to the original. It still requires some tikering but there are commercial solutions for decompilation that produce something pretty close to the original for 5 euro per 5kb for personal code or 100 euros per file for commercial code

2009-12-26: Directory Tree Traversal In Various Languages, C# and a funny IRC conversation

While looking for some sample C#/.NET code (more about it below) and doing a search I stumbled upon this page in Rosetta Code that contains directory tree traversal codes in various languages. I've been meaning to do a comparative blog post comparing File::Find, File-Find-Object and File-Next to directory tree walking APIs in other languages, but now it will be easier. Of course, the page is not perfect - for Perl, it only gives an example using File::Find (guess I'll have to add the other ones) and in many cases (such as C) it gives a custom, recursive solution instead of using an API such as libfts.

Still it's a pretty cool page. Now I fear I'll get the feeling that "it' s already been done and so I shouldn't do it" and so won't write that post.

The reason I was looking for C# information was that one of my Perl programs ran too slowly. It was written to find a good heuristic for Freecell Solver, and used a menial algorithm: check the most optimal solution for the first iteration with starting quotas between 100 and 1,000; Then using the result, check them for the second iteration, etc. The Perl program used PDL (the Perl Data Language), but: 1. It was wrapped in a lot of Perl OO code and 2. it had a lot of pure-Perl algorithmic code that I did not figure out how to properly PDLise yet. As a result, I was looking for a faster language to port it to.

I originally considered using C, which I know pretty well, but then thought about trying out C#/.NET using its open-source Mono implementation. At the end I spent a day and a bit of the evening beforehand writing 409 SLOCs of C# code, and 113 SLOCs of testing Perl code for the task (according to SLOCCount by David A. Wheeler) and that was after being delayed due to many web searches to find out how to do a lot of stuff. The reason I wrote the testing code in Perl was because I lacked the energy to learn NUnit, but now that I think of it it may be useful to test the original Perl version (which is still of some utility to me) too.

At the end of the day, the C# code worked and worked well. Mono programs start up quickly, and the C# code I wrote ran much more quickly than the Perl one, which saved me the time of waiting for so long. I think I like C# so far. It seems much nicer than Java, and with less protocols and some syntax niceties and useful APIs and it also starts up much more quickly. I still find that writing Perl 5 code is quicker, but if your Perl/CPython/CRuby/etc. code is under-performant, then C# and .NET would be viable choices.

(I should note that I've still barely scratched the surface here. A single-threaded 409-line-strong command-line program is hardly indicative of any possible advantages or alterantively pitfalls you may run into with C# and .NET. But so far what I tried worked and nicely.)

And finally, a funny IRC conversation. It's been a while since there was something on Freenode's #perl that I found really funny, but now the wait is over:

Hope you enjoyed this entry.

2009-12-29: KDE 4 Tip for KMail, Kopete, etc.: Right-aligning Text

For a long time, I've been bothered by the fact that on the Linux/Unix KDE 4 desktop on applications such as KMail, and Kopete, the Hebrew text was left-aligned. I expected it to be right-aligned when it encounters a leading right-to-left text, like it was on KDE 3, or now is the case in Kate (the KDE text editor). However, there is a way to align the text to the right for display.

To do so, press Right-Ctrl and Right-Shift on the keyboard (possibly while selecting the text). Like in Windows. It's much better in KMail than in Kopete, where you have to right-align the text this way after every message. But at least it makes life with KMail much more bearable. And please try to reproduce and comment on my "Hebrew Messages are Displayed with Jumpy Margins" bug in KMail.

Cheers and a happy new calendar year.

2009-12-30: Vim Tip: Finding the First Non-Matching Line from the Cursor

For a long time I've been viewing and editing vim text files, for example those of the UNIX find output or of archive contents, that contained many consecutive lines of similar data. What I wanted to do is find the line where these consecutive data end where they no longer end. Among the workarounds I tried was jump to the end of the file (using G) and then search backwards for the last matching line (which may not be very reliable). But a few days ago I decided that I had enough and sought a better way, and I found one.

You can do it using a negated pattern. If for example you enter / home\/shlomi\/\(Arcs\/\)\@! you will find the first line that contains "home/shlomi" that is not followed by "home/shlomi/Arcs/". For information see :help \@!.

2009-12-31: Cygwin and Perl Save the Day!

In my house, we have a Windows XP desktop computer that serves the family (while I'm normally using my Linux desktop machine.). A few days ago my father complained that drive C:\ there, which contains all the programs is almost completely occupied, and said we need to remove programs from it. Now this drive is roughly 45 GB in size, so it seemed unlikely to me that it became entirely full because of the executables.

I decided to investigate. I switched to my account, started a cygwin shell, and ran \du . > ~/du.txt from the C:\. What du does is report the size of every directory starting from the input path recursively. Then I ran my du-output-analyser written in Perl to analyse the results, and eventually found out that 20,000,000 (20 million) 1 KB blocks are being occupied by a directory containing photos. Then I sent a report for my father with my findings.

This just goes to show how much people who are not familiar with the UNIX command line and with programming are missing and are much more helpless than the more clueful UNIX power-users.

2010-01-01: Legal: ACTA and Software Patents in Israel

This post is about two legal matters. The first is about ACTA, the so-called "Anti-Counterfeiting Trade Agreement", which is a trade agreement that aims to criminalise a lot of the openness of the Internet under the guise of fighting counterfeited goods. Michael Geist has an analysis of the provisions of ACTA and tracks its progress. This is worrying and one should better take a look.

The other issue is Software patents in Israel (see "Fight Software Patents"'s wiki entry about Israel and the official call for opinions on the Israel Justice ministry site). I started preparing my own statement document, which is still incomplete and there's a letter that Jonathan Klinger (attorney in law) has prepared). Hope we can stop this madness.

Happy new civil year!

2010-01-03: פגישת קוד-פתוח-תל-אביב: Moose למתחילים - מערכת האובייקטים לפרל

מועדון הקוד הפתוח התל-אביבי (תלוקס) ייפגש שוב כדי לשמוע את הרצאתו של ירון מאירי (Sawyer) אודות "Moose, מערכת תכנות מונחה העצמים לשפת פרל (למתחילים)". ההרצאה תתקיים ביום ראשון, 17 בינואר 2010, בשעה 18:00 (שימו לב לשינוי בשעה משנה שעברה), באולם הולצבלט, מס' 007 במסדרון הבניינים למדעים מדויקים (שימו לב לשינוי במיקום משנה שעברה) באוניברסיטת תל אביב. פרטים נוספים, מפות להגעה וכיוצא בזה, ניתן למצוא באתר ובוויקי. הנוכחות בהרצאה היא חינמית ולא נדרשת הרשמה מראש.

Moose הינה מערכת תכנות מונחה-עצמים פוסט-מודרנית לשפה פרל 5. היא נכתבה מכיוון שההוגה המקורי שלה (סטיבן ליטל) קינא במה שפרל 6 סיפקה בנוגע לתכנות מונחה עצמים, ולכן במקום לעבור לרובי הוא שקד על פיתוח מערכת דומה לפרל 5. Moose שאבה השראה מיכולות ה-OOP של שפות רבות כמו פרל 6, Smalltalk, ליספ, רובי, ג'אווה, OCaml ושפות אחרות כשהיא נשארת נאמנה לשורשי ה-פרל 5 שלה.

ירון מאירי הינו מנהל מערכות ומפתח פרל. הוא מרצה על קוד פתוח, תוכנה חופשית, אבטחה וסטנדרטים של תכנות. ירון העביר בעבר את ההרצאה על דגלים אדומים בתכנות עבור שפות עיליות ביותר במועדון התל-אביבי.

אנו תמיד מחפשים מרצים שיתנדבו לתת הרצאות בנושאים שונים הקשורים לקוד-פתוח ולמחשבים. במידה שאתם מעוניינים לתת הרצאה, או שיש לכם הצעה להרצאה שמעניינת אתכם, נשמח לשמוע ממכם.

2010-01-05: Next TelFOSS Meeting: Moose, the Perl OOP System on 17-January-2009

The Tel Aviv Open Source Club will host the talk "Moose, the Perl OOP System" for beginners by Yaron Meiry (Sawyer) on Sunday, 17-January-2009, at 18:00 (note the change of time since last year) in Tel Aviv University, Holcblat Hall No. 007 in the corridor of the exact sciences building (note the change of place from last year). Further details, maps for arrival, etc. can be found on the web-site and in the Wiki.

Attendance is free, it is not necessary to RSVP and everyone are welcome.

With any other problems, feel free to contact the organiser.

Abstract

Moose is a post-modern Object-Oriented Programming system for Perl 5. It was written since its originator (Stevan Little) was jealous of the capabilities that Perl 6 provided in regards to OOP, and so instead of switching to Ruby, he worked on developing a similar system for Perl 5. Moose drew inspiration from the Object Oriented Programming capabilities of many languages such as Perl 6, Smalltalk, Common Lisp (CLOS), Ruby, Java, OCaml and other languages, while remaining faithful to its Perl 5 roots.

About the Lecturer

Yaron Meiry is a systems' administrator and a Perl developer. He gives talks about open source, free software, security and programming standards. Yaron has previousl given the presentation about red flags in programming of very high languages in the Tel Aviv Club.


We are always looking for people who will volunteer to give presentations on various topic that are related to open source code and to computers. In case you are interested to give a talk, or that you have a suggestion for a talk that interests you, we'll be happy to hear from you.

2010-01-07: bzr is Dog-Slow and Generally Sucks

The Bzr version control system (Bazaar) is a version control system from Canonical Ltd. (the parent company behind the Ubuntu Linux distribution). While being open-source, it sucks royally and this is a blog post about this.

First of all it is excruciatingly slow. Branching postr.dev (which is a bzr branch of the very small postr project takes 2:48 minutes. And that's not all:

shlomi:~/Download/unpack/net/www/TEMP$ cd postr.dev/
shlomi:~/Download/unpack/net/www/TEMP/postr.dev$ du -cs *
4.0K    AUTHORS
20K     COPYING
80K     data
4.0K    nautilus
20K     po
4.0K    postr
4.0K    postr.doap
4.0K    README
4.0K    setup.py
256K    src
4.0K    TODO
404K    total
shlomi:~/Download/unpack/net/www/TEMP/postr.dev$ echo "Hello" >> postr
shlomi:~/Download/unpack/net/www/TEMP/postr.dev$ time bzr di
=== modified file 'postr'
--- postr       2007-07-19 17:57:09 +0000
+++ postr       2010-01-07 18:00:06 +0000
@@ -42,3 +42,4 @@
     for url in sys.argv[1:]:
         p.add_image_filename(url)
     reactor.run()
+Hello

Command exited with non-zero status 1
1.47user 0.08system 0:01.69elapsed 92%CPU (0avgtext+0avgdata 60080maxresident)k
288inputs+8outputs (0major+5673minor)pagefaults 0swaps
shlomi:~/Download/unpack/net/www/TEMP/postr.dev$ time bzr di
=== modified file 'postr'
--- postr       2007-07-19 17:57:09 +0000
+++ postr       2010-01-07 18:00:06 +0000
@@ -42,3 +42,4 @@
     for url in sys.argv[1:]:
         p.add_image_filename(url)
     reactor.run()
+Hello

Command exited with non-zero status 1
1.47user 0.07system 0:01.61elapsed 95%CPU (0avgtext+0avgdata 60096maxresident)k
0inputs+0outputs (0major+5672minor)pagefaults 0swaps

So a simple "bzr diff" command takes over 1.6 seconds.

I reported a bug in inkscape - "bzr branch lp:inkscape is very slow and emits errors and leaves a broken local repository after exit" It took bzr 3 minutes to try and checkout the Launchpad's Inkscape branch, and it finished with an error and left the repository in an unsuable state. I tried it on my Debian Testing VM and ended up with the same result.

To quote Sjors: "Bzr is slower than Subversion in combination with Sourceforge." (and that says a lot).

Recently, GNU Emacs switched to Bzr. Now the question is how many decades one will have to wait to "bzr branch" its main repository.

Cheers!

2010-01-08: libtap-1.02 Was Released

libtap is a C library to implement the Test Anything Protocol (TAP), which is used by the test suites of perl 5, CPAN modules and other programs. libtap allows one to write C code that can be analysed by TAP harnesses such as Test-Run.

I started maintaining Nik Clayton's libtap, which he seems to have abandoned on my homepage.

There's also a Subversion repository.

Reading from the NEWS file:

tap-1.02:

  1. Add the COPYING file.
  2. Now disabling the thread-safety by default (in order to have it add -DLIBTAP_ENABLE_BROKEN_THREAD_SAFE .
  3. Changed configure.in to configure.ac.

2010-01-11: New Solver for "Black Hole" Solitaire in Perl

I wrote a new solver for the solitaire "Black Hole". Here's the complete story:

Yesterday, I was going to play a game on Brain Bashers when I remembered the Penguin solitaire variant, which is similar to Freecell and which I wanted to cover on the Cards Wikia. So I did a Google search for "penguin solitaire" and saw it had a page on the wikipedia.

There I saw that it was invented by David Parlett and I saw there that he also invented a solitaire called "Black Hole"

So I looked for it in PySolFC, read the instructions there and started to play. The game involves putting a card that is one above or below the foundation (wrapping from kings to aces). I noticed that there wasn't any over-populated talon or something like that there, which meant that I could probably build a DFS-based solver for it. So I set out to see if it was possible.

The first thing I did was to adapt my PySol/PySolFC game generator to generate the initial board of the PySolFC deals. This turned out to be a very complicated and frustrating task for me, because I had to understand what's going on with the code. But after a lot of playing with it, I was able to get it to generate the initial deals. The changes for that are in the Freecell Solver trunk now.

Then I started working on the solver. I decided to write it in Perl 5 because it's a good prototyping language and I know it well. When writing it, I heavily optimised for speed and low memory consumption by using bit fields ( see the vec() function ) and a gigantic hash lookup.

Then, after it was written, came the moment of truth: I ran it on the board and it reported success. Great! But what's the solution? So I added some solution tracing logic, to output the cards that should be moved. Then I was able to play it and the game was solved. Yay!

I tried it on another game and it also worked. Then I ran it on the first few games. Game #1 was reported as unsolvable, Game #2 was solved after a while, and Game #3 consumed over 15% of my RAM and then was solved.

So it seemed to be working nicely. Today I wrapped the code inside its own CPAN package and prepared a nice app-in-a-module and a script for it. he solver's code is made available under the permissive MIT/X11 licence - share and enjoy. And as I mentioned before here is its home-page.

2010-01-18: The Quest for a Good Window Manager / Desktop Environment

I'm using Linux (Mandriva Cooker) and have been routinely starting many window managers and desktop environments as the need arises. Lately, I've been working a lot with KDE 4 and I happen to agree with OSNews.com's assertion that it is slow, sluggish, and has frequent random crashes. I'd like to move to something more usable until the KDE 4 programmers get their act together. Now, I've traditionally used IceWM but it no longer cuts it for me because it does not have a different wallpaper for every virtual desktop. XFCE fails on the same premise. Enlightenment 17 has a different wallpaper per virtual desktop, but its task bar is dysfunctional, if not completely non-existent, and it has many other very annoying quirks. GNOME is just plain annoying (and doesn't have a different-wallpaper-per-virtual-desktop either), and KDE 3 is no longer present on Mandriva. LXDE does not have it either.

Now I dislike the concept of tiling window manangers, of which there has been an inflation, recently, and I am looking for a window manager that has stacking. And like I said, it also needs to have a usable task bar, a different wallpaper per virtual desktop and be responsive and stable.

Do you have any recommendations?

2010-01-20: Report on the TelFOSS "Moose for Beginners" Meeting

Last Sunday, Tel Aviv Open Source club (TelFOSS) met to listen to Sawyer's presentation about "Moose for Beginners". Despite the fact that the meeting was well-publicised, very few people came. Sawyer did not prepare a lot of slides and since there were not many questions he finished quickly. After all that, he went to demonstrate how to implement several use cases that the audience has proposed as Moose programs, and despite that we still finished early.

I asked about implementing an "XML parser", but in fact I wanted to implement a processor for a certain XML grammar. In any case, he misunderstood and told me to use Parse-RecDescent for that. I've grown to dislike P-RD for most stuff though.

In any case, I'm getting tired of the low attendance in TelFOSS meetings. We do so much work in organising and publicising the meetings, and most people don't bother to come. Are the people too busy? Did they run out of steam? Why don't they come? This makes me frustrating and unwilling to further organise the TelFOSS meetings.

2010-01-21: Perl: Using Vim's snipMate for Perl 5 snippets

I noticed that there was some Perl 5 code that I had to type or copy-and-paste again and again on many occasions. So I decided to find a way to put it in one place and then recall it. I thought of writing it myself, but then recalled the snipMate snippets extension for Vim (which Peteris Krumins covered in a blog post), which allows that. After a little reading, I was able to prepare the snippets.

I placed the following under ~/.vim/snippets/perl/_slurp.snippet :

sub _slurp
{
	my $filename = shift;

	open my $in, "<", $filename
		or die "Cannot open '$filename' for slurping - $!";

	local $/;
	my $contents = <$in>;

	close($in);

	return $contents;
}

And now I can type "_slurp<TAB>" to recall it. The existing "slurp" snippet in ~/.vim/snippets/perl.snippets is quite evil, with an ugly inline line, typeglobs, and a two args open without a die statement. Thanks, but no thanks.

Afterwards I added the following in ~/.vim/snippets/perl/_ltestb.snippet

	local $Test::Builder::Level = $Test::Builder::Level + 1;

And now I can type "_ltestb" (short for "local Test::Builder") and put this line there to create my own custom Test::More/Test::Builder tests.

Thus, using snipMate, you can create your own short-hands for commonly-used snippets like that.

2010-01-22: The CPAN Dependencies of Website Meta Language

In the past few days I resumed work on converting the Website Meta Language (WML) build-system from the GNU Autotools (also known as "GNU Autohell") to CMake. This involved implementing a lot of the existing makefile and autoconf logic in CMake and Perl.

Yesterday, I converted the "wml_common" sub-directory which turned out to contain some ancient versions of CPAN distributions with a forced-upon way to install under the WML prefix. I decided to forego installing all of them and just check for their existence in the perl version that wml uses using some WML and "perl -MMyModule" code.

I was able to find all the needed modules in the Mandriva repository, except one - File-PathConvert , which I've never heard about. I checked it on CPAN, and saw that it was deprecated and superseded by File::Spec and Cwd. So I grepped the tree for its use and found it was used only in one Perl program which I quickly converted to File::Spec and Cwd.

Now, the future build system of Website Meta Language will be simpler and it won't use deprecated modules.

Oh! And Firefox 3.6 (Gamma) is out! I've been using the betas and RCs on Mandriva Cooker for some time now, and now I've upgraded to the stable version and it's still great.

2010-01-25: XML-Grammar-Fortune version 0.0200 Was Released

I recently released version 0.0200 of the Perl 5 CPAN Module XML-Grammar-Fortune. XML-Grammar-Fortune is a module to maintain a collection of Unix-like fortune cookies as XML and convert them to XHTML and plain-text.

New in this release are a heavily fixed and enhanced code for rendering the XML to plaintext and a more enhanced META.yml file. XML-Grammar-Fortune was successfully utilised for my collection of Fortune Cookies.

2010-01-30: Project Euler Problem #10 in Haskell, Perl and C

zerothorder told how he found a solution for Project Euler's Problem 10 in Haskell. The problem is "Find the sum of all primes less than 2,000,000". It is given here below:

primes :: [Integer]
primes = 2 : filter isPrime [3, 5 ..]
    where
        -- only check divisibility of the numbers less than the square root of n
        isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes
        divides n p = n `mod` p == 0

result = sum $ takeWhile (< 2000000) primes

main = do putStrLn( show result )

He says "If this doesn't give you a nerdgasm, I don't know what will.". The problem is that this nerdgasm will last a long time. Benchmarking this program gives that 20 iterations of it run at 310 seconds - less than - about 15 seconds each (on my Pentium 4 2.4GHz machine running Mandriva Linux Cooker). So next I tried a better Haskell implmenetation that I recalled from a thread I started in the Haskell Café mailing list about implementing a sieve of Eratosthenes in Haskell:

import Data.Int

primes :: Int64 -> [Int64]

primes how_much = sieve [2..how_much] where
         sieve (p:x) =
             p : (if p <= mybound
                 then sieve (remove (p*p) x)
                 else x) where
             remove what (a:as) | what > how_much = (a:as)
                                | a < what = a:(remove what as)
                                | a == what = (remove (what+step) as)
                                | a > what = a:(remove (what+step) as)
             remove what [] = []
             step = (if (p == 2) then p else (2*p))
         sieve [] = []
         mybound = ceiling(sqrt(fromIntegral how_much))

--main = print (length (primes 1000000))
main = print (sum (primes 2000000))

This does not involve costly operations such as modulo or division and 20 iterations of it run at 135 wallclocks seconds - over two times faster than zeroth's Haskell version.

Now how about Perl? Since Perl has assignment, we have the advantage that we can create a vector of bits that we will mark with the primes by iterating over all numbers up to the root of the limit. Here is the code:

#!/usr/bin/perl

use strict;
use warnings;

use Math::BigInt lib => 'GMP';

my $limit = 2_000_000;

my $primes_bitmask = "";

my $loop_to = int(sqrt($limit));
my $sum = 0;
my $total_sum = Math::BigInt->new('0');

for my $p (2 .. $loop_to)
{
    if (vec($primes_bitmask, $p, 1) == 0)
    {
        $sum += $p;

        my $i = $p * $p;

        while ($i < $limit)
        {
            vec($primes_bitmask, $i, 1) = 1;
        }
        continue
        {
            $i += $p;
        }

    }
}

for my $p ($loop_to .. $limit)
{
    if (vec($primes_bitmask, $p, 1) == 0)
    {
        if (($sum += $p) > (1 << 30))
        {
            $total_sum += $sum;
            $sum = 0;
        }
    }
}

$total_sum += $sum;
print "$total_sum\n";

20 runs of it run at 95 walclock seconds - even faster than the Haskell version. But it gets better. Since all the primes we encounter greater than 2 are not even, we can create a map of their pseduo-halves and conserve on memory and iterations. This is the Perl version:

#!/usr/bin/perl

use strict;
use warnings;

use Math::BigInt lib => 'GMP';

my $limit = 2_000_000;

my $primes_bitmask = "";

my $loop_to = (int(sqrt($limit)))>>1;
my $half_limit = ($limit-1)>>1;

my $sum = 0+2;
my $total_sum = Math::BigInt->new('0');

for my $half (1 .. $loop_to)
{
    if (vec($primes_bitmask, $half, 1) == 0)
    {
        my $p = (($half<<1)+1);
        $sum += $p;

        my $i = ($p * $p)>>1;

        while ($i < $limit)
        {
            vec($primes_bitmask, $i, 1) = 1;
        }
        continue
        {
            $i += $p;
        }

    }
}


for my $half ($loop_to .. $half_limit)
{
    if (vec($primes_bitmask, $half, 1) == 0)
    {
        if (($sum += (($half<<1)+1)) > (1 << 30))
        {
            $total_sum += $sum;
            $sum = 0;
        }
    }
}

$total_sum += $sum;
print "$total_sum\n";

Running this 20 times takes 73 wallclock seconds, close to half that of my Haskell version.

Then I wondered how long C will take. Here is a C implementation without the halving:

#include <string.h>
#include <math.h>
#include <stdint.h>
#include <stdio.h>

#define limit 2000000
int8_t bitmask[(limit+1)/8];

int main(int argc, char * argv[])
{
    int p, i;
    int mark_limit;
    long long sum = 0;

    memset(bitmask, '\0', sizeof(bitmask));
    mark_limit = (int)sqrt(limit);

    for (p=2 ; p <= mark_limit ; p++)
    {
        if (! ( bitmask[p>>3]&(1 << (p&(8-1))) ) )
        {
            /* It is a prime. */
            sum += p;
            for (i=p*p;i<=limit;i+=p)
            {
                bitmask[i>>3] |= (1 << (i&(8-1)));
            }
        }
    }
    for (; p <= limit; p++)
    {
        if (! ( bitmask[p>>3]&(1 << (p&(8-1))) ) )
        {
            sum += p;
        }
    }

    printf("%lli\n", sum);

    return 0;
}

This was too fast to measure with 20 runs alone, so 500 runs of it took 15 seconds, two or three orders of magnitude faster than the fastest Haskell or Perl versions. But naturally, we can apply the halving paradigm there too:

#include <string.h>
#include <math.h>
#include <stdint.h>
#include <stdio.h>

#define limit 2000000
int8_t bitmask[(limit+1)/8/2];

int main(int argc, char * argv[])
{
    int half, p, i;
    int half_limit;
    int loop_to;
    long long sum = 0 + 2;

    memset(bitmask, '\0', sizeof(bitmask));

    loop_to=(((int)(sqrt(limit)))>>1);
    half_limit = (limit-1)>>1;

    for (half=1 ; half <= loop_to ; half++)
    {
        if (! ( bitmask[half>>3]&(1 << (half&(8-1))) ) )
        {
            /* It is a prime. */
            p = (half << 1)+1;
            sum += p;
            for (i = ((p*p)>>1) ; i < half_limit ; i+=p )
            {
                bitmask[i>>3] |= (1 << (i&(8-1)));
            }
        }
    }

    for( ; half < half_limit ; half++)
    {
        if (! ( bitmask[half>>3]&(1 << (half&(8-1))) ) )
        {
            sum += (half<<1)+1;
        }
    }

    printf("%lli\n", sum);

    return 0;
}

500 runs of it take 10 wallclock seconds - 54.35 times per second, and 50% better than the previous C version. And I still haven't applied platform-specific gcc optimisations.

I should also note that the executables generated by ghc are extremely large in comparison to their C ones:

$ ls -l c_* haskell_*
-rwxr-xr-x 1 shlomi shlomi   6082 2009-12-04 07:46 c_mine
-rwxr-xr-x 1 shlomi shlomi   6103 2009-12-04 07:46 c_mine_half
-rwxr-xr-x 1 shlomi shlomi   6092 2009-12-04 07:46 c_mine_micro_opt
-rwxr-xr-x 1 shlomi shlomi 796825 2009-12-04 07:56 haskell_mine
-rwxr-xr-x 1 shlomi shlomi 571717 2009-12-04 07:46 haskell_zeroth

c_mine_half is less than 1% the size of haskell_mine (and runs faster). When talking about this to other people, they said that Haskell has a very optimised primes sequence generator, which I can try using (which should be over 3 times as fast), and that it has two kinds of integers, which the other type is faster, and that it has a better way to emulate assignment. But the bottom line is that the naïve and intuitive way to write such programs in Haskell is under-performant, even in comparison to Perl, and 100 or 1,000 times as much in comparison to C.

I've written this as a separate post and not as a comment to the original blog post because I'm very limited with the markup in the commenting there (I'm going to post a comment there with a link to this blog post, though). I should note that you can find all the code I mentioned inside a dedicated Github repository, and you can experiment with it further.

2010-01-31: Escape from GNU Autohell!

I've set up a page on my web-site titled "Escape from GNU Autohell!" about the evils of the GNU Autotools (GNU Autoconf/Automake/Libtool and other GNU auto-brain-damages) and why you should convert to a good alternative such as CMake. The page converts a list of the many disadvantages of the GNU Autotools, some refutation of common arguments against CMake, and some links.

And lo and behold, some time after writing the page I spent hours dealing with yet another Autohell problem: "Mandriva's libtool causes "make install" of xine-lib-1.1-hg to fail". I actually had to build the Mandriva libtool several times to find the offending downstream patch, and libtool's "./bootstrap" script was so slow that I could finish running "make" and "make install" in xine-lib while waiting for it to finish. And it says a lot because xine uses GNU Autohell which slows down the build process considerably. Eventually, I found the offending patch after several hours, and I wouldn't have finished on time if I didn't cancel the "make check" test in the .rpm.

Autohell must die!

2010-02-02: פגישת קוד פתוח בתל-אביב: 14 בפברואר - vtiger CRM

מועדון הקוד הפתוח התל-אביבי (תלוקס) ייפגש שוב כדי לשמוע את הרצאתו של רמי הדדי אודות "מערכת ניהול קשרי הלקוחות vtigerCRM. ההרצאה תתקיים ביום ראשון, 14 בפברואר 2010, בשעה 18:00, באולם הולצבלט, מס' 007 במסדרון הבניינים למדעים מדויקים (שימו לב לשינוי בשעה ובמיקום משנה שעברה) באוניברסיטת תל אביב. פרטים נוספים, מפות להגעה וכיוצא בזה, ניתן למצוא באתר ובוויקי. הנוכחות בהרצאה היא חינמית ולא נדרשת הרשמה מראש.

vtiger CRM היא מערכת ניהול קשרי לקוחות אשר פועלת בממשק Web דרך הדפדפן. המערכת מכילה את כל הכלים הנדרשים לניהול מלא ומקיף של עסק: החל מעסק קטן של אדם אחד וכלה בעשרות עובדים. במערכת ניתן למצוא ביטוי ללקוחות, יומנים, הזדמנויות עסקיות, מכירות, שירות ועוד. המערכת מבוססת על קוד-פתוח וניתן גם לבצע בה התאמות בקלות רבה. המערכת תורגמה לשפות רבות ובכללן עברית.

רמי הדדי הינו מנכ"ל אקטיבטק אשר מתמחה ביעוץ, פיתוח ואירוח של vtiger CRM.

אנו תמיד מחפשים מרצים שיתנדבו לתת הרצאות בנושאים שונים הקשורים לקוד-פתוח ולמחשבים. במידה שאתם מעוניינים לתת הרצאה, או שיש לכם הצעה להרצאה שמעניינת אתכם, נשמח לשמוע ממכם.

2010-02-02: New Page about Text Editors and IDEs for Programmers

After answering the questions "Can anyone recommend a good text editor?" and "What is a good IDE?" a few times in the past, I set up a page listing some prominent text editors and Integrated Development Environments (IDEs) for development on my homepage. Currently it includes only cross-platform open-source text editors and cross-platform open-source IDEs, but I'm planning to expand it as time and interest permits. Additions would be welcome.

Vim is there and so are XEmacs and Eclipse, and also Padre, the Perl IDE and more specialised IDEs such as the "Eric" Python IDE.

2010-02-05: NYTProf-3 is Out!

Tim Bunce writes on his blog about the new features in Devel-NYTProf version 3. Devel-NYTProf is a profiler for the Perl programming language, which has put all the previous attempts in profiling in the dust, and now it's even better than before. Enjoy! (Thanks to Fred Moyer's post on the San-Fransisco Perl Mongers mailing list).

2010-02-11: Article Recommendation: "What Every CSer Should Know about Floating-Point Arithmetic"

I've read the paper "What Every Computer Scientist Should Know About Floating-Point Arithmetic" and can recommend it as it has many interesting insights. I had received quite a lot of reports for bugs in the "Statistics-Descriptive" CPAN distribution that were caused by expected floating-point behaviour. Some of them I was able to resolve using some trickery, and some others possibly can be somewhat mitigated using the insights I've learnt in the article.

2010-02-19: Solving gaal's Circuit Riddle Using Perl

gaal (= Gaal Yahas) mentions a riddle on his blog:

There’s a box with three input signals: A, B, C. It has three output signals: ~A, ~B, ~C (that is, the negation of each input).

Design a circuit satisfying the above description. You have two NOT gates, and as many AND and OR gates as you like.

(If you solve this, don’t tell me the answer — I haven’t solved it yet; I just want to share the misery.)

I decided that I will try to solve this riddle on my own and then attempt to write a solution using a Perl 5 program, which I've also thought about how to write. As it turned out, I ended up exploring and eventually completely solving the riddle using a series of Perl programs. I won't mention the final solution here, but I'll tell which programs I wrote to solve them and how I wrote them. The code can be found in its Mercurial repository under the MIT/X11 licence, as I mentioned in a comment to gaal's post.

Well, first of all I decided to write a program that will take two negated expressions of AND and OR gates and see if one can arrive to the final solution using them and the given inputs (A, B and C). To do this I represented every such possible expression as its components of (A + ~A) × (B + ~B) × (C + ~C) - 8 in total which fit in an 8-bit datum, and which I indexed using an array of 256 items to see which ones exist in the population. After doing the coding I realised none of the expressions I tried worked.

Then I decided to see if I there are any two possible expressions that when added to A, B an C will yield all the necessary outputs. I ran it and indeed found a single pair (and only a single pair) of such expressions (given as numbers) which I was able to analyse. I should note that the programs as I've written it using some functions and closures was too slow, so I ended up writing another Perl script to preprocess the code, and inline the functions, which has made everything much faster. (You can find this preprocessor in the repository).

Then I wanted to see if I can indeed construct these expressions from A , B, and C using two negations. When I had thought about this, I believed I needed to maintain a collection of expressions' populations, and to keep how many negations were in each. But when I got to code it I thought of a simpler way: marry all the inputs using AND and OR until no more can be married, then go over the existing ones and negate each one in turn, then add the negated vector to the population and marry it with all the rest, and then do it again for all the existing inputs. Using this I was able to find a single solution.

Then I had to output the solution. For that, I wrote a recursive expression dumper, which ended up outputting with many parentheses, so I had to post- process it a bit.

The solution I got to was very complicated and I believe it would have taken me a long time to reach it without resorting to programming. But at least it saved me a lot of time and head-scratching.

2010-02-22: New Perl Quiz-of-the-Whatever: Short Freecell Solutions

I published a new Perl Quiz-of-the-Whatever about finding short Freecell solutions. Hope you enjoy thinking about it. Here is the Perl Quiz-of-the-Whatever (formerly "Quiz of the Week") information..

2010-02-25: Review: Alternative (to Firefox) Linux Web Browsers

Since I've been blogging about Perl a lot lately (trying to maintain and advance my Planet Perl Iron Man status), I've been neglecting to blog about other things. So here's a new entry not about Perl, not even about programming - but about web-browsers.

I've been using Firefox (now at version 3.6) as my primary browser on my Mandriva Linux Cooker system for a while now, and have been mostly happy with it. Today I wanted to edit this wikibook titled "How to Write a Program". It had been written awfully (see the discussion page), but I was introduced to it after it was made a prerquisite of a different wikibook which I've written, and decided that the dependency should be kept (assuming "How to Write a Program" would be written better). Now, I could either throw away "How to Write a Program" and start from scratch, or revamp it to submission. Right now, it seems like it would be a combination of both.

In any case, after editing it, I've ran into a long-term bug in Mandriva, or in my Mandriva setup (which is exhibited in any Firefox that is running directly on it.), which I've finally reported there (though I believe I had reported it in the Firefox bugzilla too) and so had been unable to use Firefox for it.

So I've been looking for an alternative browser. I ruled out Konqueror because KHTML has been suffering from a lot of bug report, and Konqueror is just plain annoying. (WebKit has been forked from KHTML, and it proved to be more popular.). I initially ruled out Opera because it's not open-source. So that left me only with WebKit-based browsers. I asked for recommendations and people on #ubuntu-uk (great channel, BTW) recommended Chromium and GNOME's Epiphany, and some other more obscure browsers. I've decided to check them out.

I decided to go with SRWare Iron as my Chromium browser. The first thing I noticed was that the Linux download link led to a web forum post - very unprofessional. After downloading and unpacking the archive, it refused to run - /home/shlomi/apps/iron-linux/iron: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory. A symbolic link to libbz2.so.1.0.0 (done by root) fixed that problem, and then it crashed with an X error. However, it started the second time and seemed to run fine (don't know what the problem was). I noticed Chromium was indeed very fast, but then I ran into a few glitches:

So I decided to look at Epiphany. Slower than Chromium, but maybe it will work. Or maybe it won't. The first thing I noticed was that it looked ugly - very unaesthetic and several widgets in the main window were slightly out-of-place. Its URL bar also behaved unlike Firefox in the double click respect, which was also annoying.

So what can I do? I eventually decided to use Opera for editing Wikimedia wikis until the Firefox-influencing bug is resolved. It is fast, it has a menu bar, and double click in the URL bar selects the entire URL. Very nice. It's not open-source, so I'm trying not to get myself used to it, but seems like I'll need to use it.

If you haven't done so already, I suggest you read the "Joel on Software" "must-read" books recommendations page. I haven't read all the books there, but the page itself is also good. Quoting from it from our context:

A few months ago when we released CityDesk, I got an email from a customer complaining that he was used to doing Alt+F, Alt+S to save files. Unfortunately due to a tiny, unnoticed bug, that keyboard shortcut saved the file and then closed it, irritatingly. I had never noticed because I'm in the habit of doing Alt+F,S to save files, not Alt+F,Alt+S -- a tiny difference -- and Alt+F,S worked fine.

Once you get into the habit of doing Alt+F,Alt+S to save, it becomes so automatic you don't think of it as Alt+F,Alt+S. You think of it as save. And when you push the "save" button in your brain and the file you were working on goes away, it makes you feel like you're not in control of your environment. It's a small thing, but about the fourth time that it happens, you're going to be seriously unhappy. That's why I spent several hours tracking down this bug and fixing it. In a bizarre application of Murphy's Law, this fix led to a cascade of events that caused us to waste something like a week, but that's neither here nor there. It was worth the time spent. This is what it means to be concerned about usability. If you still think that something as small as how long you hold down the Alt key when you activate a menu command doesn't matter, well, your software is going to make people unhappy. These tiny inconsistencies are what makes Swing applications so unbearably annoying to use, and in my opinion it's why there are virtually no commercially successful Java GUI applications.

If I can't go to "File → New Tab" in the menubar to start a new tab then I'm going to be unhappy. If I have to triple click the URL bar in order to select the entire URL, then I'm going to be unhappy. These tiny things are why I can't use either Chromium or Epiphany. (And frankly, a lack of a menubar is a huge UI screw-up, not a tiny one).

At the moment, I'm just ranting, but I am planning on reporting these bugs to the bug trackers of the appropriate projects. I just hope Google won't think the lack of menu-bar in Chromium is a feature. "Welcome to Google UIs". (Welcome to Google Hell…).

2010-03-01: A Perl-based →with(...) method

Lately, I've been writing some PDL (Perl Data Language) code and noticed that I've been doing the following:

my $temp_pdl = $pdl->…->final_expr();
my $result_pdl = $temp_pdl->where($temp_pdl < 0);

I've been thinking if I could somehow write it like that:

my $result_pdl = $pdl->…->final_expr()->with(sub { $_->where($_ > 0)});

Which seems cleaner and more elegant. I went on IRC asking about it and someone suggested rindolf: (sub { local $_ = shift; $_ * $_ * $_})-gt;( $obj->... ); (up to an episolon, but naturally this reverses the order and messes with the flow of the code). Eventually Matt S. Trout (mst) proposed the following solutions:

sub UNIVERSAL::with
{
    local $_ = shift;
    my $sub = shift;
    return $sub->($_,@_);
}

my $_with = sub {
    local $_ = shift;
    my $sub = shift;
    return $sub->($_,@_);
};

(I had figured something like that was possible previously, but I was beating around the bush.)

The $_with approach is covered in one of Matt's blog posts titled "Madness with Methods".

I wondered if it will work with autobox and mst said that I don't know and I hope not since the UNIVERSAL approach is evil.. But it does, out-of-the-box:

#!/usr/bin/perl

use strict;
use warnings;

use autobox;

sub UNIVERSAL::with
{
    local $_ = shift;
    my $sub = shift;
    return $sub->($_,@_);
}

my $_with = sub {
    local $_ = shift;
    my $sub = shift;
    return $sub->($_,@_);
};

print +(10+1)->with(sub { $_ * $_; }), "\n";
print +(10+1)->$_with(sub { $_ * $_; }), "\n";

This prints 121 twice.

I now wonder what to do with this knowledge. I have some aspirations of releasing it to the cpan as a mini-module. Still, it's really cool.

Finally, I should note that it seems like I'm going to lose my current Planet Perl Iron Man karma due to inadequate blogging. Oh well.

2010-03-16: New Topical Pages and Pages for Perl Uses on Perl-Begin

This summarises the changes to the Perl Beginners' Site since the last update.

We added a a page about using Perl for E-mail processing, a page about Perl for multitasking and networking, a some topical pages:

The IDEs and Tools page now contains screenshots.

Many new links have been added and many typos have been corrected. Enjoy!

Perl-Begin continues to be the most comprehensive first-stop for Perl Beginners. Moreover , its content is distributed under the Creative Commons Attribution License (CC-by), and its source code is available to facilitate contributions.

2010-03-21: Tech Tip: Getting AAC to Play in gstreamer

If you're getting a message like the following when playing a video or an audio file with AAC sound in a gstreamer-based and no sound is played:

** Message: Missing plugin: gstreamer|0.10|totem|MPEG-4 AAC decoder|decoder-audio/mpeg, mpegversion=(int)4, framed=(boolean)true (MPEG-4 AAC decoder)

Then you need to install the 'faad' gstreamer element from gst-plugins-bad or as a separate pacakge. Thanks to "__tim" from Freenode for the insight.

2010-03-22: Report on the recent Haifa Perl Mongers Meeting

This is a report and some notes from the Haifa Perl Mongers meeting a while ago.

I arrived there by train from Tel Aviv with Sawyer, who was going to give the presentation about Moose. We met in the Arolozerov train station, and I was a little late since I noticed I had dressed too lightly for the cool weather. We bought tickets and went on the train to Haifa.

On the way we talked about Perl, C, C#, bugs in CPAN modules, Chuck Norris facts, and other stuff. We arrived at Haifa well ahead of time, and I led the way to the meeting at Qualcomm. As it turned out, the road I took was relatively sub-optimal because there was a way with more pavements going there. But we got there.

We entered Qualcomm, and saw Shmuel there and we were given tags for entrance (in exchange for our I.D. cards). We wanted to drink something and the cooler was broken, so we ended up using the tap. We waited for everyone else to arrive and in the meanwhile I munched on a croissant and a Rogalach.

Then the presentations started. Erez gave a good presentation about how to talk to newbies, where he said that a lot of the things we say are irrelevant. Asking people to add "use strict" will cause their scripts to stop running completely whereas they previously only had a bug, so it's not always the best strategy. The old "RTFM" mantra or "You can't just make shit up and expect the computer to know what you mean, retardo!" were also mentioned.

He then gave a few links where people can visit to help newcomers. He mentioned learn.perl.org but I was disappointed that he didn't mention Perl-Begin - the Perl Beginners Site.


In any case, then Sawyer gave his presentation introducing Moose. It was a good presentation and the audience spent more time discussing it than last time he has given it in Tel Aviv. We spent a lot of time discussing the clearing methods which turned out to be equivalent to delete($self->{'myfield'}) or so. Then we discussed the compilation-time overhead of Moose where someone said it was unacceptable because it would mean his test suite composed of many scripts would run much slower. We then discussed about whether most people use all of the features in Moose, and whether it indicates code bloat and is worth the extra overhead.


Then the meeting ended and we walked to the train station and returned home. Erez took the train with us and on the way home we discussed Java, Matt S. Trout and Planet Iron Man, Lambda Calculus, the earlier discussions about Moose and other stuff. I tried to explain to them about Lambda Calculus and not sure I was successful, but you can read what I had written about it in my "Scheme and Lambda Calculus Lecture" and what Mark Jason Dominus wrote about it in Perl and Lambda Calculus.


So it was a very nice meeting and I greatly enjoyed it. Also see what Sawyer wrote about it, and what Erez wrote about the meeting.

2010-03-27: My Mojolicious and KiokuDB Story

I was hired by Insurgent Software (one of whose directors is an old IRC/IM friend of mine) to write a Mojolicious and KiokuDB-based user authentication system. I was naturally getting paid for it and the resultant code was going to be open-source so it seems nice. I did not have any previous experience with either Mojolicious (a web-development framework) or KiokuDB (an object-based persistence system), both of which are recent Perl 5-based technologies, so I believed had quite a lot to learn. Which indeed was the case.

The first thing I noticed when starting to work on the Mojolicious project was that it lacked a good testing module, as Test::Mojo hardly provided anything I was used to from Test-WWW-Mechanize-Catalyst. So after some thought, I decided to create Test-WWW-Mechanize-Mojo based on it. This took a day of work to hack the first version, and required some subsequent updates due to misunderstanding the dependencies' APIs, or doing other things. Afterwards, I also added some XPath-related methods in the test program and then implemented some application-specific logic using Moose in a further sub-class (this time Moose-based).

This eventually turned out to work pretty well for some needs, though Test-WWW-Mechanize-Mojo still emits some warnings, and I needed to fix some bugs in it.

Our Mojolicous::Lite code was very succinct, which I could appreciate, but due to its design philosophy, Mojo/Mojolicious ended up re-implementing a lot of CPAN in slightly different ways, which required learning and studying.

In regards to KiokuDB - it worked pretty well. When trying to deploy it on my Mandriva 2010.0 laptop, I discovered that it gave me a lot of problems with the Perl debugger (perl -d), which were resolved by upgrading namespace-clean (thanks to nothingmuch on IRC for that.)

I misunderstood the KiokuDB API a little, which resulted in some things I needed to fix. I also eventually used the Moose "around" method-modifier to abstract the new_scope() functionality (which someone told me he did something similar with "before" which I'm not sure will work because it will go out of scope, before the function call).

I've also written yet another form handler/generator because I was too lazy to look at one of those on CPAN, and because mine kinda evolved.

The end of the story was that despite everything, the project was completed successfully to the satisfaction of Insurgent Software, and that I was mailed the cheque with the second half of the payment. However, this project tended to have irritated me, and also consumed me and left me able to do very little other FOSS-related activities. Still, I'm happy that I took it, and now that it's over, I resumed my open-source activities and among other things have released a new version of Freecell Solver.

2010-04-01: Announcing Freecell Solver™ Enterprise Edition

There's an exciting development today for one of my programs. Reading from the publicity:

Freecell Solver Enterprises™, Inc., on behalf of the Freecell Solver™ development team, is glad to announce the upcoming availability of Freecell Solver™ Enterprise Edition. In its Enterprise Edition, Freecell Solver™ will be enhanced to solve generalised Freecell, in which there can be an arbitrary number of card ranks. Since generalised Freecell is NP-complete. This will allow using Freecell Solver™'s ingenious, cutting-edge algorithms to solve the previously hard, provably NP-Complete problems of the Travelling Salesman problem, Subset sum, Sudoku and Enterprise content management.

Read more on the announcement to learn more about this and upcoming exciting developments.

2010-04-03: Why I Wanted to Quit Planet Perl Iron Man and How to Improve it

Note: I discovered that the Planet Iron Man/Woman people are already planning some changes to the Planet, and will deal with my complaints here as part of this. So I'm not going to quit blogging on the Planet eventually, at least not yet. I'm still posting it as a general insight for people who wish to set up successful online communities.


I decided to quit Planet Perl Iron Man and I'd like to tell you why and how I think it should be improved. The reason I quit is because of the fact that if one doesn't blog frequently enough, their status gets demoted (possibly back to Paper Man always). Now, while if I blog frequently enough I will increase my status which is motivating, demoting or even resetting the karma is demotivating. This is the Carrot and stick method.

Now, I don't want to feel obliged to blog in order to prevent my karma from being gone. I used to blog about Perl and other topics because it was fun, not because I felt I had to. As a result, the "stick" in this place is out-of-place. I request that assuming the current system remains, my status be either kept as "unknown" or even that my blog be currently taken out of the Iron Man/Iron Woman aggregator. Everyone should feel free to aggregate my Perl feed ( or any of my other feeds as you see fit), without asking me, but I don't wish to play the carrot and stick status game.

So how to improve it? Simple - avoid the stick. Make the Planet Iron Man be a game of infinite karma, similar to Stack Overflow or Perl Monks. On Stack Overflow for example, your karma never decreases. If 2 people upvoted your comment or post and 6 people downvoted it (for a total score of -4), you'll still get 2 points. Furthermore, your karma can grow indefinitely, and you also get other motivations like badges etc. Perl Monks has a similar system with its Experience Points. (I should note that I'm not fanatically trying to improve my karma on either site, and my current status on both is very low, because I'm not into being competitive in them, though I found both useful at times when I was seeking help.)

Perhaps we can make a similar program for a competitive planet, where people get points for every posts, every link from someone else's post on the planet, every upvote on Reddit, Stumble Upon and other social bookmarking sites, get extra points for making, say, 3 posts in 7 days, earn badges etc. without ever needing to fear their karma being reduced.

I should note that I've noticed other social sites being undermined (at least temporarily) by such "sticks". For example Advogato used to allow people to rate other people's blogs' quality from 0 to 10. Naturally, some people gave bad marks for some blogs which was demotivating. Luckily, it was removed during an update. On use.perl.org you could specify people you dislike as "foes" in your user page, which ended up showing in them as "freaks" (= haters). Again, that's demotivating.

There's probably still a lot to be learnt about designing successful web communities, but I think that if you want people to do things because they enjoy it (and/or you don't intend to pay them), you should not demotivate them by threatening the use of "sticks".

2010-04-10: TelFOSS Meeting on Wednesday 21-April-2010 - "Version Control Sytems"

The Tel Aviv Open Source Club will host the talk "Version Control Systems - What, Why and How" by Yaron Meiry (Sawyer) on Wednesday, 21-April-2010, at 19:00 in Shenkar College in Ramat Gan (12 Ann Frank St.) room 300 (note the change of time and place from the previous meetings - we are now meeting in Shenkar college and on 19:00.). There's a map with directions on the Shenkar college's site. Further details, maps for arrival, etc. can be found on the Wiki.

Attendance is free, it is not necessary to RSVP and everyone are welcome.

With any other problems, feel free to contact the organiser.

Abstract

Version control systems allow us to keep track of an unlimited amount of changes across any number of files. They are an essential tool for any developer.

Using version control systems can help you restore lost data, examine the differences between between multiple versions of your data, tag versions and access them quickly, simplify cooperation with your peers, easily merge changes, and maintain several branches of your data at once.

This presentation will explain what version control systems are, why you should use them (even if you think you shouldn't) and how you should use them. Most examples will be given using the open-source version control system Git.

About the Lecturer

Yaron Meiry is a systems' administrator and a Perl developer. He gives talks about open source, free software, security and programming standards. Yaron has previously given the presentation about red flags in programming of very high languages in the Tel Aviv Club and a presentation about Moose, the modern Perl 5 Object System to TelFOSS and several other clubs.


We are always looking for people who will volunteer to give presentations on various topics that are related to open source code and to computers. In case you are interested to give a talk, or that you have a suggestion for a talk that interests you, we'll be happy to hear from you.

2010-04-15: פגישת תלוקס: "מערכות ניהול גרסאות" ב-21 באפריל

מועדון הקוד הפתוח התל-אביבי (תלוקס) ייפגש שוב כדי לשמוע את ההרצאה "מערכות ניהול גרסאות - מה, למה, וכיצד" מאת ירון מאירי (Sawyer) ביום רביעי, 21 באפריל 2010 ב-19:00 במכללת שנקר ברמת גן (רח' אנה פרנק 12) חדר 300 (שימו לב לשינוי במיקום ובזמן מהפגישות הקודמות - אנו ניפגש במכללת שנקר ובשעה 19:00) ניתן למצוא מפה עם הוראות הגעה באתר של מכללת שנקר. פרטים נוספים, מפות להגעה, וכו ניתן למצוא בוויקי.

הנוכחות בהרצאה היא בחינם ולא נדרשת הרשמה מראש.

במידה שישנן בעיות, אל תהססו ליצור קשר עם המארגן.

סיכום ההרצאה

מערכות ניהול גרסאות מאפשרות לנו לעקוב אחרי כמות לא מוגבלת של שינויים במספר שרירותי של קבצים. הן מהוות כלי הכרחי עבור כל מפתח.

השימוש במערכות ניהול גרסאות יכול לעזור לך לשחזר מידע אבוד, לבחון את ההבדלים בין מספר גרסאות של המידע, לתייג גרסאות ולאפשר גישה אליהן בקלות, להקל על שיתוף פעולה עם שותפיך לפרוייקטים, לאחות שינויים בקלות, ולתחזק מספר ענפים של המידע שלך בו זמנית.

מצגת זאת תסביר מה הן מערכות ניהול גרסאות, מדוע כדאי שתשתמשו בהן (גם אם כרגע אינכם חושבים כך) וכיצד כדאי שתעשו בהן שימוש. מרבית הדוגמאות יועברו באמצעות מערכת ניהול הגרסאות פתוחת-הקוד "גיט".

אודות המרצה

ירון מאירי הינו מנהל מערכות ומפתח פרל. הוא מרצה על קוד פתוח, תוכנה חופשית, אבטחה וסטנדרטים של תכנות. ירון העביר בעבר את ההרצאה על דגלים אדומים בתכנות עבור שפות עיליות ביותר במועדון התל-אביבי ומצגת אודות Moose, מערכת העצמים המודרנית לפרל 5 במועדון התל אביבי ובמועדונים אחרים נוספים.


אנו תמיד מחפשים מרצים שיתנדבו לתת הרצאות בנושאים שונים הקשורים לקוד-הפתוח ולמחשבים. במידה שאתם מעוניינים לתת הרצאה, או שיש לכם הצעה להרצאה שמעניינת אתכם, נשמח לשמוע ממכם.

2010-04-26: TelFOSS Meeting Report: Sawyer about Version Control Systems

Here are some notes from the Tel Aviv Open Source Club's meeting last week, in no particular order. They are not exactly a report, but are probably better than nothing. I originally posted them to the club's mailing list and forwarded them to to the Perl-IL mailing list, where it sparked a small (but funny) discussion. I hope to expand upon them in this blog post.

  1. About 20-30 people came in my approximation, some of them a little late, so it was pretty successful. So hopefully it was a good decision to find a venue which can afford starting at a later hour. We'll see if the trend of successful presentations can continue.

    I think I recognised most of the attendees, but there were some new faces, which is a good thing.

  2. Sawyer gave a very good presentation in my opinion, and I've learnt some new stuff about git there. There was some good involvement from the audience. Next time, I'll try to pass the baton to a different lecturer, because I think Sawyer has been giving too many presentations lately, and we need to see some fresh blood.

    At the moment we only have one proposed presentation for June (by cool-RR / Ram Rachum ) and he still noted he may need to cancel. I've been considering to prepare a presentation/demo about jQuery and also proposed to give a "Bottom Up Subversion" presentation/tutorial. I'm not a good speaker so I'd prefer if someone can give it instead. We can also have a session of short, volunteered presentations on the spot (lightning talks or somewhat longer.).

    Nevertheless, it would be preferable if someone will volunteer to give a presentation.

  3. Some logistics about the room - there is a keyboard and a projector, but no screen. One can connect a laptop. The computer runs Windows , and we're not sure if it has OpenOffice.org. There's an Ethernet connection and also some Wifi networks.
  4. Sawyer mentioned the fact that from his experience creating a branch in Subversion (using svn copy) was time-consuming for him ("I went to prepare coffee"). This sounds strange to me, because branching and tagging in Subversion are reportedly cheap, constant-time ( O(1) ) operations, and I've verified it from what I've read about the Subversion internals. I often branched the Freecell Solver's trunk which is 26M in size and it didn't take more than a few seconds. svn.kde.org has many branches and each of them keeps the entire KDE source code and is positively huge with 100s of megabytes of files.
  5. I mentioned the mini-repository Subversion pattern - each with its own trunk, tags and branches which can be seen in action in the Web-CPAN repository. This is a single subversion repository with several mini repositories - one for each project. It can be done in git too by keeping each project in its own sub-directory, out of the top-level sub-directory.

    I should note that I also eventually decided against putting the files of the program directly on the trunk, but rather always put them under a sub-directory of the trunk, so if I need to add something else, they won't clobber the trunk.

  6. There was a short discussion about whether version control systems are using databases as their repositories' back-ends. Someone said he wouldn't trust a VCS that uses an off-the-shelf database, and I'd like to comment about that. The first thing that one should note is that like "Practical mod_perl" notes many data storages are in fact databases without us really thinking about them. Even the file-system is a database that maps filenames to the file contents. Furthermore, Subversion requires a transactional database for its use: it started from Berkeley DB (which is a database with some accepted and deliberate philosophical limitations), and then got the ability to use a backend called FSFS, which is a specialised, ad-hoc, database coded specifically for use by Subversion; and the Subversion development team have a distant goal of also creating an SQL-based backend for the repository.

    That's not all: Monotone has been using SQLite since its inception as its local storage backend, and reportedly ClearCase is using a database called "Raima". And naturally, I expect that even git, Bazaar, Mercurial and similar version control systems implement a database storage of some sort internally.

  7. We discussed the Simple Simon solitaire, which I've introduced Ido (ik) to and to which he became addicted to lately. I've ranted a bit about how "Solitaire" (or what the British call "Patience games") is actually a generic name for single-player card games, and that what Windows calls "Solitaire" is actually just some variants of Klondike Solitaire.

    We compared and contrasted both PySolFC and KDE 4's KPatience.

    The one who introduced me to Simple Simon was Stephan Kulow of KDE fame, who when I told him that I'd like to adapt Freecell Solver to solve Klondike said that "I don't think the world is ready for that." and then added that he could really use a solver for Simple Simon. After playing a few games of Simple Simon, and experimenting a little with the fc-solve C code, I was able to create a satisfactory solver, but then I also started to like Simple Simon and played a lot. And it was contagious as my mother and my sisters also started to play it.

    Right now, the KPatience's Simple Simon demo ends up in many practically infinite loops of no apparent objective. I hope to work on converting it to use my solver instead (which does not exhibit this problem), but it will take some work.

  8. About 5 of us went to eat at Spaghetim (an Italian restaurant in Ramat-Gan) after the meeting. Besides me, there were two Perl programmers ( http://szabgab.com/ and Sawyer, who was the presenter) and two Python programmers and I was appointed as the referee. We were able to explain some stuff from both languages to the other party.
  9. Szabgab mentioned that when he asked some people he gave training to if they know Perl, they said "No, I can only read it.", which to him seemed to imply that maybe Perl's reputation as a write-only language was unfounded.
  10. When we discussed Ram's Garlicsim ( http://garlicsim.org/ ) he mentioned that he is mostly a web contractor, but that he would prefer to be hired for writing simulations. Then Sawyer mentioned that often an invention brings a necessity, which reminded us of the book "Guns, Germs and Steal". It is highly recommended. Then we discussed some other books by the same author and I ended up saying that I've also read the book "The Origins of Conciousness In the Breakdown of the Bicameral Mind" by Julian Jaynes, which I enjoyed and found of a similar interest to "Guns, Germs and Steel" because of its philosophy of taking a bird eye's view of human history.
  11. Also in the after-meeting, we discussed automated tests and test coverage, and someone said that it may not be indicative of how well all the edge cases were dealt with. For example we can test that the tests cover the expression "z = x / y", but it won't warn us that we dealt with the case that "y" is zero, which we should. Someone noted he likes the green indication that he has 100% test coverage. On Perlsphere Ovid recently wrote that a general piece of advice is that one should not strive for 100% test coverage.
  12. We saw a demo of GarlicSim (which is LGPLed), and its Wx-interface (which is at the moment not open source, but has the source available for download under restrictive terms). Ram showed us how he started a simulation where the simulator calculated a line, and then it was possible to branch it into a different line of execution and continue from there. He intends to continue working on GarlicSim into the near future and then hopefully be able to give some presentations about it to the local open-source clubs.

To sum up, it was a great meeting, and I hope this trend will continue.

2010-05-03: The Quest for a List of Open Source Software for Windows

As I was editing the wiki document I've started writing called "How to start contributing to or using Open Source Software" on the Tecaching Open Source wiki, I've reached the part where I instruct people to try and use open source applications for Microsoft Windows, and was looking for lists of FOSS for Windows. I remembered the "Open Source Windows" site, which is nice and aesthetic, but not very comprehensive, and noticed they got rid of the second page, which contained some other programs. So I had to find another list.

A google search yielded many lists that were either not maintained, disorganised, not open for easy editing, or all of them. Eventually, I decided to look for something in the wikipedia and got to the List of free and open source software packages, which I was familiar with, but was not Windows-specific. However, on its bottom I found a link to the List of Open Source Programs (LOOP) for Windows on one of the Ubuntu wikis, and this seemed like the best of breed and what I was looking for. I was already able to login to the wiki, add some programs I was familiar with, and do some comsetic fixes. I also added it to the document, favourited it on StumbleUpon, and linked to it from my homepage. Now I'm blogging about it here to publicise it.

2010-05-09: Recent Hacktivity Log

Here is another boring recent hacktivity log, feel free to skip it. It was motivated by Perlbuzz's question: "What are you working on in Perl?" and the comment I've posted there in reply, though it's not restricted to Perl alone. So thanks to Andy Lester from Perlbuzz for inspiring this post.

First of all, a note about my general status recently: I've been lacking physical energy, and feel like I often cannot exercise a lot, and would rather not travel too much outside my neighbourhood (including not to many FOSS events). I don't know the exact reason why. I am still motivated to work on the computer. I also have been working primarily on code and a small amount of humour stuff, and not too many essays, articles and blog posts, which may not be a bad thing, because essays and philosophical stuff may be considered "schema-driven" (as my psychologist had put it). Besides coding, I've been playing Fillomino and other games on Brain Bashers, playing some PySol (Freecell and Simple Simon primarily), dealing with bug and problems in my Mandriva system as I run into them, chatting on IRC and Instant Messaging, writing some E-mails, including some to answer the questions of random people on various mailing lists, and doing some job hunting.

I turned 33 (or 21h like the DOS interrupt) on 5-May, and also attended my cousin's birthday party on Friday, and had a lot of fun playing ball with a different cousin's son at the party, and also enjoyed the food there. But on to the hacktivity:

Recent Hacktivity - for real this time.

I've worked on XML-Grammar-Fiction/Screenplay a lot recently. I've decided to merge the two modules because XML-Grammar-Fiction started as a fork of XML-Grammar-Screenplay, which was then improved, and I've started forward porting its changes to X-G-Screenplay. Forward-porting the changes to X-G-Screenplay proved to be very not straightforward due to the fact that I converted the X-G-Screenplay parser from being procedurally recursive to being more stateful and incremental with O(1) function call stack.

Then I had to merge the two codebases into a few base classes and two specialised sub-classes. Then I merged different parts. While there were still some things I wanted to do in my ongoing clean-ups, I decided to upload version 0.1.0 to CPAN as it is, in order to get some CPAN testing, to "release early, release often" and because perfect is the enemy of good. I indeed got some CPAN testers failures indicative of bugs which I have since hopefully resolved.

A bit off-topic from the question of this post, but I took a break from working on Perl code per-ce and started working on incorporating a new feature, which I nicknamed "flares" to Freecell Solver. I was able to split the code into many commits, each one passing all the existing and possibly new tests (some of which are written in Perl 5). Back in the early Freecell Solver 2.8.x-or-below-days, when I was less careful and methodical, I would have just written it in one swoop and then spend many fun hours with gdb trying to isolate and correct many bugs. But now I know better than to do that. I still had to use gdb and other debug mechanisms, but this time I was able to detect the source of the bug and correct it relatively quickly. Having automated tests rule!

I've had to write some text analysis logic in ANSI C, while using only mostly standard libc/libm functions and really wished I could have done it in Perl. I wrote and refactored some testing code in Python/PyTAP, because I needed ctypes to interface with the binary shared library. In the course, I've written some pretty complex Test-Count (= a CPAN module to keep track of TAP tests' counts) code (and enjoyed the fact that I can source variable calculations from common modules, which I've added for a previous project (which I don't recall what it is now.)) I also made use of a radix tree building and compiling Perl 5 program that I wrote for managing Freecell Solver's command line arguments.

Finally, I prepared a short presentation about the Mojolicious web-development framework, which I've blogged about previously, and I end up explaining why I dislike it at the bottom.

Not exactly my hacktivity but I'm glad that chromatic's suggesting of porting Python's ctypes to Perl has materialised as a Google Summer of Code project. I hope to assist Ryan Jendoubi in doing this (without being intrusive.), because it would be very cool to have something like ctypes in Perl 5.

So that's it, I guess. Happy upcoming Shavouth everyone!

2010-05-20: Perl 5 Anti-abstraction: _with_curr_line

In the old XML-Grammar-Screenplay code, I created a small _with_curr_line method (unrelated to my previous post about →with()) that read like that:

sub _with_curr_line
{
    my ($self, $sub_ref) = @_;

    return $sub_ref->($self->curr_line_ref());
}

What it does is evaluate a closure and passing it the reference to the current line to "avoid" duplicate code. In one of my comments I also wrote "# Now Lisp got nothing on us.", which as it turned out was completely untrue. Furthermore, this abstraction, which was meant to get rid of explicitly assigning the "curr_line_ref()" to a variable in many places made my code much more messy. Some notable examples are:

    # Skip the [
    $self->_with_curr_line(
        sub {
            my $l = shift;

            $$l =~ m{\G\[}g;
        }
    );

Instead of:

# Skip the opening square bracket - '['
${$self->curr_line_ref()} =~ m{\G\[}g;

And:

my $which_tag;
# We need this to avoid appending the rest of the first line
$self->_with_curr_line(
    sub {
        my $l = shift;

        # Apparently, perl does not always returns true in this
        # case, so we need the defined($1) ? $1 : "" workaround.
        $$l =~ m{\G([^\&t;\[\]\&]*)}cgms;

        $curr_text .= (defined($1) ? $1 : "");

        if ($$l =~ m{\G\[})
        {
            $which_tag = "open_desc";
        }
        elsif ($$l =~ m{\G\&})
        {
            $which_tag = "entity";
        }
        elsif ($$l =~ m{\G(?:</|\])})
        {
            $which_tag = "close";
        }
        elsif ($$l =~ m{\G<})
        {
            $which_tag = "open_tag";
        }
    }
);

(As you can see, I assign to $which_tag inside the closure and have to declare it beforehand).

my ($sayer, $what);

($sayer) = $self->_with_curr_line(
    sub {
        my $l = shift;

        if ($$l !~ /\G([^:\n\+]+): /cgms)
        {
            Carp::confess("Cannot match addressing at line " . $self->line_num());
        }
        my $sayer = $1;

        if ($sayer =~ m{[\[\]]})
        {
            Carp::confess("Tried to put an inner-desc inside an addressing at line " . $self->line_num());
        }

        return ($sayer);
    }
);

Regex fun and declaring the two variable twice.

As a result, I decided to get rid of this anti-abstraction and do a simple "my $l = $self->curr_line_ref". Now what about Lisp? In Common Lisp, one can declare a macro to define methods that will already have the "l" for the current line reference variable pre-declared and pre-assigned. Or maybe one can also temporarily rewrite the method definition macros or add hooks for the duration of the class. So I guess this could have been much better there, and Perl 5 is not as good as Lisp is in this respect.

And I want to hear mst give a talk called Patches Welcome (might as well pool my votes to a good cause - I hope the talk will be recorded and there will be slides available.)

2010-05-31: Quirky Abstractions

Joel Spolsky had written an essay titled "The Law of Leaky Abstractions" where he discusses the fact that sometimes one needs to be aware of details of the abstraction's implementation to know how to properly make use of it. Today, I'd like to talk about an analogous phenomenon which I came to call a "quirky" abstraction, in which an abstraction over several technologies (probably two or more) fails to expose certain features in one of the technologies that it abstracts.

I first noticed this concept in a Stack Overflow question of how to encrypt/decrtypt an SQLite database from within Qt. Now, the Qt SQLite backend module links against the vanilla SQLite and not its encrypted implementations, so in order to make use of them one will have to recompile Qt and possibly expose some extra functionality in the abstraction.

At the moment, I've also volunteered to help maintain the Graph-Easy CPAN module and the reason was that one of the features of GraphViz was not exposed in its Graph-Easy backend, which brought the need to hack something as a workaround.

Can quirky abstractions be avoided? I don't know. As the abstraced technologies progress and gain more features, the abstraction needs to keep up with them, which makes a situation of a quirky abstraction more likely to occur.

I'm not sure quirky abstractions are as bad as leaky abstractions, but it's still an phenomenon that we need to be aware of.

2010-06-14: Rehovot Perl Mongers meeting on 15-June and Hamakor Assembly on 17-June

Here are some upcoming open-source-related meetings in Israel:

The next meeting of the Rehovot Perl Mongers will take place on 15th June (tomorrow) between 18:00-22:00 in the Weizmann Institute in Rehovot in Room 101 in the Levine Building. See the details and map.. The talk ("Introduction to Perl 6" by Gabor Szabo) starts at 18:30.

Content: Introduction to Perl 6

I've just written and gave this talk 2 days ago at the LinuxTag in Berlin.

After a short background and a few words on Perl 5 vs. Perl 6 I'll go over a number of examples that is part of the imminent Rakudo Star release.

People who bring a notebook and will have enough network bandwidth will be helped to install Rakudo on their machine. It is quite easy to install so you could do it ahead of the meeting by following the instructions on http://www.rakudo.org/

On Thursday (17-June-2010), Hamakor, the Israeli NPO for free and open source software will have a general assembly in Shenkar college, Frink building, room 306 in 18:30.

Arrival instructions are available on Shenkar's site

Thanks to Yoram Gnat for the help in organising a place.

2010-06-24: Declaration against ACTA

ACTA, the so-called "Anti-Counterfeiting Trade Agreement", which is a trade agreement that aims to criminalise a lot of the openness of the Internet under the guise of fighting counterfeited goods. The Free Software Foundation has started a declaration against ACTA, which everyone should sign.

Also please help spread the word.

2010-07-11: Tech Tip: Automatically Restoring the Wifi Connection on Linux

Having bought a new laptop and after installing Mandriva Linux on it, I noticed that the wifi (wireless networking) connection tends to get disconnected after a while. Eventually I wrote this script to automatically restore it. Feel free to re use it under the terms of the MIT X11 License. You may need to adjust some parameters. Run this script as root in the background (using tmux or GNU screen or whatever).

#!/bin/bash
while true ; do
	if ! ping -c 1 10.0.0.1 ; then
		/etc/init.d/network restart
	fi
	sleep 5;
done

Hope it helps.

2010-07-15: Tel Aviv Open Source Club Presentation: "GarlicSim" by Ram Rachum

מועדון הקוד הפתוח התל-אביבי (תלוקס) ייפגש שוב כדי לשמוע את הרצאתו של רם רחום אודות GarlicSim ביום חמישי, 22 ביולי, 2010 ב-19:00 במכללת שנקר ברמת גן (רח' אנה פרנק 12) חדר 300 (שימו לב לשינוי במיקום ובזמן מהפגישות הקודמות - אנו ניפגש ביום חמישי, במכללת שנקר, ובשעה 19:00). ניתן למצוא מפה עם הוראות הגעה באתר של מכללת שנקר. פרטים נוספים, מפות להגעה, וכו ניתן למצוא בוויקי.

הנוכחות בהרצאה היא בחינם ולא נדרשת הרשמה מראש.

במידה שישנן בעיות, אל תהססו ליצור קשר עם המארגן.

סיכום ההרצאה

GarlicSim הינה תשתית עבור הדמיות (סימולציות) ממוחשבות הכתובה בפייתון. היא מהווה פרוייקט קוד פתוח שאפתני בתחום המחשוב המדעי, ובפרט בזה של הדמיות ממוחשבות. בכוונתה להטוות מחדש את הדרך בה אנשים חושבים על הדמיות ממוחשבות, וליצור סטנדרט חדש לדרך שבה יוצרים ומשתמשים בהדמיות.

GarlicSim הינה פלטפורמה לכתיבת, הרצת, וניתוח סימולציות. היא מספיק כוללנית כדי לטפל בכל סוג של הדמיה: פיזיקה, תורת המשחקים, התפשטות מחלות, אלקטרוניקה, וכו. ייתכן שאתם תוהים מה משותף לכל ההדמיות האלה. אנו ננסה לענות על שאלה זו בהרצאה. אנו נראה דוגמאות של הדמיות אחדות ב-GarlicSim ושל כלים שהיא מספקת להרצתן וניתוחן.

אודות המרצה

רם רחום הוא מפתח פייתון. הוא עובד בעיקר על GarlicSim, אבל ידוע גם כמי שיצר את PythonTurtle, על מנת להפיץ את פייתון לאלפי ילדים ומבוגרים ברחבי העולם.


אנו תמיד מחפשים מרצים שיתנדבו לתת הרצאות בנושאים שונים הקשורים לקוד-הפתוח ולמחשבים. במידה שאתם מעוניינים לתת הרצאה, או שיש לכם הצעה להרצאה שמעניינת אתכם, נשמח לשמוע ממכם.

2010-07-30: Git Tip: git status

I've had to work with the git version control system extensively in the past few weeks (due to the proliferation of projects I'm involved in that use it.). I found the standard "git status" output to be far too wordy and annoying, and sought a way to improve it and make it more subversion like. "man git-status" helped with this: one can write git status -s to get an output similar to svn status (or svn st for short) and git status -s . to restrict the output only to files below the current directory (and not get the ../… files). Enjoy!

2010-07-31: Recent Hacktivity Log

Another boring recenty hacktivity log. First of all, in case you have not heard Rakudo Star, a usable implelmentation of Perl 6 has been released after a lot of work. Now on to the hacktivity.

File-Format-CRD

I released the first version of File-Format-CRD to CPAN. This is a CPAN distribution that processes Windows Cardile (.CRD) files (does anybody still remember those?) and allows to convert them to other formats. I wrote it because the only program with source that I found for them was written in C and converted them to invalid HTML. I based my efforts on it.

After I wrote the program I was able to extract the contents of an old CRD file I had with riddles, but then was appalled from the immaturity there (as I wrote it during high school). Still, the module may prove useful. It seems like the second CPAN distribution in the File-Format-* namespace after File-Format-RIFF, but I could not find any place better to put it.

Test-WWW-Mechanize-LibXML

Test-WWW-Mechanize-LibXML is another off-shoot of my project for Insurgent Software, that provides some primitives to test web pages' content using HTML-TreeBuilder-LibXML. Right now, it requires some funky multiple-inheritance to work together with Test-WWW-Mechanize-Catalyst because they both override the same function. But it's there.

libfilefind

I've continued working on libfilefind, which is a translation of File-Find-Object to C using glib. Translating the Perl code to C is very tedious, and since I have a lot of Perl code, I decided to translate it all at once, pray, and then test it. After that, I hope to create Perl bindings to this library, and put them on CPAN in order to increase File-Find-Object's speed.

catable

I've done a little work on Catable, which is our Catalyst-based blog engine.

Homepage

I've done some work on the homepage, adding yet-unannounced pages, and also adding some more "<meta />" keywords to existing pages.

Project Euler

This weekend, I've made a lot of progress with solving Project Euler problems. One problem which took me a long time to get right was eventually solved using Test-driven development (TDD) and then I made more progress including solving one problem by hand.

Games-Solitaire-Verify

I released a new version of Games-Solitaire-Verify and this time took Sawyer X's advice and included a small command-line application inside, which proved useful for me as well in my Solitaire research.

2010-08-07: TelFOSS Meeting: Kaltura, the Open Source Video Platform

The Tel Aviv Open Source Club will host a presentation about Kaltura, the Open Source Video Platform by Zohar Babin on Thursday, 12-August-2010, at 19:00 in Shenkar College in Ramat Gan (12 Ann Frank St.) room 300 (note the change of time and place from the previous meetings - we are now meeting in Shenkar College, on a Thursday and on 19:00.). There's a map with directions on the Shenkar College's site. Further details, maps for arrival, etc. can be found on the Wiki.

Attendance is free, it is not necessary to RSVP and everyone are welcome.

With any other problems, feel free to contact the organiser.

Abstract

Kaltura (also see its Wikipedia entry) allows publishers of all sizes to easily, quickly, and cost effectively enhance their web site with video and interactive rich-media functionalities, including video management, searching, uploading, importing, editing, annotating, remixing, sharing, and advertising.

Unlike the alternative proprietary solutions, Kaltura’s platform is completely flexible, extensible and free! Web publishers, value-added-resellers, and integrators use our software development kit to customise a unique rich media experience that fits their specific purposes, and seamlessly integrates with many popular content management systems. Moreover, Kaltura’s reference implementations and growing library of applications, extensions and plug-ins allow publishers to select off the shelf solutions for rapid self-serve deployments that can be fully enabled within minutes.

Visit Kaltura.org for the community site, forums and all the open source projects that are part of the Kaltura Platform.

In the lecture

This lecture is an open invitation to learn more, try out and become a part of this fast growing and exciting open source solution.

About the Lecturer

Passionate about people, ideas, technology and video; Always ready to chat about innovative ideas or creative solutions, Zohar is a community technology activist and programmer. Among his interests are the study of communities, collaboration, open source, digital art, Video and 3D.

Zohar works at Kaltura where he heads Developer Relations & Community.

He is a fellow manager of flashoo.co.il, the Israeli Flash Platform developer community, founder of WebiTalks the Israeli web developer community and frequently speaks at web events worldwide.

Twitter: @zohar And his fresh new Blog

A video artist by heart, developer by night. :)


We are always looking for people who will volunteer to give presentations on various topics that are related to open source code and to computers. In case you are interested to give a talk, or that you have a suggestion for a talk that interests you, we'll be happy to hear from you.

2010-08-09: Report on August Penguin 2010

Hi all. This is a report on the August Penguin 2010 conference, which I've attended and enjoyed. I decided to write it in English due to the proliferation of Hebrew reports, out of convenience, and in order to publicise the Israeli open source activity abroad. So here goes nothing.

First of all, an apology: I wanted to publicise the conference more intensively, but I got distracted by the fact that the opening paragraph in the site was supposed to be fixed (and I did not know how to fix it because I'm not a Drupal expert and there was no edit button there), and that I started working intensively on a for-pay project (and a Perl and Catalyst one at that) shortly before the conference, which turned out to be time-consuming and also distracted me. So I apologise if it resulted in fewer people attending the conference, and I'll try to keep it in mind for next year.

In any case, I posted an E-mail to the mailing list asking for a ride, and Omer Zak (a.k.a "TDDPirate") volunteered to give me a ride. After negotiating an hour, I realised I needed to wake up at 06:00 AM, in order to catch the ride. The alarm woke up me up and I had relatively little trouble to get up. I made some preparations, ate something, and started playing a deal of Freecell in PySolFC (which later on during the day motivated me to find a performant scan for Freecell Solver to solve it), and didn't finish in time. Then I went to meet Omer.

He picked me up on Keren Kayemet Boulevard a long stretch before the designated meeting place. Apparently he was early (or I was a bit late), and had to diverge from the main road. So we drove all the way to Weizmann Institute where the August Penguin conference took place. The ride was relatively uneventful, there were no substantial traffic jams, but we saw another car with some FOSS-related stickers on the way (probably going to the conference as well). We arrived early before the doors of the hall were open, but other people got there too and I've talked with them a little and also posed for a photograph with my name.

After they opened the doors, I helped carry some food and stuff from the cars, and then ate some of the pastries that were brought and drank some assorted juices - all of which were good. (I think I had eaten too much during the conference, because later on, I didn't have too much appetite for the midday meal).

In any case, I met a few old and new people during the conference. One of them told me he met me on Freenode's #perl, where I guided him a bit and that he was on Freenode often. He brought an Archlinux-based laptop, and also said he's been writing some Perl scripts for various tasks. I've also met Dor for the first time as he was volunteering in the Ubuntu-Israel stand.

I should note that I had volunteered to guide a blind attendee and she eventually arrived and I showed her where everything was in the hall. During one of the breaks, I also showed her where Ori was. She asked me to escort her to the bus station in order, but eventually someone else volunteered to escort her a short time before I went out of the hall to do so.

I enjoyed several of the presentations there especially the ones about the 0AD game (which looked great based on the screenshots and screencasts), and the Open Knesset (= the Israeli parliament) project. I stayed up to the giving of the Hamakor prize, where I was invited to the stage to receive the third place for the communal prize for the Israeli Perl Mongers.

Omer eventually gave me a ride back home after wrapping up the transcripting business there.

So the conference was very nice and I enjoyed it a lot. Here's to next year's conference, and hope to see you in the upcoming Tel Aviv Open Source Club meeting this Thursday.

2010-08-25: Perl Debugger Tip: A Session Startup File

Here's a small Perl debugger tip: in order to have a file whose commands will be executed at the start of the debugging session (for example in order to get to a certain point in the code) - a session startup file, similar to gdb's --command=cmds.gdb flag - one can do the following:

First of all write a file with the debugger commands you want (let's call it cmds.perldb) and then when inside the perl debugger say:

source cmds.perldb

This will execute all the commands.

I noticed that after a while the perl debugger stores it inside the history (assuming you're using Term-ReadLine-Gnu), so you can recall it with s and then pressing the history-search-forward and history-search-backward that were set up (Mandriva assigns them to Page Up and Page Down).

Enjoy and I'm sorry for having neglected this blog for a long while!

2010-08-31: Is the web becoming fragmented?

Peteris Krumins writes:

So I participated in the 48 hour Node.js Knockout competition together with James Halliday and Joshua Holbrook. Our team was called Dark Knights and we created an online chess application called Node Chess.

Oh, and it works only [on] Chrome. Ancient-browsers-please-be-gone!

My question is: how can you call Firefox, whose latest stable release was on the 23 of July this year and which has perfectly usable and nightly builds ancient?

It is highly possible that due to the recent hype surrounding HTML 5 and its mutually partial implementation by the different browsers, that we are entering a situation where many sites or demos will only work on particular browsers. This didn't start with Peteris' post - naturally. Previously, someone from the Israeli Internet Society referred me to some demos that said required a WebKit-based browser, and during a presentation about HTML 5 in an "Alphageeks" meeting, the presenter had to use three different browsers, because all the features he wanted to demonstrate did not work on all of them. And this is without taking the account Apple's block of non-"Apple Safari" browsers from its HTML 5 demos and the fact that www.spice-space.org obnoxiously redirects you to "This site requires JavaScript" page if JavaScript is disabled, and many other sites do not function properly without JavaScript enabled.

And as a commenter on Reddit for a web demo said, while the demoscene people have been trying to produce demos that utilise the most out of the computer's resources, the web demos have come to waste a lot of resources in creating anachronistic demos, whose only selling point is that they run inside a browser. As Joel on Software notes: Combined with the speed and responsiveness from Ajax, FogBugz has almost reached the level of speed and fluidity of my dry cleaner's DOS 2.0 character mode database application. And that's pretty darn responsive for a web app.

Are we headed into another "Best viewed with Netscape 2.0", "Best viewed with Internet Explorer 4.0", etc. era of web fragmentation, because we opened the Pandora box of HTML 5? As for me, if I were a judge on that competition that Peteris took part of, I would fail his project due to not being capable of running on my ancient browser.

2010-09-12: Perl Debugger Tip: Viewing Using The Pager

Here's another Perl 5 debugger tip: if the output of a command is too long, one can prefix it with "|" (a vertical bar / pipe) to display it using the pager. So, for example |x $my_object will display $my_object using the shell's pager ( GNU less or more or whatever). Enjoy!

2010-09-23: Copying Ubuntu Bug No. 1

Tel Aviv, Israel: "Ubuntu can't have all the fun only for itself", open source distributors are saying as they rush to copy its Bug No. 1 titled "Microsoft has a majority market share".

Debian, Ubuntu's parent distribution, has set up Debian Bug No. 1 which also reads "Microsoft has a majority market share". Mandriva, another competing distribution has set up Mandriva Bug No. 1: "Microsoft has a majority market share and Ubuntu has a majority market share on the Linux desktop". Fedora, Gentoo, Archlinux, Slackware and other distributions are expected to follow suit.

Mandriva Bug No. 1 Thumbnail

Mandriva's Bug No. 1.

In the meanwhile, Microsoft Corporation (MSFT) have decided that they won't stay out of the game, and have unveiled their brand new web-based bug-trackers, with "Microsoft Windows Bug No. 1: Windows does not have a 100% market-share", and "Microsoft Internet Explorer Bug No. 1: We are losing market-share to Firefox and other browsers.". Among the enthusiastic responses from commentators on the bugs one can find "Why is this a bug, FFS???" by "MS-must-Die" and "If this is a bug then I'm a Jackalope, though you should fix many of the f***ing bug dependencies." by "Slashdot Anonymous Coward".

Some prominent open-source projects such as youtube-dl, abcde (A Better CD Encoder) and bashdb (the Bash Debugger) have decided to duplicate Ubuntu bug No. 1 with all of its comments as their own "Bug No. 1"s noting that "Microsoft's dominance of the market threatens the very fabric of our existence, and it is our utmost priority to fight it".

youtube-dl's Bug No. 1 Thumbnail

youtube-dl's Bug No. 1.

Ubuntu's Bug No.1 proved to be a good publicity stunt for Ubuntu and a focus of active (and often tangential) disucssion. However, now it seems that Ubuntu are not alone in this, and it remains to be seen how their competitors will benefit from copying it.

Ubuntu's former CEO Mark Shuttleworth is pleased with the ongoing popularity of the "bug No. 1" meme. "We believe that Ubuntu's bug No. 1 was the key differentiating factor in the success of Ubuntu, and we are glad other open source projects finally realise that. Hopefully, we will soon open more bug No. 1s (different bugs with the same number) for 'Ubuntu has an equal marketshare to distribution X'".

Meanwhile, some voices on Ubuntu bug No. 1 have called for converting its status from "critical" to "release critical", which will make it block a new Ubuntu release until the bug is resolved. "I am positive that we should not release a new version of Ubuntu while Microsoft still has a majority market-share", said an anonymous commenter, "it certainly must block new releases, and Ubuntu releasing new versions with such a grave bug still open only amplifies the problem."

Licence

Creative Commons License
This work is licensed under the Creative Commons Attribution-ShareAlike 2.5 (Unported) License (or at your option any greater version of this license). Copyright holder is Shlomi Fish and year of copyright is 2010.

To attribute, please mention my name, and link to this page and to my homepage.

2010-10-02: Estimating the Size of the Perl 5 MojoMojo Wiki

There was an myth going on that "you cannot write large-scale programs in Perl". It's been a while since I've heard it but it's possible that Perl just went out of the hackers' radar. In any case, today I wanted to address it by estimating the size of MojoMojo, a wiki written in Perl. MojoMojo stands at the top of the CPAN Top "Heavy" 100 with dependencies on 330 direct and indirect CPAN distributions.

Here's the executive summary: MojoMojo and its dependencies contain approximately 631,222 source lines of production code (what is under the distributions' lib/ directory) - all of it Perl - and approximately 272,027 lines of testing code (what is under the t/'s) , about 265,833 out of which are Perl. The total is 897,055 SLOCs of Perl.

Here is the lib/ report by SLOCCount:

Totals grouped by language (dominant language first):
perl:        631222 (100.00%)




Total Physical Source Lines of Code (SLOC)                = 631,222
Development Effort Estimate, Person-Years (Person-Months) = 174.27 (2,091.23)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 3.81 (45.68)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 45.78
Total Estimated Cost to Develop                           = $ 23,541,431
 (average salary = $56,286/year, overhead = 2.40).
SLOCCount, Copyright (C) 2001-2004 David A. Wheeler
SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL.
SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to
redistribute it under certain conditions as specified by the GNU GPL license;
see the documentation for details.
Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."

And here is the t/ report:

Totals grouped by language (dominant language first):
perl:        265833 (97.72%)
pascal:        5888 (2.16%)
java:           168 (0.06%)
ansic:           96 (0.04%)
cpp:             30 (0.01%)
sh:              12 (0.00%)




Total Physical Source Lines of Code (SLOC)                = 272,027
Development Effort Estimate, Person-Years (Person-Months) = 72.01 (864.08)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))

Now for a longer story of how I collected these statistics. At first I wanted to download all the dependencies. For that I needed to list them. I looked at CPANDB, but could not figure out how to do it properly using it, because its API still has many SQLisms, and really needs a convenient high-level API. My SQL skills were also too low to hack on the SQL directly. However, someone on #toolchain referred me to CPAN-FindDependencies, whose synopsis did exactly what I needed. I ran it and it generated a file with all the dependent modules and their CPAN distributions. This was filtered using this shell one-liner:

cat ~/mojomojo-deps.txt | ruby -ne 'puts $1 if / \(([^\)]+)\)/' |
    sort | uniq > dep-dists.txt

(Note - I've used ruby instead of perl here, not because I don't like Perl, but because I've been trying to learn Ruby lately and could use the practice. I can do the same with Perl, too, by adding -l and replacing "puts" with "print".)

Then I got a list of paths like A/AB/ABIGAIL/Regexp-Common-2010010201.tar.gz , A/AB/ABW/AppConfig-1.66.tar.gz etc. After trying to get the CPAN client to download them to the current directory, I've given up and instead used the following script:

#!/bin/bash
cat ../dep-dists.txt |
    (while read T ; do
        echo "Doing $T" 1>&2
        L="$(basename "$T")"
        if [ ! -e "$L" ] ; then
            lwp-mirror http://biocourse.weizmann.ac.il/CPAN/authors/id/"$T" "$L"
        fi
    done)

If you're using it, please replace the biocourse.weizmann mirror with a nearer mirror - I used one suitable for Israel. Then I had a list of tarballs. I cded to a different directory and ran a loop to unpack them all:

for I in ../dists/*.tar.* ; do tar -xvf "$I" ; done

Then I used the following loop for extracting only their lib directories:

ls -d ../unpacked/*/lib |
    perl -lne '$x = $_; s{\A../unpacked/}{}; s{/}{-}; print "cp -a $x $_"' |
    bash

(Note: it's not safe for paths containing special characters, but I didn't have to worry about it here.)

I used a similar loop for t and then ran SLOCCount (which is written in Perl) on the directory structures yielding the final statistics.

Some conclusions are:

  1. Perl code can scale to large code-bases (over 600K lines of code). On the original link I've pointed, the one who made that statement said after I pointed him to some data disputing his claim that it was possible but it was not easy. However, writing all this code was not extremely difficult and I didn't hear that the maintainers of these CPAN distributions have quit in despair because maintaining the distributions was too hard. So it's possible and not too difficult.
  2. It's probably a bit worrying that there is a relatively small percentage of testing code out of the production code. I expected it to be higher.

Finally, I should note that having discovered that MojoMojo can also be used as a blog engine, I'm considering to abandon my Catable project, which aimed to create a Perl and Catalyst-based blog engine (and which still only has basic functionality) and simply use MojoMojo. Thanks to EdwardIII from #perl for pointing me to that.

Cheers!


I should note that my previous post (which was a joke) took an ugly turn when many of the commenters there thought I was serious and flamed me. I don't usually receive so many comments, but their tone depressed me a little. This post is serious, I promise.

2010-10-10: Bash Tip: Writing Numbers with Commas

One thing that vexes me a lot about computing conventions is that numbers in most computer programs cannot be written with commas. Quick, is 60000000, six million or sixty million? Isn't 60,000,000 much better? Now in Perl, one can use underscores in to separate the digits of numeric constants:

my $one_milliard = 1_000_000_000;

Here's how to achieve a similar effect in Bash:

MAX_DEAL='32,000'

freecell-solver-range-parallel-solve 1 "${MAX_DEAL//,/}" 500 -l by

We've declared the number constant MAX_DEAL with commas, and we got rid of the commas before passing it to the program using the //,/ trick. The latter is part of Bash's parameter subtitution. Search for "${var//Pattern/Replacement}" for the specific use, which does global replacement of a pattern - in our case - a comma, with a replacement - in our case - an empty string.

Death to the unreadable numbers!

2010-10-27: Rehovot.pm Meeting on 9-November-2010

The Hebrew text will be followed by an English one.

מועדון שוחרי הפרל של רחובות ייפגש שוב ביום שלישי ה-9 בנובמבר 2010, במכון וייצמן, כדי לשמוע את ההרצאות הבאות:

  1. Mojolicious - דבק לווב של HTML 5. (אודות התשתית לפיתוח יישומיי ווב) מאת דותן דימט.
  2. POE - מערכת ריבוי-משימות מבוססת אירועים עבור פרל. מאת ירון מאירי (סווייר)

הפגישה תיערך החל מ-18:00 ביום שלישי ה-9 בנובמבר, בחדר 101 בבניין לווין במכון וייצמן ברחובות.

פרטים נוספים ניתן למצוא באתר של שוחרי הפרל של רחובות.

The Rehovot Perl Mongers will meet on 9th November 2010 (a week earlier than expected) between 18:00 and 22:00 in thethe Weizmann Institute in Rehovot in the Levine Building. See the details and map..

The planned presentations are:

Hope to see you there.

2010-10-28: Tech Tip: Making Single Click on a URL Do the Right Thing on Linux

I had a long-time problem on my Mandriva Linux system where, after I set Firefox as the default browser on KDE, it either made KDE first download the page to the hard-disk and redirect Firefox to access it from /var/tmp - or, if I set the default browser to "firefox %u", it opened the link in a new window twice.

Today I decided to go to the bottom of it, after seeing that an upgrade of KDE on Mandriva Cooker did not fix the problem (which I was told would). After consulting some people on IRC, I realised that I could not reproduce the bug on a new UNIX user account, so it was a configuration problem here. Then I realised that /usr/bin/www-browser which is what the BROWSER environment variable points to also opens it this way. After looking through the source code of www-browser, and testing some command-line invocation of capture-shells there, I suspected the problem was with a stray firefox.desktop file.

Running locate found two such files inside my home directory: /home/shlomif/.gnome2/panel2.d/default/launchers/firefox.desktop and /home/shlomif/.local/share/applications/firefox.desktop. Moving away them both resolved the problem and now if the default browser is set to "firefox", the Firefox opens a link in a single new tab after a single click.

Cheers, and hope it helps.

2010-11-12: Recents Enhancements to Perl-Begin.org

This post summarises the recent enhancements to the Perl Beginners' Site.

Perl-Begin continues to be the most comprehensive first-stop for Perl Beginners. Moreover , its content is distributed under the Creative Commons Attribution License (CC-by), and its source code is available to facilitate contributions.

2010-11-13: Recent Hacktivity Log

This is another recent hacktivity log. First of all, I should note that my new day job (working from home for a Tel Aviv-based startup doing Perl and Catalyst (a web-devel framework for Perl)) has been consuming a lot of time and energy, which has taken its toll on my pro-bono open-source work. But I still have a lot to tell.

Freecell Solver

I've done a lot of work on the Freecell Solver trunk, including the long-planned RCS-like state storage of keeping only the moves and the previous state in the states collection. Right now, what I'm struggling with is constructing a good meta-scan that will yield short solutions, based on the new scans I found recently, and fixing a nasty crash after enabling some recent run-time command line flags. I've quite neglected working on it recently due to being distracted with other things.

libfilefind

libfilefind, the C port of the File-Find-Object CPAN module is now passing an initial test. It's much faster than the FFO Perl-based program, but still slower than an equivalent Perl program written using File::Find. So I now need to profile and optimise it further.

Perl-Begin

I did a lot of work on the Perl Beginners' Site. Today I completed adding all the items in the TODO list that I wanted to incoroporate into the "Perl elements to avoid page", so now I have time to blog instead. :-)

Project Euler

I made a lot of progress with solving Project Euler challenges, and in the process discovered a bug in the "primes" program from the bsd-games package. Reading from a bug report I sent to its maintainer and the Debian maintainer:

I discovered a bug in bsd-games' primes:

after downloading http://ftp.de.debian.org/debian/pool/main/b/bsdgames/bsdgames_2.17.orig.tar.gz and doing yes "" | ./configure (the interactive configure script is very annoying, BTW) and "make primes/primes" on my x86-64 laptop running Mandriva Cooker I'm getting the following:

{{{
[shlomif@lap primes]$ ./primes 7757777777 7757777778
7757777777
}}}

However, 7,757,777,777 is not prime:

{{{
$ factor 7757777777
7757777777: 65827 117851
}}}

This tripped me on this Project Euler challenge:

http://projecteuler.net/index.php?section=problems&id=111

KDE's Konsole

Konsole is KDE's console window. I recenty reported a long time bug in Konsole and after I was done, I triaged some bugs. I discovered a crash bug when triaging a different crash, and also tried to reproduce many other crashes.

perlipc

I sent a series of patches to perlipc - the Perl Interprocess Communication document in an attempt to modernise it. This started a dicussion on perl5-porters, and eventually Tom Christiansen posted an "authorial perlipc edit", which was applied immediately. Some of my changes were not incorporated there and now I'll need to start over, but I guess that's life.

Kakurasu Solver

I've written a solver for Kakurasu using lp_solve and its Python bindings. I asked for a code review on the Python-Israel mailing list, and got many comments, which I've incorporated, and I think it's now ready for release.

CPAN Smoking

I've worked on setting a CPAN smoking environment on my Acer x86-64 laptop. I followed the instructions on the CPAN Testers Wiki and it went pretty straightforwardly. Then I started the smoking, but noticed that Perl/Tk got stuck for a long time on its JP.t test. I'd like to find a way to avoid it.

In the process, I tested the installation of my HTML-Widgets-NavMenu and noticed it was still using Class-Accessor, so I converted it to Class-XSAccessor, which should make it faster.

I also received and applied a patch for Config-IniFiles, which I've incorporated into its 2.59 release. I had to quickly release a 2.60 release because the patch caused one of the test files to fail if IO::Scalar was not installed.

2010-11-20: Tech Tips: Fixing KDE and GNOME problems

These are a few assorted tech tips in my tech tips column. The first one is about how to make sure KDE's Konqueror in file manager (or Doplhin for that matter) will remember the state of its "Detailed" vs. "Icons" view for particular folders. While there are many variables for this equation, my problem in a particular place, was that I tried to do it on a Samba SMB/CIFS share (a Windows-like share) that was mounted with permissions of 755 and under my UNIX user-ID/group-ID (but still read-only) which fooled konqueror and dolphin to believe it could write its settings there. Reconfiguring samba solved the problem, though it seems like I found two bugs - one in Samba and the other in KDE.

The second issue also required a lot of experimentation on my part to solve. My problem was that some X.Org applications such as KDE's Konsole, XFCE's Terminal and the IRC client XChat had larger fonts on XFCE and IceWM, than on GNOME and KDE. After a lot of trial and error with stracing the sessions, and moving away dot-directories and dot-files, I realised the GNOME problem was caused by a stray problem in ~/.gconf/ which moving that directory away solved it (one can solve it more hermetically by getting rid of the offending configuration problem, but I did not bother with it yet). The KDE problem persisted afterwards, but was easier to fix: I acked ~/.kde4/share/config for "dpi" and found share/config/kcmfonts and a line in another file that set the dpi to a different value which I fixed, and that made both the GNOME and KDE font size and DPIs (dots-per-inches) to be the same as those of IceWM and XFCE. Now the relief has made me very happy.

If you find these tips helpful, drop me a note at the comments below.

2010-11-28: Introducing Module-Format

This weekend I worked on Module-Format, a new CPAN distribution, and yesterday released version 0.0.1 of it to the CPAN. In a nutshell, Module-Format allows one to convert strings representing Perl modules (and their encapsulating distributions) to representation-netural objects, and to render them into other formats of modules. To better explain the motivation, let's start with a few use cases for this.

If you're working on Debian, Ubuntu or a derivative and would like to install Perl distributions using apt-get, you can define the following shell function:

up()
{
    apt-get -y install $(perlmf as_deb "$@")
}

And then inovke it like up XML::RSS XML-LibXML DBIx/Class.pm (note the inconsistency - perlmf handles all those formats for you), and it will install them.

Similarly in Mandriva using urpmi you can use:

up()
{
    urpmi --auto $(perlmf as_rpm_colon "$@")
}

And then type the same command. (It will actually be better because urpmi will fetch individual rpm provides modules such as perl(XML::RSS) that may be contained in larger distributions).

Similarly to install stuff using cpan2dist, one can say something like:

function c()
{
    cpan2dist --verbose --format=CPANPLUS::Dist::Mdv \
        --install --timeout 0 $(perlmf as_colon "$@")
}

So this is what perlmf, which is the command-line program that ships with Module-Format can do. Module-Format provides a more generic Perl-based API for that, and replaces some ugly and ad-hoc shell functions that I defined for that.

Module-Format is: 1. Free and open source software distributed under the permissive MIT/X11 licence. and 2. Only dependent on core modules and List::MoreUtils (a dependency which I'm planning to remove in a future release). The source code is maintained in the web-cpan Mercurial repository, so feel free to contribute.

Happy module formatting.

2010-12-16: Tel Aviv Perl Mongers Meeting on 29-December-2010

The Tel Aviv Perl Mongers (part of the Israeli Perl Mongers) will meet on 29-December-2010 at Shenkar College, Anna Frank 12, Ramat Gan, room 323. The gathering is going to be at 18:30 and the talks will start at 19:00.

We are going to hear these talks:

Why Perl? (Sawyer X)

A short talk on what makes Perl such a compelling language to learn and work with, while focusing on Modern Perl and innovative technologies.

What's new in Perl 5.10 and 5.12 (Gabor Szabo)

Find out what exciting features new versions of Perl offer.

Read only state for fun and profit (Yuval Kogman)

Even though Perl is highly optimised for modifying data in place, using read only state can be very rewarding. What is immutability and when it should be applied.

2010-12-17: Accentuate.us - a service for helping save human languages

I was requested to publicise this announcement.

Are You the Right Type?

As of today, about 7,000 languages are spoken in the world. More than 90% of these are expected to disappear before year 2100, leaving a best case of only 700 languages. Language death is an irrevocable loss akin to animal or plant extinction–it is the loss of a repository of culture, tradition, and world view.

Nineteen-year-old entrepreneur Michael Schade, of Spearhead Development, has teamed up with Dr. Kevin Scannell, of Saint Louis University, to revolutionize computer input and help save endangered languages.

Due to inadequate keyboards, or clumsy input methods, speakers of hundreds of languages often simply leave out the accents and other special characters when typing, which leads to ambiguities and confusion. We believe strongly that speakers of all languages have the right to type their languages quickly, easily, and correctly.

We are proud to announce the free and open-source Accentuate.us, a new method of input for over 100 languages that uses statistical reasoning so that users can type effortlessly in plain ASCII while ultimately producing accurate text. This allows Vietnamese users, for example, to simply type “Moi nguoi deu co quyen tu do ngon luan va bay to quan diem,” which will be automatically corrected to “Mọi người đều có quyền tự do ngôn luận và bầy tỏ quan điểm” after Accentuation.

To date, we support four clients: Mozilla Firefox, Perl, Python, and Vim, with more to be added shortly.

Mozilla Firefox - Available today at http://accentuate.us/get/firefox through Mozilla AMO, the Firefox add-on brings the power of the Accentuate.us service to anywhere on the web: blogs, web mail, social networking, and anywhere else where users input text. Please check out the short demonstration video at http://accentuate.us/help.

Perl and Python - The Perl and Python clients available on http://accentuate.us/ are command-line utilities that work either with argument flags or through stdin. This versatility has proven powerful, allowing Accentuate.us to be easily integrated into any program that can operate through such inputs, such as vim.

Vim Provided by Bill Odom, co-founder of vim-geeks and previous President of The Perl Foundation, and available today on http://accentuate.us/, the vim plugin implements user-customizable maps to Accentuate over visual, motions, and ranges. It harnesses the Perl or Python clients, exemplifying just how easy it is to plug in to the Accentuate.us API.

Plans are currently underway for extending the Accentuate.us service, including enhancements to the Firefox add-on and support for all 114 languages. Support is needed especially for African languages, many of which use special characters. There are, for example, over a million words of Lingála on the web: blogs, news articles, Wikipedia entries, etc., but well over 95% of this content is typed in plain ASCII, despite the fact that the standard spelling system makes heavy use of diacritics to mark tone and special “open” vowels such as ɛ and ɔ.

We appreciate any assistance that we receive to improve service and keep it free, including translating the interface, hosting a server, or donating to help us maintain existing servers. Find out how to help at http://accentuate.us/contributing.

2010-12-26: Reminder: Tel Aviv Perl Mongers Meeting This Wednesday

This is a reminder that the Tel Aviv Perl Mongers will meet this Wednesday (29-Decebmer-2010).

The Tel Aviv Perl Mongers (part of the Israeli Perl Mongers) will meet on 29-December-2010 at Shenkar College, Anna Frank 12, Ramat Gan, room 323. The gathering is going to be at 18:30 and the talks will start at 19:00.

We are going to hear these talks:

Why Perl? (Sawyer X)

A short talk on what makes Perl such a compelling language to learn and work with, while focusing on Modern Perl and innovative technologies.

What's new in Perl 5.10 and 5.12 (Gabor Szabo)

Find out what exciting features new versions of Perl offer.

Read only state for fun and profit (Yuval Kogman)

Even though Perl is highly optimised for modifying data in place, using read only state can be very rewarding. What is immutability and when it should be applied.

2010-12-29: C Pointer Comparison Gotcha

Some days ago I integrated a new BSD-licensed balanced binary tree, based on the red-black tree of kazlib into Freecell Solver (fc-solve) and ported two of its collections to it. I've written a comparator function for it and asked my IRC pal Amadiro to test the new fc-solve with some memory optimisations on one of his University's HPC machines which he had access to. However, it crashed pretty quickly.

I was able to reproduce a similar crash on my x86-64 laptop, and tried to investigate it. Then it hit me, because the comparator callback was this:

extern int fc_solve_compare_lru_cache_keys(
    const void * void_a, const void * void_b, void * context
)
{
#define GET_PARAM(p) ((unsigned long) \
    ((const fcs_cache_key_info_t *)(p))->val_ptr)
    return GET_PARAM(void_a) - GET_PARAM(void_b);
}

Do you see the problem? Subtracting two unsigned longs will always yield a non-negative value, which will yield the comparison void. I considered several alternatives to this including using longs instead of unsigned longs (which will still be a problems on platforms where long is less than the pointer size), or subtracting const char * (which may overflow the int return) and eventually opted for the safest version of:

typedef const char * fcs_lru_side_t;

extern int fc_solve_compare_lru_cache_keys(
    const void * void_a, const void * void_b, void * context
)
{
#define GET_PARAM(p) ((fcs_lru_side_t) \
    (((const fcs_cache_key_info_t *)(p))->val_ptr))
    fcs_lru_side_t a, b;

    a = GET_PARAM(void_a);
    b = GET_PARAM(void_b);

    return ((a > b) ? 1 : (a > b) ? (-1) : 0);
#undef GET_PARAM
}

This way I'm sure that the comparison works correctly, and indeed after that Freecell Solver ran fine on the HPC machine, and would have appeared to scale to up to 600 million positions within 64 GigaBytes instead of the previous 400 million positions.

If you're a sucker for pretty graphs, you may wish to inspect the OpenDocument spreadsheet (.ods) of the memory consumption of the solver, though the data is quite estimative.

2011-01-02: Report on the Tel Aviv Perl Mongers Meeting

This is a report on the latest Tel Aviv Perl Mongers meeting. During that day, I woke up relatively early (07:00 or so as far as I can recall), and did not go to bed before night time so I was a little tired later on. Before the meeting, I spent some time trying to get the game Aquaria, which I got as part of the 2nd Humble Indie Bundle to work properly on my Linux system, unsuccessfully. (A more recent update is that someone eventually helped me after I filed a bug report, and now I am able to play the game nicely.). This delayed me a little, and I left relatively late to the meeting.

After I arrived to Shenkar, I met Ido and a friend of his, and guided them to the room, which was different than the one where the Tel Aviv Open source used to meet. We met some people there, including Sawyer and I decided to go to the nearest kiosk and buy some snacks. I bought some chocolate chip cookies, Doritos and a chocolate and hazelnut treat and returned with still a lot of time before the lectures started.

In the meantime, other people arrived there, and we talked about how to get more people to contribute, about making use of CPAN modules in projects and about which versions of Perl are used in the enterprise. Then the talks began.

Sawyer X gave an entertaining presentation titled "Why Perl?" with several jokes in between that got the audience to burst into laughter. He mentioned this page titled "Your language sucks" where several of Perl 5's shortcomings are mentioned (along with other languages), and concluded with bad things about Perl, like the fact that it makes you lazy, and that you can't stand programming in other languages. He also mentioned there several ways to start with Perl, but I was disappointed that he did not mention the Perl Beginners' Site (Perl-Begin).

After that, Gabor came and talked about what's new in perl-5.10.x and perl-5.12.x. He covered most bases and then when he got to perl-5.12.x's while (my ($index, $value) = each(@array)) I asked him if there was also keys(@array) and he said he doesn't know. Today I tried it out and there is indeed keys(@array):

shlomif:~$ perl -E 'my @a=(50 .. 60); say join(",",keys(@a));'
0,1,2,3,4,5,6,7,8,9,10

There's also values(@array), but it's likely the same as "@array". There are many small and large improvements in perl-5.10.x and 5.12.x and he mentioned that the perl5-porters are now switching to a yearly release cycle, which will make improvements materialise in production more rapidly.

After that, Ilan did a round of introductions where every person introduced themselves and said which languages or technologies they work with and why they came to the meeting. It was entertaining.

Then came Yuval's presentation about "immutable state for Perl". He spent a short time explaining what is mutable state, and that if we want to create a modified object, we need to clone it with different constructor arguments. He spent the rest of the time touting some of the advantages of immutable state, like easier distributed NoSQL storage due to lack of cache invalidation, and he described how he handled a problem in KiokuDB by creating a buffer for it, and also focused on a recent problem we had at my (and previously his) work for Reask where HTML::FormHandler misbehaved due to modified state after reset, and we were unable to resolve it and had to switch to a different form generator (HTML::FormFu).

Some people with whom I talked felt this presentation was a little bit abstract and "floating in the air" and I tend to agree, but I still enjoyed most of it.

I left shortly after that presentation because I was tired (as I had woken up early that day) but based on Sawyer's post, some people went to dinner afterwards and had a good time. I walked with mksoft to the bus station and talked about games in Linux and the Humble Indie Bundle 2 which we both had bought. Then a suitable bus had arrived and I took it home.

To sum up, I had a great time, about 20 people came, which meant the meeting was a success, and I'm looking forward for the future meetings. Meeting at Ramat Gan is much more convenient for me than the Rehovot meetings and I met some people I didn't see in Rehovot like Ran, who also gave me some money in exchange for the refreshments I bought (Thanks!). I'm looking forward for the next meetings and thanks to all the people who had organised and publicised it.

2010-12-16: Tel Aviv Perl Mongers Meeting on 26-January-2011

The Tel Aviv Perl Mongers (part of the Israeli Perl Mongers) will meet on Wednesday, 26-January-2011 at Shenkar College, Anna Frank 12, Ramat Gan, room 323. The gathering is going to be at 18:30 and the talks will start at 19:00.

This meeting will try to focus on web development and will (as any meeting) include both beginner talks and intermediate ones. We will also start a tradition of exhibition! Module exhibition, that is. Pick a module you want to exhibit and take 5 minutes to talk about it. If you want to exhibit a module, you can talk to me privately (if you need a laptop, or funny hats) or you can just step up in between talks.

We are going to hear these talks:

What the hell is web development? (Sawyer X)

A nice, easy, hopefully interesting, understandable introduction to web development.

The web stack (Ilan Arad)

A hands-on, discussion-oriented talk on advanced understanding of the web stack, from server-side to client-side. (advanced talk)

As always, there are a few good reasons to come to the Perl Mongers meetings:

Please feel free (and do) invite anyone you want!

2011-01-29: Update and Recenty Hacktivity-ish Post

Well, I'm still alive and kicking, so I decided to put a small update about what I have been up to.

First of all, I attended the FogBugz and Kiln World Tour, which I did in order to see Joel "on Software" Spolsky speak, and was otherwise a blast. In the second part, they gave several good reasons for why Mercurial and git are better than Subversion, and the demo of FogBugz was also instructive, but seems like it contains many features that won't be useful in my daily work as an open-source developer. They also said that the reasons they prefered Mercurial over git were because it had a somewhat friendlier command names (which are more familiar to people coming from Subversion and CVS) and because it worked better on Windows, and the git people were hostile towards fixing the bugs that git exhibited on Windows. Otherwise, they said that git was a fine distributed version control system.

I also attended the latest Tel Aviv Perl Mongers meeting, which was nice, and this time stayed for the after-party dinner. I'd like to give a more detailed report about it later.

I'd like to announce that, in case you have not heard about it, KDE 4.6.0 is out.. It has many big and small improvements, including some speed boosts, and runs like a dream on my Mandriva Linux Cooker system (though Amarok got broken with the upgrade to MySQL-5.5.x). Here is also the OSNews.com coverage and Slashdot.org coverage). I should also note that I've recently downloaded and installed the Amarok 2.x for Windows distribution and it ran very fine there, but as opposed to Linux could not play the video files I tried at all.

After some length attempts to contact the originator of the Text-Format CPAN module, I was finally given a comaint status and I uploaded Text-Format-0.53 to CPAN, its first release since 1998, and the first one that contains a dash before the version digits (a small difference, but it confuses some packaging systems). There's still some work to do there, but I haven't forgotten it.

Also in CPAN-land, I've uploaded a new version of Config-IniFiles, which contains a patch I applied from Peter Xu, who asked me to be his mentor in his first steps in contributing to open-source. I've expressed my willingness to be a FOSS mentor for some time now and several people contacted me about it, but this is the first time that a code contribution came out of it, and I'm happy with Peter's patch. I guess good things come to those who wait.

I've recently started reading Kent Beck's "Test Driven Development: By Example", which I can recommend so far, and also been working on some other open-source projects which I'd rather not tell too much about now.

Hope you are also doing fine, wherever you are and happy hacking.