Subject: Free Software-like equivalents of Google?

From: Omer Zak <[email-masked]>
Date: Mon, 07 Nov 2005 12:11:54 +0200

Now Google is in the news, and "threatens" to engulf the entire world
and become the next Microsoft.

Just as Linux is competing with Microsoft and is checking its threat, it
may be a good idea to have something, which competes with Google.

Google is based upon a worldwide network of servers (several of them in
physical proximity to each other).  Free Software has been used to build
cooperating networks of computers for several years (FidoNet, UUCP, the
TCP/IP suite itself upon which the Internet is based, various grid
computing platforms).  So it should not be an insurmountable obstacle to
develop software for cooperating information servers.

It may be a good idea to implement the following Free Software projects:

1. "Holographic" database - database which is distributed among a
network of cooperating computers, in such a way that no single missing
computer would cause data to be inaccessible.  (When one speaks about
search engine results, it is acceptable to lose data about low-ranking
Web pages.)
Such a database would allow people to assemble gigantic databases and
store them in several cooperating computers over the Internet - at
reasonable cost.

2. Distributed Web spider network - a network of coordinated spiders,
which traverse the WWW and feed an holographic network with analysis of
the Web pages accessed.

3. Distributed search engine - which receives search requests and knows
to dispatch them to few database computers and get results.
In the past, there used to be some Web sites, which allowed you to
submit a query to few search engines in parallel.  Then the results of
those searches were combined and presented to you.  The same principle
could be used now.

Then go on and reimplement other Google services (except perhaps for
revenue-generating services) on the same platform.
                                                --- Omer
-- 
Sent from a PC running a top secret test version of Windows 97.
My own blog is at http://www.livejournal.com/users/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html

---

From: Arik Baratz <[email-masked]>
Date: Mon, 7 Nov 2005 12:43:44 +0200

> those searches were combined and presented to you.  The same principle

When there is no single entity governing those databases, and they
will become popular, eventually commercial enterprises will start
thinking up ways of making money off these databases, and subvert the
integrity mechanisms that those databases have.

They are trying to do it to Google today, and there's every reason to
believe that they will try to do it to your futuristic distributed
search engine. You could battle it with integrity algorithms and
reputation systems, but as we all painfully know, a net of millions of
zombies can subvert a reputation system.

While I don't like shooting your idea down, how do you propose the
system can be made resilient to the huge economic pressure that a
system in that order of magnitude will inevitably attract?

-- Arik

---

From: "Nadav Har'El" <[email-masked]>
Date: Mon, 7 Nov 2005 13:03:13 +0200

> integrity mechanisms that those databases have.

The fact that we're talking about open-source software, and about a
distributed database, does NOT entail that the database does not have a
single owner, or is not controlled by a single entity.

In fact, I can imagine a (free) software like NFS or AFS or GFS or whatever
on steroids, which enables one to build a huge multi-terabyte filesystem
using hundreds of separate computers; On this "file system", applications
such as a search engine or mail server could be built. The fact that such
a file system allows you to build on many separate computers doesn't
necessarily mean that you would give control of each computer to a different
person: rather, you'll allow every organization that has sufficient means
to buy 10, or 100, or 1000, or 10,000 cheap computers, and run a huge
search engine or mail server or whatever on these computers, just like Google
runs now.

In other words, the software will be free, but real cash will still be
needed in order to buy the hardware.

> search engine. You could battle it with integrity algorithms and

The fact that a free software search engine will have the same problems
that Google now has (spammers, DoS attacks, etc.) is obvious. But there
is no reason why a free-software implementation should be MORE suceptable
to attacks than Google. I don't subscribe to the belief that just because
Google's algorithms are secret, spammers find it harder to manipulate their
ranking.

-- 
Nadav Har'El                        |       Monday, Nov 7 2005, 5 Heshvan 5766
[email-masked]             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Willpower: The ability to eat only one
http://nadav.harel.org.il           |salted peanut.

---

From: Omer Zak <[email-masked]>
Date: Mon, 07 Nov 2005 12:57:37 +0200

> system in that order of magnitude will inevitably attract?

How about a reputation system based upon personal views?

Each item of knowledge will have one (or more) entities which vouch for
it.  Those entities are, too, subject to vouching by other entities.

An user will select some entities upon whose word he relies (in a fuzzy
logic way) and each time he accesses the database, the results will be
ranked according to degree of endorsement they received from the
entities (directly or indirectly) chosen by the user.

For the special case entity==Google's PageRank algorithm, this reduces
to the ranking provided by Google.

If an entity vouches for a knowledge item, which the user rejects as
inappropriate, the user will declare in his profile that this entity is
not to be relied upon in the future.
                                       --- Omer
-- 
Sent from a PC running a top secret test version of Windows 97.
My own blog is at http://www.livejournal.com/users/tddpirate/

---

From: Arik Baratz <[email-masked]>
Date: Mon, 7 Nov 2005 13:23:59 +0200

> single owner, or is not controlled by a single entity.
[snip]
> needed in order to buy the hardware.

So you're talking about re-doing Google. Okay, only I don't think Omer
meant that.

> Google's algorithms are secret, spammers find it harder to manipulate their

I do think that the algorithms' secrecy plays some part. There is a
way around it - consistantly change the algorithm, stay ahead of the
curve.

-- Arik

---

From: Arik Baratz <[email-masked]>
Date: Mon, 7 Nov 2005 13:26:44 +0200

> How about a reputation system based upon personal views?
[explanation snipped]

How exactly are you going to prevent spammers from using a huge
bot-net to vouch for their stuff, masquarading as people? Give them
all turing tests? There are easily workable workarounds around these.

-- Arik

---

From: Omer Zak <[email-masked]>
Date: Mon, 07 Nov 2005 13:35:57 +0200

> all turing tests? There are easily workable workarounds around these.

The same way you don't get pr0n stuff when you surf or search for
information about Web search algorithms.

If you find that you got into spam information, based upon endorsement
from an entity, then you stop relying on that entity in future searches.

Then those zombies and Turing-souls can endorse each other to the
seventy sky, you just do not use their endorsements for any purpose.
                                            --- Omer
-- 
Sent from a PC running a top secret test version of Windows 97.
My own blog is at http://www.livejournal.com/users/tddpirate/

---

From: Arik Baratz <[email-masked]>
Date: Mon, 7 Nov 2005 15:21:59 +0200

> seventy sky, you just do not use their endorsements for any purpose.

What's preventing another set of zombies searching for the
information, getting the spam and marking it as a good and proper
response?

-- Arik

---

From: Zohar Kelrich <[email-masked]>
Date: Mon, 7 Nov 2005 15:34:28 +0200

> information, getting the spam and marking it as a good and proper

More emphasis on "you" as singular in Omer's description -- maybe  
"thou" would fit better :)

The idea is a web of trust, I trust you, and you trust Jim, so I have  
trusted trust ofJim. We don't ever need to start trusting the zombies  
in the first place, and if we do trust them and they fail us, we can  
revoke that as well.

You could also have various trust levels for certain people in  
certain fields, perhaps. However, this would be a good infrastructure  
for a distributed index, rather than a distributed search engine.

---

From: Arik Baratz <[email-masked]>
Date: Mon, 7 Nov 2005 16:15:52 +0200

> in the first place, and if we do trust them and they fail us, we can

The problem is that:

1. I don't think it scales well. Is the PGP web of trust well connected?

2. A zombie network can gain reputation because it doesn't have to
expose itself all at once. It can masquerade as a benign network of
nodes, and gain reputation, and then slowly subvert the network. It
will be difficult to root out.

-- Arik

---

From: Uri Even-Chen <[email-masked]>
Date: Sun, 13 Nov 2005 13:55:14 +0200

> may be a good idea to have something, which competes with Google.

I think Google is already a new Microsoft, and should be treated as
such.  Google is a monopoly and treats its customers the same way other
monopolies do (Microsoft).  There is no real competition for Google
Adwords, Adsense and even Google Search and Gmail.  When a company
becomes a worldwide monopoly, they become arrogant and give poor service
to customers with high prices.  It's not a coincidence that Google likes
to hire people who formerly worked at Microsoft.  Even the new CEO of
the Israeli branch of Google used to work at Microsoft.

Here's one example:
http://www.google.co.il/intl/iw/management.html

This Hebrew page is corrupted and I think Google don't care.  I already
wrote Google about it about 2 years ago (and a few times since then).  I
think it's disrespect towards Israeli & Hebrew reading customers and
customers in general.

And there are many other examples.

Best Regards,

Uri Even-Chen
Speedy Net
Raanana, Israel.

E-mail: [email-masked]
Phone: +972-9-7715013
Website: www.uri.co.il
--------------------------------------------------------