P. S.: Post Scriptum

This morning I received a mail from Copenhagen. It was very moving, and describing a situation of chaos, strong commitment, and braveness. It told the story of people fighting with non violence, and shouting that they want change.

And I am afraid all this is useless. I feel once again what I felt looking at Iran insurgency. But stronger.

Let’s focus on Copenhagen. The sensation is that there is a lot of people on the street asking for a strong carbon tax. Count me among them. But there is more. I am afraid people have ideas, and those ideas are not being heard. And then people assume the worse, and assume the world leaders, the ministers, and everybody who is inside is on the pay check of some big corporations. And then they demand change. But now they do not focus any more on the small change. The key point. Now they want a huge change, that will not happen. And then there are rallies, and people pushing, and the police resisting. And violence. Yes police violence should not be there. And I feel this is not the way. It is not by shouting “Shame on you”, that you win the heart of the police men. It is not by shouting to people that you get yourself heard. As it does not change if I write this in normal letters or in CAPS LOCK. It is the content that matters And when you are shouting, when you are polarized, you are already making violence. This is not the way.

Now, there are people who work hard to negotiate among different positions. The Center for Non Violent Communication is probably one of the best. It was created by Marshall B. Rosenberg. One of the student of Gandhi (or so I remember from his book. The wikipedia page does not seem to mention it). Now Marshall has worked in the past as a negotiator between groups, and I am sure there are a number of very good negotiator working inside the conference to negotiate between the  key people. What I don’t think there is, are negotiator between the people in the conference and people outside.

It is like all the effort is concentrated in getting the communication going between those big players. But no work has been done to get enough communication between the inside of the conference and the outside. The assumption seem to be that either there are no good ideas outside or it is just impractical to engage them. I think both of those assumption are wrong. Yes, we still need to develop the tools to make an efficient brainstorm with millions of people. But the idea of having everybody writing their own ideas, and voting on the ideas they like is already a good start. Why is there no system like this to harvest the ideas from the people?

I was just looking at a youtube cnn conference where people sent questions, and voted on the questions. Again there is the assumption that normal people are just ignorant. This is not true. Not anymore (if it ever was). Not with the internet that let anyone study any topic.

In all those situations we need to set up systems where people can chip in their ideas. While it is happening, can read each other ideas. And the most voted ideas emerge from the noise to the people who are making the discussion.

[crossposted on the moblog, and the facebook notes.]

One of the leit motif in spirituality is to reach an integration among the various parts of oneself. There are many important reasons for this, which I am not going to enter right now. Becoming One is not seen in Taoism as a spiritual goal, but as a spiritual prerequisite. It is not school, it is preschool. Until you are one you cannot really get involved with spirituality. It is like if in your family you decide to build a house, but not everybody agrees on that. Then one part of you builds it in the morning, and someone else of the family will destroy it in the evening. Maybe using the bricks for something else.

The idea that we are many, that each of us is many, is quite common. In psychology is common, Junghian Psychology, if I recall well. Again, in Taoism it even reaches the point of believing that this is true in a litteral point of view. Each of us, is seen as a patchwork of different spirits (shen). And when you die each spirit will then go its way. As such in Taoism until you have reached a real integration between your parts of yourselves (your spirits), you cannot even have reincarnation unless you have developed a unit which is integrated enough to go through the trauma of death without shattering in a 1000 little pieces.

And another idea that is very common (you have it in Taoism, but also in Christianity, for example), is the idea that one day, one time, at some point we will all get together. Christian say “sit by the father”. In Taoism the idea is that any person who have showed a spark of interest for spiritual work will eventually join together in some place beyong space and time, a sort of heaven. And the joke then is if people are following the 1 lifetime program, the 10 lifetime program, the 100 or 1000 lifetime program, to reach it. And the faster it is, the rougher it is.

I have to say I am amazed by how well is Facebook helping in this integration work, for me. I have many friends, on facebook. But more importantòy I have friends from different groups. Each friend knew a different Pietro. Some were from my spiritual life (taoism, tai chi, meditation, …), some from my academic world (artificial life, mathematics), some are Go-brothers, others people I knew from childhood, or from high school, or middle school. And with each of them I was a different person. And now they are all together. All in the same place. And the internet does indeed feel a little bit like this place beyond space and time. And I read of many of them. But what is more important, is that, as I write about my life, I am forced to write in a way that is acceptable for both my academic side and my spiritual side. I can only write in an integrated way, because I know that friends from both worlds will read me. In this sense facebook is catalysing an integration in me. Is helping me to become one.

I know many people are having problems with facebook. I think a lot of the problem is that they are not ready or willing to have this integration. For me Fb is pretty easy: to become my friend you need to know me. With very few exceptions I do not add anyone who is not someone I personally know. But if I have met you, and you want to befriend me, then you are in. I don’t keep people that I know out of the door. Because that would be equivalent to keeping some part of myself out of the door, the part of me that interacted with them. You are all invited to the party. I sometime even go back in time, and look for people I once knew. People that were important in my life. Or people I wished I had the time to know better. Maybe now we have another occasion. But then on my status, in my notes, in the caption of my photo, I try not to speak thinking about one in particular (I might have done it, but mostly I try to avoid it). I speak to all my friends at the same time. And if anyone comments, I answer that person, personally. The answer is personal, but anybody can see it, and thus the integration goes deeper. I write in English and in Italian, because those are the languages with which I live, work, chat, play and love. My inner dialogue is sometimes in Italian and sometimes in English, depending where I am, what I am thinking of doing. And my facebook reflects that.

Most of you know that I use facebook pretty frequently. I update the status often, sometimes more than once a day. But what some of you have not realised is that I do not do much less on facebook. I avoid facebook applications. I only use the ones that are truly useful, that add functionalities that were not there, and are truly helpful. If I want to wish to my friend Happy Chinese New Year, I will do it in person, or through the status. Not through an application. In this way the integration proceeds. I very rarely invite people to use applications. I only do so when I think an application is very very good. (The “skip this” button is my friend). I invited my friends for the geo tagging application. I would do it for the “cause” application. Maybe the iRead could be another one, and the application to play Go online. Here you go, this makes it 4. And when I invite people I only invite people I think will appreciate it (or should, they know it or not ). I consider the other applications to be equivalent to spam. I try not to spam my friends. When a new application arrives (elves, and pirates, etc…) , I usually just block it. If an application is requiring me to send invitation to let you proceed, I report it (because it is breaking the TOS, and ruining the party for everybody), delete it and block it. With absolutely no pity, whatsoever.

I see often people who get tired of facebook. But very often those are people who are not using facebook as a tool to interact with friends that are far away (in space or time), but as a game. Those are the friends that use more of those facebook useless applications. They get tired, but what they are really getting tired are those useless applications. They are right in getting tired. They just need to use facebook, instead of be used by it. And then fb will stop being a toy, and become an instrument. You will forget about facebook, and think about your friend.

Keeping the application to the minimum necessary.
Speaking to everybody. Inviting all your (real life) friends.
It is fairly easy to let facebook help you in the integration process.

It is now the time to present the next project we have been working to: TagBay. And I say ‘we’, because is this project I am not alone. I did it with a friend of mine, Derek, who accepted, very patiently to code, some of the idea I have been tinkering around in the last year or so. I am speaking about how tags, and tag clouds, and distance between tags, and so on.
So, in brief we made a web site to tag material that is being sold on e-Bay. Anybody can tag any object that is being sold. Not only can any object be tagged but you can tag sellers, too (oh, we are not responsible for offensive tags, eh!).
Tags on objects can be made private or public, and you can also search among your tags, among everybody else tags, and eventually (when we code it) it will be possible to search among the tags of another user, like in del.icio.us.

Now that the summary for the people who have no time has been done, let’s try to explain the idea in the details for those who have a bit more time.

Pages:
On TagBay, right now, there are 3 type of pages: e-Bay Search Pages, TagBay Tag Search Page, TagBayUser Tag Search Page, Item Page, and Seller Page.

• Search Page: It is possible from inside Tag Bay to make searches on e-Bay on specific keywords. The user can then add tags to each object that came out, store the tags added all at once, or store the tags of a single object. The same thing can be done in the Tag Search Page
• TagBay Tag Search Page: In this page the user gets all the results for a single tag that someone have used. Nothing fancy (for now). Items where the tag only appears as a private tag will not appear here.
• TagBayUser Tag Search Page: In this page the user gets all the results for a single tag that he have used. If the user is logged in and is looking at his own tags, also the items tagged in a private way will appear.
• Item Page: Each object has its specific page. From such page any user can see what are the public tags that other users have used for that page. Also they can define their personal tags for that object, if their tags are going to be private, and the tags of the seller.
• Seller Page: And then there is the seller page, and in the seller page any user can tag any seller. The use of tag for sellers is still limited, but will be increased in the future.

The natural use of the site

• For a seller or for a shop A seller might want to use the site to tag all the objects that he is selling, giving for each object all the tags related. Thus increasing the possibility for it to be found. We suggest to list the tag in the order of importance, as soon we are going to use the order consider the importance in the search page.
  Also, if a person wants to make a cool list of objects, they can tag exactly those objects, with a tag they never used, and then link to the page in their directory of this tag. Thus creating on the spot, their lists. Also sellers will want to tag their objects, and people making searches will tag objects to make lists of objects they want to follow, before jumping on a transaction. We think there is more than enough material to generate interesting behaviour. It doesn’t have to be exactly the same emergent behaviour that we are used to see. After all we are just exploring the possibilities of social folksonomy.
• A shop To the possibilities before, a shop who is selling on ebay might be interested to make sure that the shop itself (remember that you can tag sellers, and not only shops) have all the tags related to the merchandise that they are selling
• Someone buying Our suggestion for someone who wishes to buy, on e-Bay, would be to first look under the tag search, to see if there is anybody who has already tagged any object that they are interested in. This does not necessarily be someone else who is buying, but also someone who is selling. Then tag the objects they are interested themselves, to have it in their own list of objects. Then they could go to the search e-Bay page with the necessary keywords, and add the chosen tag to all the objects interesting. At that point a first selection have been made, and all the possible objects have been tagged. At this point, he could choose one or those objects, change the tags to private, and start betting on it.
• Someone suggesting And finally if someone is just trying to suggest some possible objects, he could search e-Bay for those objects, tag them with a unique tag and present the url of the list to whoever is interested.

There are many other ways to use TagBay. In a sense TagBay is a toy, and not a game. And as every good toy it can be used for many different games. We suggest here only some of them. Also TagBay itself is rapidly evolving. We have tons of stuff we are interested in including, and if you have been reading my blog, you know how my problem is always to find people to code my ideas, more than to find them. And this is why I am so happy for Derek work!

Difficulties that we found:
There were a number of issues that came out when we started developing this program.

1. Public vs private tags:
   Why would someone tag an object if they are interested to buy it? After all aren’t they making it easier to others to find it, by adding those tags?
   This was a serious doubt that we had, and finally we decided to give the possibility to users to tag objects privately. Yet there have to be a balance between private tags and public tags, as public tags are necessary to generate the emerging folksonomy that we wish to use. So we decided for a compromise: public tags can be done from the search page, but private tags requite you to go to the specific object page. In our view (but we are ready to be proven wrong) someone would go to the search page, tag all the entries where he might be interested. Then chose one, and tag that one in a private way.
2. Limitations due to the temporary nature of the objects
   Considering that most object exist on ebay only for few weeks before being sold, wouldn’t this be not enough time to make a tag cloud and let all cool emergent properties that folksonomy induces, appear?
   Maybe, but sellers also can tag the objects they are selling, thus giving a fresh start to all the objects. Also side by side to tagging object we are giving the possibility to tag sellers. Which eventually should survive each transaction and build up an interesting tag cloud.
3. I spoke about sellers tagging their own objects, but wouldn’t this invite people to spam your site? After all, wouldn’t it be much better for a seller to add many tags to be present in many searches?
   Ah ha! You think tag clouds can be spammed. This is false. Tag clouds cannot be spammed, and no one understand this. And we shall use this site to prove it. We have nothing against spammers, they are absolutely welcome in our site and spam it as much as they feel. Add all the tags they want to each object they sell. It will make ABSOLUTELY NO DIFFERENCE in the search page. Tag clouds are unspammable. And our engine will use tag clouds as its base. Everybody else uses tag sets. And this makes them easily spammable. So, no we don’t fear spammers. In fact we hope that spammers will come to our TagBay site. They are just people trying to sell their stuff, we are trying to make sellers meet with buyers. Wouldn’t be bad to single out spammers just because they are spammers.

TagBay is obviously still in beta, and there are many things that need to be coded. If you have any idea on how to make it better please do not hesitate to contact me. If you want to make a difference on what the final product will be now is the time to do it. Also all new suggestion implemented should be listed in a special page with links to the original suggester home page.

I think it’s the time to present what have I been doing in the last days. A number of improvement have beed added to this web site. In short I have upgraded to wordpress 2.0. I also moved to the next version of wikka. Some of you might remember that I offered some money to whoever could write some code to get the tag plugin to generate an rss list. I didn’t, at the time, explained why. I will now.

Wordpress 2.0 gives the possibility to start categories on the fly. Just adding them, by listing them. Essentially this makes the category in wordpress work like tags (or keywords, for academics). But categories in wordpress also have an rss feed connected to them. Albeit with some bugs, like linking to the whole blog and not to the particular category. So I passed most of the first of January adding to the entries the relative tags as categories. So now I have no need of an rss feed for the tag page, as the tag page has been substituted with the category pages.

You also will rememer that I installed Wikka. The wiki engine. Now wikka is not only open source but also easy source. It is so simple that even I could hack the code. That is very simple! So I changed the code and inserted the possibility to have default pages. In short if before if you were to look for the url http://wiki.pietrosperoni.it/someunexistingpage and there is no page in the wiki called “someunexistingpage” the result would be that the wiki would ask you to edit the page, and you would be redirected to http://wiki.pietrosperoni.it/someunexistingpage/edit.
Now he would create on the fly the page someunexistingpage with the default content. And the default content I chose was the 4 rss feeds:

• the feed from my blog from the category: someunexistingpage
• the feed from my technorati from the tag: someunexistingpage
• the feed from my delicious bookmarks from the tag…
• and the feed from the popular pages in delicious, always from that tag

So for each tag I now have a wiki page with the most relevant rss appearing there. But being a wiki page I also can add other rss feeds, write definitions, comments, todo lists. In short modify it as I see fit.

Still it is not perfect. As it writes the page the first time, from that moment the page is set. I can delete it, but I cannot, for example, change the default content for all the pages that only contain the default content. I tried to write a plugin to do that, but I failed when I confronted the fact that I needed to write a plugin {{defaultpage}} who should have activated other plugins:{{rrs}}, for example. Something that I ignored how to do.
Also having the same string to work for delicious (as tag), wordpress (as category name) and wikka (as pagename) puts some heavy constraints on what the string might contain. For example I am already running ashore for all the tags that contain a dot inside (aaargh, del.icio.us!) or an accented letter (aargh, dear italian).

If you want to see how the pages look like just see the idea page. But any link from the right column (provided they have no dots inside or accents) will work fine.

This is going to be big. It’s called tagsurf. When we were setting up the taoist discussion board, at Tao Bums, I was looking for a board that permitted me to tag individual messages with different tags. The reason is that over there we are now a group of friendly people and every thread start with a topic, but often touches many separate ones. The board had to be in PhP for reasons only knew to the web master, but that we all were happy to follow. So we started looking around, but no board with tagging facility went up. Nothing. I had to admit that the idea was quite new, and I have not seen any such board around in any case. And then we decided for phpBB which being open source would have had new versions with any new cool geeky thing appearing every so often. Well. Now I finally found the first tag based discussion board. It’s called tagsurf. And is very cool. You get to write messages and tag them. As tag you can use any word up to any size. Now, the result of this is that you can tag thing with the url of something. So immediatly a series of utilities started appearing:
People (first one I saw doing it was Russell Beattie) added a tagsurf button. In short if you click on that button you get all the comments on tagsurf that uses your permalink as a tag. In a sense it is outsourcing the discussion board.
Yes, I added it too, is down near the little technoraty bubble, and I just needed to add:
<a href="http://tagsurf.com/post?tag=<?php the_permalink() ?>">Tagsurf this</a>

in the template.

I also went back to see how was tagsurf behaving in del.icio.us. It seem that, as it often appear in other cases, the meme is 6 days old. At the beginning few people noticed it, and now is starting to explode. I too found out because of the delicious discussion board, which I would suggest anybody who is interested to anybody who is interested in delicious OR folksonomy

I think this tagsurf will and can have great impact. They already have some API defined.

I also got an eye to their privacy policy. It seemed simple and clear. Yet now I cannot find it anymore. I suspect that they might be working on it right now.

I also made a small bookmarklet to post an entry on tagsurf about a specific page. Just drag the word bookmarklet on the bar and it should work. Of course for it to work you have to be logged in in tagsurf.

Great points:
• trackback: every post gets is an entry point for trackback. In other words anything you say can receive trackback from anything else. You say something here, and it get people in the blogsphere chatting. And you can follow their conversation. This is something very important that was missing in all the bullettin board I have been using. In a sense many discussion board are only looking in. This is also looking out.
• trackback 2: Every post that you make can send trackback to anything you want. The software to do this automagically respect to the other posts inside tagsurf is still missing, but I can’t imagine it not appearing very soon.
• possibility to mix different threads: since each post gets as many tags as the poster want it is quite easy for people to join different threads of discussion.

Problems I might see coming.
• Spam, spam, spam: I recieve about 30 spam trackbacks a day. And they get filtered by cool programs and finally deleted by me. Yet those programs need me to make the final judgement. Who will make the judgement for all the trackbacks in all those posts? Will the user have to? Can someone close the trackback from his own posts? I see many problem and much discussion over here.
• copyright: This is another big one. Let’s say that I post a cool entry in tagsurf, who gets the copyright of it? It might be important. Imagine that someone takes it, and wants to add some extra tags. But adding tags is not allowed at the moment. So he copies the post and just reposts it with the extra tags. Do I have a say on it?

All together I think this is a wonderful piece of new technology. When tachnoraty started his tag page I wasn’t very impressed, but this, I think, will make some huge effects. And still I can’t see all the implications.

ADDENDUM: just as I ended this post I read fully the great and very interesting post from Russell Beattie. And I found that he had made exactly the same bookmarklet. Oops. Well, I hope he will not sue me, I haven’t copied his code. I just reinvented the wheel.

ADDENDUM to the ADDENDUM: As I was looking at all the people who were commenting on the thread on Russell post I noticed another post with the same bookmarklet. And I thought I would have been the first . At least I get to see if the trackback to posts over there actually works.

ADDENDUM to the ADDENDUMto the ADDENDUM: trackback does not seem to work, or the comment is being held back for security reasons

Did some more debugging. Now any unicode the user used in the tags should be ok. Still there is a big brick wall in terms of memory usage. And some users are not having any luck just out of the fact that their map is taking so much resources that it goes beyond the ISP limit. I could work hard and distribute the whole calculation so that all variables are stored on disk, so the memory would never be hit, but honestly, it is not my top priority at the moment. I am here to help those users run the program on their own machine. And eventually we might solve that problem too. So, what are my top priorities:

• Add an rss feed.I would like to add an rss feed that every time a new map is done, the feed gets updated. It wouldn’t just tell the name but all sorts of data, like the list of the Main Tags. So the users could see if they might be interested in checking the new buddy’s map
• Insert a way for user to delete their own maps. If I am going to go into hosting business, I am not going to be one of those hosts where you can add info, but you cannot delete it. I am aware that users info ultimately is adding value to my site, as such I want users to be happy in having their map here. Not forced.
• Insert a general log of all the maps that are being started, and ended. Right now such a log is absent, and there are about 200 maps completed, and more than twice maps that have been started. So about 300 have been dropped. I bet many of those users would have success, if they tried right now, after those 3 deubugging session. Still I want something that tells me: Warning warning warning, map dropped. Bug? OutOfMemoryError?
• Add the number of posts inside a tag. Just obvious
• Probably add some of the MainTags as keywords to each single map. The problem is: which? All is too much. All the ones that contain more than x posts, y subtags is not flexible enough. The solution should be: if a MainTag is part of a ParetoFront of Delicious than the keyword should be there. The fact that this means writing a whole program that stores in a database the latest ParetoFront is just a small detail . And before you ask: no, I will not need anybody’s password to do that, and the data will all be public.
• Add a bookmarklet to save a map in your own delicious, with the keywords as tags
• Change the map, so that it can run on a single tag. Useful for big complex maps like mine, and others.
• Make it change the Title of the Map Page, to show the owner of the map. Useful if people want to add the maps to their delicious pages.

And then there are some tests I would like to make, like:

• Check if it would make sense to show all the tags that appear with a single tag, and not the subtags.

There is more? If you can think of other modifications , please drop a line in the comment section. Also if you tried to run the map maker and it is not giving you satisfaction let me know. I’ll whip it appropriatly. HarHarHar. (I’ve always wanted to say that!)

Some people (few) were in the unfortunate situation that the tool would calculate their map, and would correctly add it to the make map page, but then the map could not be open. If you were one of those people, I have good news. I tracked down the bug (this time only derived by my stupidity) and nailed it. So, please try again. Insert again your data, calculate it again, and then open it. With this I ended debugging the obvious big errors. If you try now and the tool does not work, please drop me a line with your username, maybe send me by email (available via my homepage) your complete list of all the posts. And I will see what I can do for you. If you don’t contact me I have no way to know that the tool failed, and I will not be able to help you.

I went on programming at my favourite Python program: Delimind.

In short: Made a new release of the Deli Mind program. Here is the source code (just remember to change it from a .txt to a .py). Now similar tags are clustered together.

1. Here is how it looks like.
2. Here is how the previous version looked like.
3. The original from Brownhen (may he live long and prosper) used to be here, although now it is missing.

All on the same data. Mine, now.
Go and enjoy.
(Later addition: while the program works well for small databases of links, like mine at the time in which I wrote this entry, it doesn’t scale well on size. For this reason it crashes for most of the people who try to use it with more than 1000 bookmarks. For this reason I was forced to change the link on the cluster example to a database with fewer nodes.)

Now the tecnical stuff for those that have a bit more patience.

Tags are not all the same, some are more similar than others. So, for example, the tag “September11″ and “GeorgeBush” have more links in common than “GeorgeBush” and “intelligence”. The idea behind this version of DeliMind was to cluster tags that had links in common. Since distance is generally not a transitive property (if I am near to you, and you are near to Jim, I am not necessarily that near to Jim), while clustering is (if I and you are in the same cluster, and you and Jim are in the same cluster, then me and Jim have to be in the same cluster… unless people belong to different clusters, but that’s a complication).

So I started by making a matrix of relations among tags (all_dict). Each tag, respect to each other tag could either be

1. Once contained in the other
2. Identical
3. Disjointed
4. With # bookmarks in common

Then according to the number of links each of the two tags, and the number of links in common I invented a measure of similarity. If #A is the number of links in tag A, and #B is the number of links in tag B, and #AB is the number of links in common.
The the relative similarity (SAB) will be:
SAB= sqrt((#AB/#A)*(#AB/#B))

I actually played with various measures:
SAB= ((#AB/#A)+(#AB/#B))/2
SAB= Max(#AB/#A,#AB/#B)
They all went from 0 to 1, and were quite similar… (I am not going to discuss the relative properties)
But the first one just seemed the one that made more sense, and at the end, the resulting map was the one more close to my personal intuition of what should be in what cluster.

Once the similarity matrix was done I started studying the clusters. Generally for each triplet of tags A, B, C I would modify
SAC:=min (previous SAC, max (SAB, SBC))
And I would continue going through all possible triplets, and then starting again from the beginning until no new change were happening.

Why? The idea is that the similarity between two tags measure how easy it is to jump from one to the other. Visualise each tag as an island, and then you have an animal who can jump from one island to the other. But it can only jump up to a certain distance. So if he can find a succession of tags between two tags, A and B, where the similarity (the similarity is the inverse of the distance) is always above its jumping ability (that is, the distance is below its jumping ability), then the animal can move from A to B. If not A and B are in different clusters. Effectively unreachable.

But we don’t know how far can our beast jump. So in this way we end up having a similarity number that sais: somwhere, between A and B is possible to find a succession of tags, such that the distance is never above x, so SAB is equal to the minimum between the original SAB and x.

If it does feel complicated don’t worry. I got confused a few (hundred) times programming it. And just could not understand why those damn tags were not clustering… until I got it right.

So, now you have this nice matrix, only between your main tags (the one that are not contained in another tag, cfr previous version), and you (or actually I) need to cluster the tags.

Not also that you don’t need to cluster the tags only one time. Once you made a clustering (for animal which can jump d), you can still partition inside the clustering for animals that can jump less than d.
The first time I just asked him to cluster each possible number. That is, if a number was present assume that someone was able to jump exactly that distance. In this way I got a heavily clustered map. It was a mess, but a promising mess. I then saw that most of the interestign things were happening between distances of 0.333333 and 0.6666.

That is, it made quite sense to ask for the clusters generated by putting together tags that had one third of the links in common, and tags that had up to two third of the links in common.

This is how I got clusters:

• porno, sex and eros
• GeorgeBush, September11, politics, economy, historical, terrorism, usa
• green, sustainability
• …

Then I just applied the same process in the subtags of each tag.

Ok, I can be satisfied, I can go and have something to eat.

As always, if you find it useful drop me a line, I appreciate.

Pietro

So, I just modified the deli.mind script, originally from brownhen.
The original would take the public bookmark from delicious and make a free mind map out of them.

(For those who have no time to read the whole post, I immediatly tell you that I modified the code. The new code can be found here, and an example is here -open some nodes to see the difference!-).

The program is written in python, and I wasn’t very happy with the result. I mean it was great to have the map, but at the same time I have so many tags, that it was pretty much useless. Now the fact is that we tend to reuse tags that we have already used. This generates a positive feedback dynamic, that tends to create a bunch of very common tags (even among your own tags) and many many tags used only one or two times. I bet you could also plot them into a nice power law picture (but, alas, you need at least 1000 tags, to make it statistically meaningful!). This is generally true, but is particularly true for people who, like me, tend to store each link with around 10 different tags. This means that this long list of tags, that was using up my screen, was mainly composed of completely unimportant tags, with only few interesting among them.

Not only this, but some tags, tend to appear only in conjunction with other tags. For example, the tag “python” comes always with the tag “programming”. In a sense it is a “sub tag”.

Oops, are we back into hierarchy, aren’t we?

Well, not exactly, first the same link can be present in different non hierachically related tags, and second two tags can have links in common, but not be completely hierarchically related (think about the tag ‘September11′ and ‘GeorgeBush’ as a good example). The last thing to note is that from time to time there are tags which have exactly the same links inside, either because they are synonimous (’del.icio.us’ and ‘delicious’ for example) or because I had not stored enough links to differentiate between the two.

So the new program extracts the information about the relation among the tags, and uses it to build a more interesting mind map.

More precisly two tags can be:

• Identical,
• One inside the other,
• Viceversa,
• With a non empty intersection, but with some extra links,
• Completely disjointed.

This information is then used to create the new mind map.

With the following novelties:

• Sub tags are shown as a sub branch of their parent tag.
• Tags that are equivalent are shown together with a little empty branch as their parent, to connect them all.
• A sub tag can be sub tag to more than one tag.
• Each tag also is followed by two numbers: # of links & # of sub tags.
  So you have an idea about how big is the tree you are going to explore.

You can see my “hierarchical delicious free mind map” in java format here while the code is here.

I also fixed a couple of bugs. That would give some fake results. (i.e. being tagged as ’socialsoftware’ does not mean being tagged as ‘war’, etc…)

This isn’t the end, I am planning to work on this some more, when I have time.

Disclaimer: This was also my first tentative hack in python. So I am sure I did plenty of things in a clumsy, slow and redundant way. But I am learning.

Acknowledgment: I am very grateful to brownhen., because if he didn’t release the first version of the script I would not have started at all.