In other news, lack of WiFi at my hotel encouraged me to walk around downtown SF. I should have gone earlier and rented a bike (shop closes at 6pm.)

The Web is constantly turning out new and extraordinary services many of us are unfamiliar with. During TED University at this spring’s TED2007 in Monterey, Julius Wiedemann, editor in charge at Taschen GmbH, offered an ultra-fast-moving ride through sites in many different areas, from art, design and illustration, to daily news, blogs and curiosity. Now, by popular demand, here’s his list of 100 websites you should know and use.

• Google lists the post on my favorite wordpress plugins as the second hit for my name.

• Spammers target this highly visible post for all the spam (about 90% on just this post.) Fortunately, akismet blocks almost all of them.

• I have a plugin installed that organizes the dates in my sitemap by the most recent updates. The amount of spam causes the last modified date on the post to be updated regularly.

• Google uses this information to repeatedly crawl the aforementioned post, thinking it’s a highly active page.

• Positive feedback ensues.

You can find tons of styling with CSS here.

Most sites use CAPTCHA to determine legitimate users. A friend recently told me about a new method that spammers are using to circumvent CAPTCHA. A site with high demand content (like free music or porn) harvests CAPCTCHA on demand from another site and displays it to a user. The user is able to decode the text and types in the correct text, which in turn is used to spam the site where the CAPTCHA is taken from.

This is incredible.

A more useful version of this technique is reCAPTCHA. Instead of posting spam, the user is presented with words from a book that is currently being digitized, some of it correctly recognized by software and some incorrectly. This is compared with what others have typed in, and if correct, your comment is posted and the words added back to the database.

I’m trying to figure out if the evil way or the good way was thought of first. I’m guessing evil.

The Web space as of now is pretty crowed and it’s difficult to differentiate the wheat from the chaff. The next social network, or the next youtube clone isn’t very interesting. Skimming through these papers brings to light some of the harder problems in the field:

• spammers posting portions of existing comments with a few links changed.
• a website that has the authority in one area might start writing about other things. How do you assign a pagerank to this site?
• RSS aggregators clustering. No fun reading about five different takes on the same breaking news. Much like Google News.
• Google news itself. “collaborative filtering using MinHash clustering, Probabilistic Latent Semantic Indexing (PLSI), and covisitation counts.”
• mining and clustering other kinds of information (like chemical formulae)
• advertisements and click fraud
• personalization
• and a whole bunch of stuff on scalability, privacy, security and the semantic web

Also speaking at the event is Prabhakar Raghavan. I have previously heard him talk about the convergence of the social sciences and the web. Awesome speaker.

Eigenfactor.org is a non-commercial academic research project sponsored by the Bergstrom lab in the Department of Biology at the University of Washington. We aim to develop novel methods for evaluating the influence of scholarly periodicals and for mapping the structure of academic research. We are committed to sharing our findings with interested members of the public, including librarians, journal editors, publishers, and authors of scholarly articles.

The ranking is done using an algorithm similar to Google’s PageRank. Read about the other features on their “Why EigenRank?” page.

Here for example is a listing of the highest ranking (by eigenfactor) journals in Mathematical Physics:

1. PHYSICAL REVIEW E
2. JOURNAL OF PHYSICS A-MATHEMATICAL AND GE
3. COMMUNICATIONS IN MATHEMATICAL PHYSICS
4. JOURNAL OF COMPUTATIONAL PHYSICS
5. JOURNAL OF MATHEMATICAL PHYSICS
6. PHYSICA D-NONLINEAR PHENOMENA
7. JOURNAL OF STATISTICAL PHYSICS
8. COMPUTER PHYSICS COMMUNICATIONS
9. MODERN PHYSICS LETTERS A
10. CHAOS

I’m yet to figure out a way to make use of this service effectively. My day job deals with Medical Imaging, which is a form of Computer Vision, which in turn falls under Artificial Intelligence. Computational Anatomy, one of the emerging subfields of medical imaging is fairly mathematical, using ideas from group theory and functional analysis. To add more to the mix, one of the most used algorithms for image/volume segmentation is the Level Set Method, first published in the Journal of Computational Physics. I can’t judge a paper just by looking at where it’s published, can I?

I think there’s a huge potential for programs to analyze voluminous amounts of data. Most data analysis is done either using Excel, or using command line tools such as awk and sed. A smart way to distribute the post-processing or analysis would be cool.