Six Weeks, Four Interns, One mission: Push the edge of Blog Search

At Twingly we continously try to redefine what we do and the way we do it. This summer we are redefining what it means to be a summer intern: Be the Rockstar, Not the Peon.

Mission statement: You and your fellow team mates are to embark on an epic mission. During six weeks, you are the focal point of our company. You get the best tools, the best technology, the best knowledge and a huge pile of blog data to play with.

Your objective is to push the edge of blog search into what was previously thought impossible. Think bigger, do better. Bring the entire breadth of the global blogosphere into one interface, placing any blog in the world a mouse click away. Allow the user to explore a universe rather than scroll a list of result. Make it visual, make it interactive. Bring the user to amazing discoveries.

Oh, and you get six weeks. You better start right away.

Regards,
Martin Källström
CEO Twingly

Follow the project on http://summer.twingly.com/. Starting from today, there will be a daily YouTube episode posted by the team at 6 PM GMT (10 AM in San Francisco, 19:00 in Stockholm).

Twingly Summer of Code Team 2008

Twingly launching out of Beta with new widgets for bloggers

All systems nominal. We are go for launch!

This morning, Twingly is live to the public audience. I’m proud and awed by the effort put down by our team up to this point, as well as a good darn grateful to all the beta testers that have provided feedback during the beta period. More than 3500 people have participated.

Twingly.com now covers over 30 million blogs all over the world but we focus on being number one in Europe, both regarding spam control and working with several different languages.

Since we are going public we are finally able to make our widget platform available for bloggers. You can create a widget out of any search result or pick one from the new blog profile page, see for example the profile for Engadget. You access the profile page trough the search result or by hacking the url. Find the widgets from your own blog or use a widget for another blog if you like. You can actually hack any search query into the widget, which is truly powerful.

But this is only the beginning as we’ll make more and more kinds of widgets available. So please participate in working out the Tech plan!

BTW, there is some functionality that is only provided to registered users, but the registration is completely painless and instantaneous. More specifically:

– Participating in the Tech plan
– Voting up posts in the search result (liking posts)
– Sending feedback (so you wont have to enter your contact details every time)

The user registration will be the basis of a lot of things to come, claiming your blog will allow you to control how it is presented on Twingly. Please note that using the widgets does not require any kind of registration!

Key functionality on Twingly.com:

– Spam free search
– Social search. The users enhance the search results by voting on posts they like. Bloggers enhance the search results by linking to posts they like
– Subscribe to search results by RSS and alerts via email
– Language functionality: Translation of search results and filtering based on language
– Twingly widget platform. Parts of Twingly.com can be incorporated into blogs
– Hot Right Now. Overview on hot topics in the blogosphere
– User directed development through a tech plan open for voting.

All in all, thanks to all that have participated in the great effort leading up to this point.

Now, lets get to work with the next big thing in blog search.

Twingly Meltdown Report

Hours

Lots and lots of users gathered to bring our servers to their knees in the first Twingly Meltdown. Our tech team were quite sure that a group of regular users couldn’t do any damage to our servers. So towards the end of the Meltdown Hour I shared a small script with everyone on the Twingly Skype Chat that made it possible for all users to send thousands of simultaneous requests.

Before people started using the script, execution went flawless. Our database servers who are protected by caching did hardly show any load at all. On the web fronts the traffic produced visible CPU load. Some users in faraway countries experienced periods of slow searches. Overall, search was fast.

When the more aggressive load script was deployed by lots of users, the tune turned a little more hostile in the tech team chat. Since the script used the undocumented and far less tested json API, it managed to bring out a bug in the code. The web servers are put under much more strain when they have to deal with errors in execution. Technically, this is the difference between performance testing and stress testing. So this actually affected the experience of other users, who at times got an error message while searching.

Big thank you to all participants, hope you had a lot of fun trying to crash Twingly. It helped a lot to see how the system behaves under strain, real world traffic is really hard to simulate properly.

Top image is the web log stats from the day of the event. Clearly visible is the fact that by the end of the event, a lot less users managed to send a lot more searches.