Twingly is having trouble with data collection. A database server was brought down in a controlled manner to change a faulty memory stick but turned out to not play nicely when it was time to bring it up again. A raid partition problem combined forces with human error to kill the server database on the server completely.
Our full team of technicians are working around the clock to bring indexing back to life again. No data loss have occured, the problems are temporary and will be solved. Twingly.com and Twingly Blogstream remain functional except for the fact that new data is currently not indexed. Twingly Blogstream administration is also affected.
Once indexing is online again the backlog of pings will be processed and order restored. We’re very sorry, please bear with us.
UPDATE: The exact problem is that a file belonging to the database running on the faulting server existed only on the crashed raid set and the database refuses to start without it. The missing file doesn’t contain vital data, but it is required by the database and our efforts to recreate the file have so far been without result. The database is a MS SQLServer unfortunately still taking part in our indexing for legacy reasons. Blog data passes it before being sharded into MySQL.
UPDATE 2: Bad news are now worse. Gandalf (the server in question) has been pronounced dead for the time being. Data recovery service is an option, but shortest path to restoring service is now to reroute data around Gandalf straight to MySQL. This ups the complexity plenty and will take several long hours, we are looking at getting fresh data in again some time during the afternoon tomorrow (thursday). Stay tuned for updates right here and on twitter.
UPDATE 3: The admin system for our customers is up and running again. We are also getting in new posts in our index, which means fresh posts both in our search and on the web pages of our customers. This is currently done via a route around Gandalf, as mentioned above. The pings are now being processed, but since there is a few of them in the queue at the moment there will be some delays if you ping us at the moment. Stay tuned for more updates. /Kristoffer