Gandalf fighting the Balrog
Twingly is having trouble with data collection. A database server was brought down in a controlled manner to change a faulty memory stick but turned out to not play nicely when it was time to bring it up again. A raid partition problem combined forces with human error to kill the server database on the server completely.
Our full team of technicians are working around the clock to bring indexing back to life again. No data loss have occured, the problems are temporary and will be solved. Twingly.com and Twingly Blogstream remain functional except for the fact that new data is currently not indexed. Twingly Blogstream administration is also affected.
Once indexing is online again the backlog of pings will be processed and order restored. We’re very sorry, please bear with us.
UPDATE: The exact problem is that a file belonging to the database running on the faulting server existed only on the crashed raid set and the database refuses to start without it. The missing file doesn’t contain vital data, but it is required by the database and our efforts to recreate the file have so far been without result. The database is a MS SQLServer unfortunately still taking part in our indexing for legacy reasons. Blog data passes it before being sharded into MySQL.
UPDATE 2: Bad news are now worse. Gandalf (the server in question) has been pronounced dead for the time being. Data recovery service is an option, but shortest path to restoring service is now to reroute data around Gandalf straight to MySQL. This ups the complexity plenty and will take several long hours, we are looking at getting fresh data in again some time during the afternoon tomorrow (thursday). Stay tuned for updates right here and on twitter.
UPDATE 3: The admin system for our customers is up and running again. We are also getting in new posts in our index, which means fresh posts both in our search and on the web pages of our customers. This is currently done via a route around Gandalf, as mentioned above. The pings are now being processed, but since there is a few of them in the queue at the moment there will be some delays if you ping us at the moment. Stay tuned for more updates. /Kristoffer
Hardcore
*big bubble no trubble*
Praying for a good surf!
Phuu….
I thought i was banned, this was “fantastic” news…
Thanks for letting us know
Good Luck – allt ordnar sig. Alltid.
Hej, jag förstår frustrationen och ber djupt om ursäkt. Det vore lätt att skylla på andra faktorer, men till syvende och sist är det vi misslyckats med att uppfylla förväntningarna. Det är väldigt tråkigt att det inträffade samma dag som ni lanserade.
Vi jobbar fortfarande med att få igång indexeringen, uppdateringar kommer i bloggen.
Vänligen,
Martin Källström
VD Twingly
This comment was originally posted on http://tekniskbeta.no/)” rel=”nofollow”>Teknisk Beta
Twingly er fortsatt nede, og det er ekstra kjipt at det skjedde i dag. http://tekniskbeta.no/twingly-gikk-ned/ (oppdatert m/kommentar)
This comment was originally posted on http://tekniskbeta.no/)” rel=”nofollow”>Teknisk Beta
Hej, tänkte bara uppdatera er på att problemen börjar lösa sig och att det förhoppningsvis kommer börja funka igen under eftermiddagen.
Återigen, vi är väldigt ledsna över detta och ska göra allt vi kan för att det inte ska hända igen.
Hör gärna av dig om du har frågor eller undrar något.
/Anton Johansson
anton@twingly.com
skype: antjoh766
This comment was originally posted on http://tekniskbeta.no/)” rel=”nofollow”>Teknisk Beta
@Anton
Takk for oppdateringen! Som jeg skriver, lover det godt for tiden fremover selv om det er litt kjipt i dag.
This comment was originally posted on http://tekniskbeta.no/)” rel=”nofollow”>Teknisk Beta