Trouble at the hard end of the ware

Gandalf fighting the Balrog
Gandalf fighting the Balrog

Twingly is having trouble with data collection. A database server was brought down in a controlled manner to change a faulty memory stick but turned out to not play nicely when it was time to bring it up again. A raid partition problem combined forces with human error to kill the server database on the server completely.

Our full team of technicians are working around the clock to bring indexing back to life again. No data loss have occured, the problems are temporary and will be solved. Twingly.com and Twingly Blogstream remain functional except for the fact that new data is currently not indexed. Twingly Blogstream administration is also affected.

Once indexing is online again the backlog of pings will be processed and order restored. We’re very sorry, please bear with us.

UPDATE: The exact problem is that a file belonging to the database running on the faulting server existed only on the crashed raid set and the database refuses to start without it. The missing file doesn’t contain vital data, but it is required by the database and our efforts to recreate the file have so far been without result. The database is a MS SQLServer unfortunately still taking part in our indexing for legacy reasons. Blog data passes it before being sharded into MySQL.

UPDATE 2: Bad news are now worse. Gandalf (the server in question) has been pronounced dead for the time being. Data recovery service is an option, but shortest path to restoring service is now to reroute data around Gandalf straight to MySQL. This ups the complexity plenty and will take several long hours, we are looking at getting fresh data in again some time during the afternoon tomorrow (thursday). Stay tuned for updates right here and on twitter.

UPDATE 3: The admin system for our customers is up and running again. We are also getting in new posts in our index, which means fresh posts both in our search and on the web pages of our customers. This is currently done via a route around Gandalf, as mentioned above. The pings are now being processed, but since there is a few of them in the queue at the moment there will be some delays if you ping us at the moment. Stay tuned for more updates. /Kristoffer

8 thoughts on “Trouble at the hard end of the ware

  1. Rakel, o February 11, 2009 / 20:21

    Hardcore

    *big bubble no trubble*

    Praying for a good surf!

  2. Rage February 11, 2009 / 21:46

    Phuu….

    I thought i was banned, this was “fantastic” news…

    Thanks for letting us know 🙂

  3. Mary February 11, 2009 / 22:57

    Good Luck – allt ordnar sig. Alltid.

  4. Martin KÀllström February 11, 2009 / 23:41

    Hej, jag förstÄr frustrationen och ber djupt om ursÀkt. Det vore lÀtt att skylla pÄ andra faktorer, men till syvende och sist Àr det vi misslyckats med att uppfylla förvÀntningarna. Det Àr vÀldigt trÄkigt att det intrÀffade samma dag som ni lanserade.
    Vi jobbar fortfarande med att fÄ igÄng indexeringen, uppdateringar kommer i bloggen.
    VĂ€nligen,
    Martin KÀllström
    VD Twingly
    This comment was originally posted on Teknisk Beta

  5. Anton February 12, 2009 / 12:01

    Hej, tÀnkte bara uppdatera er pÄ att problemen börjar lösa sig och att det förhoppningsvis kommer börja funka igen under eftermiddagen.
    Återigen, vi Ă€r vĂ€ldigt ledsna över detta och ska göra allt vi kan för att det inte ska hĂ€nda igen.
    Hör gÀrna av dig om du har frÄgor eller undrar nÄgot.
    /Anton Johansson
    anton@twingly.com
    skype: antjoh766
    This comment was originally posted on Teknisk Beta

  6. Anders Brenna February 12, 2009 / 13:52

    @Anton
    Takk for oppdateringen! Som jeg skriver, lover det godt for tiden fremover selv om det er litt kjipt i dag.
    This comment was originally posted on Teknisk Beta

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s