SourceLair Blog

  • Post-mortem: Why several users got spammed with emails

    Recently, we switched our Celery broker to Redis. This was a great improvement in the way we handle tasks, but there was a caveat which we did not forecast.

    This caveat of the Redis broker led to the follow up emails we send to our users 4 days after their subscription being sent multiple times.

    We have tracked down the issue and changed the way we send emails, but this does not minimize the number of emails several users received.
    We are deeply sorry about this incident and we have taken measures so that it does not happen again.

    The issue

    When using the Redis Celery broker, tasks that do not get executed some time after they are received, they are rescheduled - reference.

    Copying from the Redis documentation:

    If a task is not acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

    Since tasks were being scheduled 96 hours after a user was created, there was a theoretical limit of 95 emails being sent to a user, since the email was being rescheduled every one hour.

    The good fact is that we deploy new code several times a week and the duplicate tasks were being rescheduled only once after each deployment, crucially reducing the actual amount of emails that were delivered to a user. The most emails sent to one user were 87, which was during the weekend.

    How we solved the issue

    When we became aware of the issue, we immediately canceled pending emails and started working on an alternative way to tackle the issue.

    We have created a daily cron task that collects all users that were created 4 days ago and dispatches the emails, all done using Celery beat.
    This improves performance and makes sure duplicate emails will not be sent again.

    Thoughts for future improvements

    We have solved the case for now, but spamming is one of our worst nightmares and definitely something we don't want to come from our service. Thus, we have started working on some thoughts on how to effectively rate limit emails to users, so that incidents like this one never happen again.
    The first thought is limiting the amount of emails we can send to a user during a specific period of time and add alerts to our administration team when this limit is reached.

    We are constantly doing our best to create an awesome service for you and we give our word that we'll continue to do so.

    Be creative and enjoy coding in your browser.

  • The colours of productivity

    Here at SourceLair we are striving to make our users' lives easier and we are always trying to find ways to enable their productivity.

    For these reasons, we have been doing some internal research and testing for some time now, and one of the things we found out, is that the human brain responds very well to vibrant stimuli, in a way that promotes productivity and relaxation.

    Today we are happy to introduce the result of our research efforts.

    Hot Dog Stand is here

    Inspired by the all time classic Hot Dog Stand theme of Windows 3.1 this is a very minimal theme, easy to the eye, that is sure to boost your productivity and make you reach your true potential.

    So, head over to www.sourcelair.com open your editor settings, or go instantly to the Theme selection by opening the Command Palette (Ctrl/Cmd + Shift + P) and entering the command "Quicksettings", and try it out for yourself, today!

  • Read between the words

    Today we are happy to announce a new little convenience feature for our editor that allows you to view whitespaces in your code.

    To enable the feature just go into your editor settings and toggle the switch.

    Whitespace settings

    When you return to your code you should now notice that every whitespace such as spaces, tab characters and line breaks are appropriately marked and displayed to you.

    Visible whitespace

    So, go to www.sourcelair.com and check it out yourself in seconds.

  • Scheduled Downtime on March 28 - Hardware Upgrade

    Hardware plays a critical role in your SourceLair experience. That's why we always make sure you have blazing fast hardware.

    Tomorrow, Sunday March 29, we'll be having a small downtime at 21:00 UTC and for a maximum of one hour. We're moving our servers to a new datacenter and we're also upgrading the specs at the same time in order to meet you needs.
    We'll post updates on the status of the migration at our Twitter account, so make sure you follow us there to be up to date.

    The good news don't end here. We're also glad to announce that we are joining the "Rackspace Startups" program, which is giving us the oppurtinity to scale seamlessly and provide you with state-of-the-art hardware.

    Stay in touch and don't hesitate to get back to us. We love hearing from you.

  • Partnering with the National Bank of Greece

    Today we are excited to announce that we have raised € 200,000 of equity funding from the National Bank of Greece.

    When we started SourceLair we imagined a world where coding would be frictionless and available to everyone and towards that vision we're going to primarily invest our funds.

    The main two pillars in which we will invest is people and infrastructure.

    We are growing our team

    The needs of our users are growing steadily and our team will follow this course. In the next days we will open up job positions in order to ensure the fast release cycles of SourceLair, along with introduction of more features and bug resolution in even less time.

    We are upgrading our infrastructure

    We value the reliability and responsiveness of our service, thus we'll be upgrading our hardware in order to better serve the increasing needs of our users.

    We can't wait to share with you what we 've been working on lately and what is coming up next! Stay tuned!