SourceLair Blog

  • Introducing partial commits

    Today we are happy to announce support for partial commits on both Git and Mercurial projects.

    Simply make any changes you want and open up the commit prompt. You can now choose which of the files in the list you want to include in your commit.

    Partial commits from SourceLair

    Visit www.sourcelair.com and try it out.

  • PHP projects on SourceLair

    Last year we introduced PHP functionality to SourceLair's "HTML" project type. While our PHP support worked really well for our users, it was not clear enough that coding your PHP app should happen in an "HTML/PHP" project.

    Thus, we decided to branch-out PHP to its own project type. HTML projects will continue to support PHP, so that people that already code PHP apps on SourceLair won't have to take any further actions.

    PHP projects on SourceLair

    Coding PHP projects on SourceLair works like a charm. Every PHP project is equipped with a public URL, you can install your favorite libraries with composer (which ships pre-installed in your SourceLair project), while you can enjoy several pre-installed PHP extensions like php5-mysql, php5-sqlite and php5-curl.

    Check out our new PHP project type right away at www.sourcelair.com.

  • Ceryx - A dynamic NGINX

    Reverse proxying hundreds, or even thousands of contained micro-services is an interesting problem and one that we face daily at Sourcelair. That's why, today, we're glad to announce Ceryx, a dynamic reverse proxy using OpenResty, Lua and Flask that can be used to proxy hosts to any number of services, with it's configuration being available instantly. Ceryx is a project we've been working on the last couple of months and we're open sourcing now.

    We love NGINX

    Abstract

    In SourceLair, we're quickly provisioning dev environments and try to make web development frictionless, leveraging the power of the cloud. One of the nice goodies we provide, is a public URL for each project, that is always available and auto refreshes with your newest code. Thus, we need to start and stop multiple user containers per hour and route each user's public URL to the correct container without downtime.

    Previous solutions

    Ceryx is an internal project that we've being developing over the past year. We have experimented with different technologies and stacks in order to find the best solution for our case. Throughout the way, we've kept the same API making these changes seamless - that's a nice topic that we won't cover here though, you can find my slides from the API Meetup Athens.

    Twisted, MongoDB and Redis

    At first, we started with a custom reverse proxy, based on Twisted. Twisted is a very nice, event-driven networking engine written in Python. The service was querying MongoDB for routes, if it did not find one in its Redis cache. The cache was populated either by a database query hit, or an API call to add, update or invalidate a route. This was working nice for us, was fast and Twisted had a nice reverse proxy API to work with. The things that we missed from this implementation was that by default Twisted did not have every reverse proxy header you would expect set up and this would lead to invalid redirects in several cases. It was great fun working with it though!

    tproxy and Redis

    After Twisted, we looked towards tproxy. tproxy is a TCP routing proxy (layer 7) built on Gevent by the creator of the famous Gunicorn server and heavily inspired by Ruby's ProxyMachine. We created a lookup layer, that instead of static routes or a file, it would lookup the route at a Redis backend. We had completely split the service from MongoDB at that point, since the routes are ephemeral and can be easily recreated. Also, we split out the API to a separate service using Flask. The main issues were that tproxy development was a bit abandoned - last commit is over a year ago and that we would need to rework several bits and pieces for optimal performance. Also, there was an interesting bug with responses that did not contain a response length, which would be kept open until timed out.

    NGINX and etcd

    We have decided that we would leave the proxying to a server that did it well out of the box, so we looked at NGINX and HAProxy. Since we were more familiar with the former and happy with its performance - all of our front facing servers are NGINXs - we went with that. We created a watch script, which would watch for key changes in etcd and reload the configuration of the NGINX. Also, we changed our API to work with etcd as its backend. This dramatically improved performance, but after several weeks we noticed that the configuration change was not as fast as we thought. NGINX reload returns almost instantly, but the configuration is applied to accepting workers over time, which resulted in slow updates in the routes. The most important issue for us, was that sometimes this reload took over 10 seconds and led to our users seeing the "Server is hibernated" page multiple times until the new configuration was applied.

    OpenResty and Lua to the rescue

    Recently, Github blogged about them using Lua scripting in NGINX to make reloading of Github pages more often - previously they did so every 30 minutes or so.
    As we were looking for an alternative and intrigued by the post, we started looking deeper in OpenResty - which is an NGINX flavor, compiled with the Lua JIT and other 3rd-party NGINX modules.

    We decided to go back to Redis as the backend, since we already had the API ready and Redis is in-memory, thus pretty fast in querying. We also use Redis for other services and caching, so we would not need another cluster to take care of.

    The result was Ceryx, which is now Open Source and available to every one. This contains both the NGINX lua scripts and the API and can be easily deployed using Docker Compose.

    Stitching everything together

    NGINX provides several hooks for Lua code to be executed in several stages of a request. Ceryx takes action just before the proxying stage - at the "access_by_lua_file" stage, where it calls a Lua router.

    The router then queries the local in-memory cache of NGINX and the Redis backend, to determine the target host and port of the incoming host, or returns a wildcard target if not found. If a Redis query returns a result, then this result is cached for 5 seconds, so that subsequent requests do not hit Redis - this is a nice improvement for when static files like CSS, JavaScript and images are needed, where multiple requests are being thrown at the same time. The caching timeout can be increased, in order to easily tailor Ceryx to each application's needs.

    At the same time, a simple Flask service is available, which provides a CRUD API for routes and updates the Redis backend with new routes. Both the proxy and the API service share the same environment variables for configuration of Redis for consistency.

    First impressions

    After the first weeks of the last Ceryx flavor, we've seen great improvement in the speed of our reverse proxy and a big reduction in the times a user would see the hibernated server page. In more detail, before the upgrade of Ceryx we had an average of 10 page views per development session, while now we have only 2.5.

    Next steps

    Ceryx is Open Source under the MIT license, so we'll be glad to see contributions or bug/feature requests. What we're planning on doing is add StatsD metrics to Ceryx, so that we can further improve and optimize several bits and pieces accordingly.

    Looking forward to your ideas and contributions at the Ceryx Github project

  • Hello Node.js

    JavaScript - and Node.js in particular - is getting a lot of attention lately. Be it the nice threading model, the ease of adoption or the ability to have full stack developers just coding against JavaScript, Node.js is here to stay.

    Removing friction from your development workflow is our first priority and we couldn't ignore the fact that you might be using different frameworks to code.

    That's why, today onwards SourceLair sports Node.js out of the box. We did our best to make Node.js support framework agnostic, so that you can easily code on your favorite framework with the least effort. You can configure your workspace accordingly and enjoy the awesome SourceLair sauce.

    Bootstrapping at a glance

    Bootstrapping your Node.js project is as easy as counting to 4 - the number clicks needed to create it. When you create your project in SourceLair, by default you'll have an Express.js project up and running. If you want to install a different framework, like Sails.js, koa, hapi, etc, you can easily install it using our npm integration.

    Using your scripts at SourceLair

    SourceLair does not only support Node.js, it understands it. When you code your project in SourceLair, the scripts section in your package.json file will be populated automatically in Command Palette. Then, you'll be able to open Command Palette using Ctrl + Shift + P (or Cmd + Shift + P on a Mac), type "scripts" to list the all or your script's name and then run it. It's that easy!

    Node.js support in SourceLair

    What are you waiting for? Your next Node.js project awaits you at www.sourcelair.com, move on and hack the heck out of it!

  • Introducing MySQL servers

    Every modern web application is backed by at least one database and thus, databases are components that are tightly coupled with every developer's workflow.

    This is a friction that we want to take away from every developer and today we're stunned to introduce MySQL database support in SourceLair. MySQL is one of the most common databases out in the wild and is one more step towards providing our users with the tools they need for building great applications.

    MySQL Databases in SourceLair

    Bootstrapping a database

    Starting today, each time you create a new project in SourceLair, you can select to connect a MySQL database to your project. Seconds later when your project is created, you can easily access your database from our fully featured Linux Terminal or directly from your web application and enjoy your secure, isolated database.

    If you already have a project in SourceLair, you can of course take advantage of this great new feature without the need to create a new project. Just head to your project settings and select "Connect a database" option. Your database will be up and running in a glance.

    Security

    Your database user and password is predefined, but you can easily change them by connecting to your database. Apart from that, we have made sure you have one more layer of security by isolating your database and making it available to your project only. This means that neither another SourceLair user or another computer can directly hit your database, ensuring that your data will be safe and making you comfortable to test your applications with any dataset you want.

    Availability

    Databases are available to everyone. You don't need to have a SourceLair Pro subscription to create a database and since we're developers too, we know that we don't buy unless we taste.

    That's why you can easily create your database for free, experiment with it for a month and get billed afterwards only if you like it - no strings attached!

    Enough said, just head to www.sourcelair.com, create and enjoy your new MySQL database.