Continuous Delivery

Hornbill's platform and the applications that run on it are designed from the ground up to support Continuous Delivery for software releases. Our platform and applications are continuously and automatically updated to ensure that all of our customers always have the latest features and fixes, this is done completely automatically without any service down time and we guarantee that all customisations and integrations will continue to work.

Overview

There are many ways to deploy software but at Hornbill we deploy continuously to remove the need for our customers to ever have to perform an upgrade again. Doing a software upgrade often means a project that will include planning, evaluating, testing, training and roll-out, this approach is really a relic of the 80's that was born out of the fact that software is complicated and users of the software were often expected to have some degree of technical expertise. In todays world, mass adoption of complex software systems has become commonplace where users no longer need the technical expertise they once did. The idea then is software is delivered As-A-Service, and just like other services you take (for example your home cable TV service) you simply consume the service. From time to time your service provider will update your service, update your set top box with new firmware, add or remove channels etc. As the consumer of the service you don't care what version you are on, but you are happy to avail of the new channels so long as your service continues to work. Another example might be Facebook, if you use Facebook (or Gmail or LinkedIn or Twitter etc) and I ask you the question "what version are you on?" I would expect your answer to be "I have no idea" simply because you don't need to know, or care, you just expect the service to work when you log in. Thats how we think about Hornbill, and we believe all software should be deployed and consumed like this so thats the foundation upon which we built our platform.

The Quality of Our Service is our Reputation

Just because we do Continuous Delivery does not mean we don't have a formal process, in fact, complete opposite is true. Our delivery process is embodied in our fabric and our very culture and runs right through the core of everything we do. There are a number of key characteristics to the way we work that make Continuous Delivery possible.

• Hornbill is a loose coupled, layered and a highly distributed architecture.
• Hornbill is made up of a large number of independent micro-services.
• Hornbill has 100% automated functional test coverage for functions used in production for all micro-services.
• Hornbill includes fast regression tests for elements (such as user experience) where it is not possible (or cost effective) to automate.
• Hornbill has a fully automated deployment pipeline, all packaging and deployment is done without human intervention.
• Hornbill is organised into a small development teams who assume responsibility for feature and fix deployment.

Hornbill Update Deployment Process

When we write code for a fix or a minor feature/enhancement we follow a strict methodology to facilitate both agility and continuous deployment. Each time we do a build, which are themselves fully automated, we have four stages of release before code makes it into production. These are known as test, dev, beta and live and every push of code has to transition through these four stages. Code can be blocked from progressing at any stage and for any reason.

• test - when we initiate a build the code is compiled and the releasable content is created and published into our deployment system on the test stream. All of our automated test services are sitting on this stream, so when a new build is available our test instances will upgrade themselves and initiate the automated testing. As the tests are run our central build controller is notified of the test results so we have a dashboard that allows our development teams to see the test progress. If *any* test fails, the stream is blocked and the build is no longer deployable. Once all automated and regression tests have passed the release is marked as successful and the build is published to the dev stream

• dev - all development instances used by Hornbill's development teams sit on this stream. Developers use many instances every day in the process of developing both our platform and our applications. Once a new build is available in dev all developer instances are automatically updated. This not only enables developers to validate recent code changes in a production environment but its our first line of defence to pick up anything unexpected. This is also where we run any manual regression testing that needs to be completed. The project build master will make a judgement about when a given build on dev is fit for promotion to the beta stream. A build will typically sit for 24 hours on dev before being pushed forwards. While on this stream we monitor performance, API and system logs to make sure there are no unexpected error conditions. If there are any problems then the build master for the project in question will reject the build which will prevent it moving forwards.

• beta - our beta steam is used by all pre-production instances. Hornbill's own production instances run on this stream, as do our beta test customers, some sandboxes, sales demo instances and partners instances. At this point we do not expect to see any issues but if anything does show up the build is blocked from progressing any further. While on this stream we monitor performance, API and system logs to make sure there are no unexpected error conditions. Once the build has been on beta for 48 hours without issue the build master for that project will promote the build to live.

• live - our live stream is production. Once a build is pushed to live, all production instances will update themselves to that build. Hornbill is an always upto date solution using continuous development and updates with 0 downtime occur throughout the day. However, for platform and applications that may require upto 2 minutes of downtime, occur at 0500 UTC for Platform and 0530 UTC for applications Mon-Fri.

Hornbill Major "New" Feature Update Process

If we add something major that is brand new, for example introduce a new application, a new report a new addtional view that did not previously exist then we simply follow the above Hornbill Update Process and leave the In-App training to guide our users to the new functionality.

Hornbill Major "Change" Update Process

Our major change update process is different to our continuous Update Process set out above. We are always conscious of our customers workflows and general usage expectations and do not just change the user experience or the way our system works. However, we do have to evolve and take our customers with us and we love adding great new features or sometimes significant step changes. In order to introduce such changes/features we provide Feature Option Switches that will allow users to switch between the new and existing behaviour/experience for a period of time known as the preview period, typically this is between 30 and 90 days depending on the feature and the feedback received during the feature preview. A typical workflow for this type of release would be as follows: -

• deploy - in oder to get the preview to our customers we follow the Hornbill Update Process set out above. However, instead of enabling the feature we would typically provide an in-app notice indicating the presence of the new feature along with a feature switch right within the affected view(s) that are being changed. For example, at the time of writing this page we just introduced an all new Email interface which was substantially different (and improved) but as a result the user workflow was also substantially different. So we provided a "Try the new Xyz Feature" button which when pressed will switch the users session to use the new view, they would also have a button to switch back should they wish. Individual users can switch between the old and new way of working as they see fit. When the new feature is enabled, we will generally also provide a in-app link to documentation and/or a conversation to solicit feedback and allow users to ask us questions about the new feature.
• preview - we will leave the feature in preview for a period of time and react to any feedback by enhancing the new major feature using the Hornbill Update Process described above.
• notice - once we are satisfied that the feature is fit for purpose and we have addressed any concerns from our user community we will issue notice to change, this is typically 30 days and will be shown in the application in the form of a countdown giving our users time to make the switch and adjust their workflow.
• finalise - on the planned date the option switch will be removed and all users will be switched over to the new feature.
• stand-off - we leave the now depreciated feature in the codebase for a stand-off period, typically 30 days and in an emergency Hornbill can revert to the old feature during this time. However, we would only do this in exceptional circumstances on a case by case basis and only to provide a short term work-around should a serious problem be found thats impacting a specific customer.