Getting your Feet Wet with me on Continuous Integration

It's that time again, time to try a new fancy feature that our beloved Kentico developers have given us! Like candy on a pole, or that shiny precious ring that rules them all, the draw is inevitable to see what this new thingy can do.  

Our adventure starts with a new project we are starting:  A general development Kentico site that we can build out new and exciting functionality, then export it to our customers.  With multiple developers working on different, or similar items, we needed a method to organize and properly promote this development.  Thus in came Continuous Integration.

What is Continuous Integration?

I guess I should explain what it is first.  Continuous Integration is a development model that Kentico created in version 9, but more fully supported in version 10 (to the point where it is actually usable).  I believe it was created at first to help ease the primal instincts to Source Control EVERYTHING. 

You see, Kentico exists as Database+Files, with many of the items that actually make up the site existing in the database.  This makes it hard for Source control to 'save' your changes.  Virtual objects can help in this endeavor, but there were still limitations to what you could virtually objectify, and rolling back was a bit hit or miss.

So, if you could take everything in the database, and track it in the file system somehow, then you could utilize Source control, push changes, and roll them back.  This is what Continuous Integration does.

How does Continuous Integration Work?

So the basic premise behind Continuous Integration is this:

1. Every database object/table has a Class in Kentico
2. Classes can be Serialized into XML (and then XML can be deserialized back into a class)
3. Every class also has Insert/Update/Delete capabilities.
4. Through the Global Events, every object that is inserted, updated, or deleted triggers global events in Kentico

So how Continuous Integration works is whenever an object is Added, updated, or deleted (#3), it is caught through the Global Events (#4), then that class object (#1) is serialized into an XML file (#2) and stored in the CIRepository in the App_Data folder.

Then your source control can detect the file changes (add, update, or remove) and you can push these changes to your repository.

Other users can PULL your file changes, and then a script is run (the continuousintegration.exe in the BIN folder) that reads the serialized objects in your CIRepository, and performs any needed inserts, updates, or deletes.

Lesson 1: Realize CI's Strengths, Weaknesses, and Purpose

Although source control is very important, and CI can greatly aid in tracking these changes, everyone needs to realize what CI is meant for.  This is stated in it's documentation, but I'll summarize.

Strengths

• CI can track pretty much any change, big or small. 
• CI is built for source control integration, allowing you to push full feature sets from branches to master repositories

Weaknesses

The weaknesses will take a little more explaining.

The first weakness is although this is meant to help organized collaboration of developers, it is not strong at "quick" sharing of changes.  Each time you want another developer to 'get' an update you made, you need to push your changes to the branch, they need to pull it down, and then turn off their site and run the continuousintegration.exe to processes the changes, then start the site back up.  This takes a lot more time than if both were using the same database where the changes would be instant.

It should be noted, that in the case where you want quicker collaboration and are using CI just to track changes, you CAN configure the CI repository to point to a network drive so both you and the other person can have their own files, point to the same database, and see changes instantly and share a repository.

The second weakness it is highly encouraged you do NOT use object versioning, page workflow (outside of basic Edit->Publish, of which only the Publish is tracked).  Again the purpose is that the Source Control would be your version control, which means rolling back must be done by pulling an older version of the file, running the continuousintegration.exe and then resuming.

Purpose

Because of the weaknesses and it's strengths, everyone needs to understand what CI is suppose to be used for and NOT suppose to be used for.

Use CI for development ONLY.  This is not a model to use for a staging server, or a live server.  CI takes resources, slows performance, and with limited check-in/check-out, should only be used for development.

Use CI for more compartmentalized development. Since it is a little bit of a chore to import other's changes, this is a good model for developers working on their own areas, and minimizing overlap.

Use CI so each developer has their own space.  Sometimes it can be a resource nightmare to try to put together a shared Database, and speed from local machines to that database can sometimes be slower.  I know personally that nothing beats working on a Kentico instance on my own computer, with a database on my own hard drives (especially solid state hard drives).  Makes development super fast!

Lesson 2: Get it Right the First Time

The next lesson is, because it can be a hassle to push small changes to other developers, I found it's bets to make sure your site is "ready" before you turn on Continuous Integration.

Make sure you do all these items first, to minimize headaches later:

1. Hotfix your site to the latest version
2. Make sure any Modules to house development are set up
3. Add in your developer users
4. Add in any custom tools you want to have initially (like some of my awesome tools)
5. Create the initial Folder structures according to Kentico's Folder Structure and naming conventions.

Also make sure to communicate with your team where various development should be stored, and make sure they follow Kentico's Best Practices for Customization!

While failure to do these things aren't deal breakers (although performing a hotfix is a bit of a tedious task), it will provide more "push pull process" tasks than you may want to do.

Lesson 3: Don't use TFS (Team Foundation Server)!

Lesson 3b: DO use TFS (Team Foundation Server)

UPDATE September 2017 - Just to give you all a heads up, GitHub ended up having it's own problems, and after some work I found out how to overcome TFS's fault.  Read this blog article with exact steps on how to properly set up Kentico so it catches everything!

The below is the normal part of the original blog, so you can ignore it, i was wrong!

---

There are many Source Control systems out there to choose from, but one I must say you should not, is TFS.  

Why?  I'll tell you why!  Because TFS by default (and me and my buddy could NOT find a way to override this) does NOT include the BIN folder, and it was sketchy on catching App_Data folder changes.

For comparison, I went with Visual Studio Online's GitHub (recommended, and available to those with MSDN subscriptions)

The first part of setting up Continuous integration was you need to include every single file into the master.  TFS made that a nightmare, and kept missing things (like the bin folder, the continuousintegration.exe file, etc).  When I tried again with GitHub, it found all files, no exceptions.

The bigger issue was the App_Data.  TFS by default ignores this, because the App_Data folder is kind of intended to house instance-specific information, not stuff you normally push.  Things like search indexes, or other resources.  The CIRepository is by default stored in the App_Data folder, which means that TFS was actually ignoring the files by default.  It also misses some files (.user and .class) as mentioned in this article, so you have to make sure to configure it to catch these.

GitHub? Found all the changes, right away, no problems.

So I will recommend, use TFS.  If you absolutely must, make sure to use these instructions to change the CIRepositories default location to somewhere where TFS may better catch it, and make sure to read that article and add the special ignore rule exception so it doesn't miss stuff.

Lesson 4: Structure is Important

The last lesson is to think hard about your Source Control structure.  I am not an expert at source control, but luckily I know someone who is, and I'm glad I did.

Make sure to discuss with your team how you want to structure your repository.  I can't tell you in this article what is the best way, because that may change depending on what you want to do, how many developers, what level of sharing there will be, and if you'll be utilizing source control tools and methodologies.

I can tell you what we decided to do.  We decided to have the Master Repository that will house all approved features, and 1 Shared Developer branch.

We went with 1 shared branch vs. a branch per developer because if there are multiple branches, it can get harder to 'share' small changes across the development team.  So all the developers push and pull from the development branch.  Any approved features then will be merged into the master, where we will have our "Finalized Development Site" pull from.  This site will house all the finished development that we can then export to other user sites.  

Final Remarks

This article was written after only spending about 2 days tinkering with CI, so it's by no means comprehensive.  I do plan on doing a follow up sometime down the road as we start working on the beast, and hopefully have found some aspirin for any migraines we may come across (which of course I'll inform you so you can hopefully avoid the pain yourself).

Happy developing!

Update: March 16th - Visual Studio Online Github Note

A small update, sometimes when using Visual Studio Online's Github, it will refuse to pull a co-worker's push because it may overwrite some local files.  If you know you want to do a pull, all your content is pushed up and figured out, but you run into this, follow the below steps in the picture to resolve!