Professional .NET Articles

Feb 9, 2014

Commit Stage in CI Jenkins and Psake

The “COMMIT STAGE” is the second job in my delivery pipeline.
Please, refer the general article for setting up the delivery pipeline with Jenkins and Psake here.
You can take a look on the article dedicated to preceding job in my pipeline “BUILD STAGE” here.
The primary job of “Commit stage” is to quickly validate the artifacts that are archived in the preceding job of the pipeline instance. Its job can be summarized with 2 major activities:

Getting the artifacts
Testing the artifacts

1. Get the artifacts generated from “BUILD STAGE” job
In order to extract artifacts, “Copy Artifact Plugin” must be installed.

In order execution of the tests to work, “COMMIT STAGE” is the first place where the artifacts have to be “extracted” and used.
As first step of build actions in the job definitions, you have to set up the “Copy artifacts from another project” action.

“Target directory” will be created if it doesn’t exist. The path can contain tokens and it will be in my case the number of the revision that triggered the pipeline instance. So, within Artifacts, a folder named with the revision name will be created and all artifacts will be copied from Jenkins home folder to it.

2. Testing the artifacts
Next actions that have to be executed are:

Execute unit tests with NUnit and fail the job if one of the unit tests fail

Generate NCover report, regardless if all unit tests execute successful

Check code coverage thresholds and fail the job if it is below given value

This screenshot gives understanding of what the actions for covering the above actions are:

Ncover.cmd

ECHO %REVISION%

"C:\Program Files (x86)\NCover\NCover.Console.exe" "%ProgramFiles(x86)%\NUnit 2.6.3\bin\nunit-console-x86.exe" "C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\WebProjectTests.nunit" /xml="C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\TestResult.xml" /noshadow  //w "C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests" //ias "Project1.dll" //at  "C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\coverage.trend" //onlywithsource //x "C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\coverage.nccov"

"C:\Program Files\NCover\NCover.Reporting.exe" "C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\coverage.nccov" //lt C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\coverage.trend  //or FullCoverageReport:Html:C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\report_output

The script executes the unit tests and creates coverage report based on the output.
FullCoverageReport has been generated for Project1.dll.
For details about command syntax refer NCover web site at http://www.ncover.com/
Bear in mind NCover is not a license-free tool.

Build.cmd

It’s the same bat file that is already discussed in the previous articles.

The “commitstage.ps1” Psake task copies the NCover report from “WebProject.UnitTests\report_output” to C:\CI\NCoverReports, because that folder is used from Ncover plugin.

Also the report is archived with 7z in PowerShell and the archive is moved to the Artifacts\$revision folder. You can omit this build step, or leave the Psake task empty. It won’t affect the job’s behavior.

Nunit.cmd

set OLDDIR=%CD%
IF not EXIST "C:\CI\NunitResult" (
 cd "c:\CI\"
 md NunitResult
)
cd %OLDDIR%
"%ProgramFiles(x86)%\NUnit 2.6.3\bin\nunit-console.exe" C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\WebProjectTests.nunit /xml:C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\output.xml
if defined ERRORLEVEL if %ERRORLEVEL% == 0 goto success else goto fail_build
: fail_build
copy C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\output.xml C:\CI\NunitResult /y
echo 'there are unit tests that failed.'
exit 1
: success
echo 'all unit tests succeeded'
copy C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\output.xml C:\CI\NunitResult /y
exit 0
The script executes the unit tests and copies the result to C:\CI\NunitResult. This folder path is used by NUnit plugin.

The problem is NUnit plugin for Jenkins doesn’t work with relatives to workspace paths, but only with absolute paths.
NUnit report is available for the job’s instance regardless tests succeed or fail.

Ncover-threshold.cmd

"C:\Program Files\NCover\NCover.Reporting.exe" "C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\coverage.nccov" //lt C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\coverage.trend  //or MethodModule:Xml:C:\CI\Artifacts\%REVISION%\Source\WebProject.UnitTests\%REVISION%.xml  /e //mc BranchCoverage:95:Module


if defined ERRORLEVEL if %ERRORLEVEL% == 0 goto success else goto fail_build
: fail_build
echo 'Tested modules have branch coverage below the given threshold 95%.'
exit %ERRORLEVEL%

: success
echo 'Tested modules are above the given threshold 95%.'

exit 0

Code coverage 95% has been set up as threshold. If the coverage is less than this value, NCover.Reporting fails with exit code '3'. The job instance fails, which automatically invalidates the pipeline

The post-build actions leverage NCover and NUnit command executions to generate reports available for each of the job’s instance executions

NCover and Nunit plugins generate reports that are available for each of the run instances

At last, the “Commit Stage” triggers next job in the pipeline called “Acceptance Stage”, by post-build step “Trigger parametrized build on other projects”. The context $REVISION is passed as parameter.

Build Stage in CI Jenkins and Psake

The build stage is the first job in the delivery pipeline.
Please, refer the general article for setting up the delivery pipeline with Jenkins and Psake here.
The primary job of build stage is to update the source code repository till the latest revision, to build the artifacts and to archive them so they are available to the downstream projects.
Building the artifacts is pretty much building the Visual Studio solutions in your project. Yet this can be not intuitive job to do.
If projects in the solutions that are to be built have post-build or pre-build actions that refer 3^rd party software, building job can become very complex.
If you integrate existing solution in CI pipeline, review all related projects’ post and pre-build actions and check relations with 3^rd party software. Make sure referred 3^rd party software supports command-line invocations.
If you build your CI delivery pipeline for new project from its beginning, choose wisely your 3^rd party software that is going to be integrated with the project. It must support command-line invocations.
The first step for setting up the build stage is to define the repository type and to setup its details.
In my case, as you can see from the screenshot below I use “Subversion”. Once you provide the “Repository URL”, authentication link will be displayed so you can enter repository credentials. After they are saved, they can be changed any time by clicking the question mark button next to the repo url field.

The “buildstage.ps1” can be elaborated, so it can start doing “real” job.
Let’s assume that source content is:

Mvc.net project (regardless the version)

Let’s name it WebProject and consider it is part of the WebProject.sln. It will be located in “C:\CI\Source\Project\WebProject\” folder

Installer vdproj that outputs deployment package *.zip

Let’s name it ProjectPackager.vdproj

Deployment web package will be outputted in “C:\CI\ Source\Setup\Output\”

SSDT DB project (solution)

Let’s name the solution ProjectDB.sln. It is located in “C:\CI\ Source\ProjectDB\”

Unit Tests solution

Let’s name the solution WebProject.Tests.sln

The IDE is Visual Studio 2010.
This scenario covers pretty much everything needed to have a web based software solution, which can be rolled out to production environment via installer.
So, the job that build stage does can be split in 3 major actions:
1. Cleanup the solutions and blank folders that keep previous version of deployment packages
Here is the body of the “Clean” Psake task:

task Clean { 

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start Clean task..."
      If ($revision)
    {

        Write-Host $revision
    }
    else
    {
        Write-Host "Revision is EMPTY"
    }

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start Clean task..."

      Write-Host "Clean WebProject.sln..."
      exec { msbuild  "$code_directory\WebProject.sln" /t:Clean  /v:n /nologo}

      Write-Host "Clean ProjectDB.sln..."

      exec { msbuild  "$code_directory\ProjectDB\ProjectDB.sln" /t:Clean  /v:n /nologo}
    
      Write-Host "Clean WebProject bin and obj folders..."

      Remove-IfExists $code_directory\Project\WebProject\bin

      Remove-IfExists $code_directory\Project\WebProject\obj

      #delete packages folder
      Write-Host "Clean Output folder along with packages in it(if exists)..."

      Remove-IfExists $code_directory\Setup\Output

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Complete Clean task..."
}

#helper methods
function Create-IfNotExists([string]$name) {
    if (!(Test-Path -path $name)) 
    {
        New-Item -Path $name -ItemType "directory"
    }
}
function Remove-ThenAddFolder([string]$name) {

    Remove-IfExists $name
    New-Item -Path $name -ItemType "directory"
}

function Remove-IfExists([string]$name) {
    if ((Test-Path -path $name)) {
        dir $name -recurse | where {!@(dir -force $_.fullname)} | rm
        Remove-Item $name -Recurse
    }
}

As you can see from the “Clean” task code, solutions can be cleaned command line using msbuild command.

If msbuild is not recognized, please check your environment path variable and make sure it contains the path to the command.

Below are some errors you can face, while trying to accommodate psake scripts

error MSB4019:

The imported project "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

If you spend some time googling for solution, you will find out there are 2 options:

a) Copying the following folder from your development machine to your build server fixes this if it's just web applications

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\WebApplication

b) Download the "Microsoft Visual Studio 2010 Shell (Integrated) Redistributable Package"

This in my option is preferable resolution.

Download and install chocolatey (http://chocolatey.org/) and install the VS2010 redist from http://chocolatey.org/packages/VS2010.ShellIntegratedRedist

After chocolately in installed, install the VS redistributable package with command:

cinst VS2010.ShellIntegratedRedist

After installing the redistributable package:

2. Build the solutions you need

Here is the body of the “Build” Psake task:

task Build { 
    Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start Build task..."
 
    Write-Host "Build WebProject.sln in " $config "configuration..."
    exec { msbuild "$code_directory\WebProject.sln" /t:Build /p:Configuration=$config /v:m /nologo}

    Write-Host "Build ProjectDB.sln in Debug configuration..."
    exec {msbuild "$code_directory\ProjectDB\ProjectDB.sln" /t:Build /p:Configuration=Debug /v:m /nologo}

    Write-Host "Build WebProject.Tests.sln"
    exec {msbuild "$code_directory\WebProject.UnitTests\WebProject.Tests.sln" /t:Build /p:Configuration=Debug /v:m /nologo}

    Write-Host "Build WebProject web application package..."
    exec {devenv.com "$code_directory\WebProject.sln" /Build "Release|Any CPU" /Project "$code_directory\Setup\ProjectPackager.vdproj" /$config }

    Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Complete Build task..."
}

Msbuild is used for building the solutions and devenv.com is used to build the vdproj project types.

Make sure that the path to devenv.com is part of the global environment PATH variable.

Unfortunately there is no way to build vdproj without installing Visual Studio 2010 on the build environment. Regardless how bad practices this is, there is no way to overcome the restrictions.

Alternative is to use Wix (http://wix.codeplex.com/ ) and InstallShield, which are the new installer projects since VS2012.

Make sure you have all 3^rd parties that are referenced from the GAC installed on the build machine. That might be various reporting tools, loggers, etc…

Below is a list of common build errors that you may face during working the “BUILD STAGE” job in Jenkins:

Windows SDK required

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets(2342,9): error MSB3086: Task could not find "AL.exe" using the SdkToolsPath "" or the registry key

"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A". Make sure the SdkToolsPath is set and the tool exists in the correct processor specific location under the SdkToolsPath and that the Microsoft Windows SDK is installed

Resolution:
http://www.microsoft.com/en-us/download/confirmation.aspx?id=8279
http://stackoverflow.com/questions/2731365/running-msbuild-fails-to-read-sdktoolspath

Solution file error MSB5009

Solution file error MSB5009: Error parsing the nested project section in solution file.

Resolution:

Regenerate your solution – create empty folder, delete it, rebuild.

Bear in mind that if you have to install SVN client on the build machine for a reason (it might be required for example by the usage of SubWCRev in post-build event), the version of the subversion client depends the version of the SVNkit that Jenkins plugin supports.

“Subversion Workspace Version” in “Manage Jenkins” -> “System Configuration” correspond to SVNkit version in the installed subversion plugin. (1.7 in this case)

3. Define what the artifacts are and archive them

That is definitely the most important part of the build stage in your delivery pipeline. It’s crucial all actions in the delivery pipeline instance to be executed against the very same artifacts if we don’t want to compromise the output of the pipeline.

Once they are built (only once for the instance), they have to be shared with Jenkin’s jobs. There are various artifact repositories over the network like Nexus, Artifactory, etc. They all turned out to be hard to implement with .Net project (at least to me).

That’s why I chose the built in capability for archiving artifacts that Jenkins has. It stores the artifacts on the file system. All paths on the screenshot are relative to the job’s workspace path (“C:\CI”).

The archived artifacts are available for preview for each of the executed/completed job’s instances.

At this stage, a build can be triggered through commit change in the source control management. Project’s solutions are cleaned and rebuilt, artifacts are identified and archived. They are accessible for each of the instances of “BUILD STAGE” job history. Email is sent on success or failure of the build job.

Related links:
https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin (Email notifications plugin)
http://msdn.microsoft.com/en-us/data/tools.aspx (SSDT for Visual Studio)
http://wix.codeplex.com/
Read full article!

Feb 8, 2014

Continuous integration for .Net with Jenkins and Psake

Building delivery pipeline for .Net Web Applications with Jenkins and Psake

I’m going to review the steps needed to implement continuous integration pipeline for .Net web project with Jenkins. The article is intended to serve as base on how to automate .Net projects with Jenkins and Psake. It helps to align the bunch of the tools and components that are part of the complex CI process and reviews a setup from the scratch. The idea behind the process is full automation of building, testing and publishing Microsoft .Net web application.
My CI list of compounds is:

MVC.Net Project

A bunch of other various .net projects

Jenkins for CI Server

Windows 2008 R2 SP1 OS on the build host

Psake for building automation tool, that is based on Powershell

Subversion for SCM system

I chose Psake over MSBuild and Nant, which are quite popular in the .Net automation world. The reason for this is Psake being developed in PowerShell, which makes it in my opinion more flexible and powerfull. C# developers are likely to get used to it faster than to the XML declarative structure and syntax of MSBuild, Nant build frameworks.
As you can see on the schema below the pipeline process is triggered by a commit change in the subversion SCM.
The trigger for this is a commit of code in SCM (source code management) system that occurs in the source control repository. Then few jobs related in pipeline are run on Jenkins.
· Source code is compiled. This job might be comprised of building different type of .Net projects
· Unit tests are executed. A threshold for code coverage has been checked with code coverage tool like NCover or MSTest.
· Target application (which in my case is Web application) is published to Front-End server along with clean database setup on the Back-End server.
· A bunch of acceptance(integration) Selenium Tests are executed against the newly published application
Even one of these pipeline jobs fails, the rest of the pipeline jobs are cancelled (they never get started) and email notifications are sent to people involved in build process and includeded revisions’ owners. This leaves really small room for failing continuously the delivery pipeline without identifying the problem. Technically, every commit produces a testable output, which at the end of the delivery pipeline is ready for manual testing, load testing and sign off for publishing.
Another option is to use scheduled poll that triggers the process. It has some disadvantages compared to hook trigger for each of the commits. Scheduled poll may comprise different revisions and potentially many of them might be the reason for failing the delivery pipeline. This makes the identification of the problems a bit harder.
automated process schema

The first part of the delivery pipeline, which is responsible for building the source code generates the output called artifacts. All artifacts should be used for the rest of the jobs in the running instance of the delivery pipeline. This guarantees the pipeline is not compromised and no “dirty” code changes are included between the executions of separate pipeline jobs. In the given scenario that i review, where unit and selenium tests are executed the artifacts are:

The package generated when publishing is done with Visual Studio. This is pretty much the bin folder, all pages, JavaScript files, etc. Its content may vary depending on the type of the web application and used libraries and frameworks

Bin\Debug|Release of unit tests projects along with *.nunit project file

Bin\Debug|Release of unit tests wrappers for selenium tests along with *.nunit project file. The assumption here is that selenium tests are been exported to C# code that is executed through NUnit tests

SSDT (http://msdn.microsoft.com/en-us/data/tools.aspx) Bin\Debug|Release folder in case SSDT is used for creating database. Otherwise, all related *.sql scripts (no matter how they are maintained) must be treated as artifacts.

Artifacts are the piece of information/state that is going to be tested and published. That is why it is very important all artifacts to be/to stay the same between different pipeline stages in one instance of the automated pipeline.
For more details about generating and sharing artifacts refer “BUILD STAGE” article.

How to build the backbone of the delivery pipeline

At the beginning you have to start with installing the pieces that will allow you to create empty Jenkins jobs that run successfully with Psake, when a code change is committed in the source control version you use.
You have to create few Jenkins jobs, to link them in pipeline and each of the jobs must execute empty Psake task. After that you can put “flesh” on the top of the jobs, which is subject of the other articles.
Below is a breakdown of the steps that build the backbone of the delivery pipeline:

1. Install Jenkins

Download the Windows native package and install Jenkins CI server. It is installed as windows service. Make sure it is started and access Jenkins interface to its default address http://localhost:8080/ on the build server. By default the access is anonymous, so the first things you have to figure out are what kind of authentication you want for your CI server, and what the access rights for your users will be.
The setup can be done through the menu “Manage Jenkins” -> “Configure Global Security”

In my case I have 2 users defined. One is full privileged “build master” user. The other one is called “scripts”. As you can see it has very limited privileges – “Overall Read”, “Job Read”. That is sufficient for starting the upstream job on commit via hook. “Scrips” user will be used later in the post-commit hook for authenticating against Jenkins and triggering instance of the delivery pipeline.
The anonymous user is technically disabled by revoking all access rights that is has by default.
After you do the security setup and logout, you will be asked for login credentials for any further log in in Jenkins.

2. Create jobs in Jenkins

In my case the delivery pipeline is comprised of 3 related Jenkins jobs. It is important to make a note that all jobs in the delivery pipeline share the very same custom Workspace.The custom workspace path is located in “C:\CI”, and that overrides the default workspace locations in “Program Files (x86)”.
The custom workspace settings can be set in job definition “Advanced Project Options”, “Advanced” button next to the section.

Jenkins default root path definitions can be found under “Manage Jenkins” -> “Configure System”, “Advanced” button. It points to “Program Files (x86)”, where is default install location of Jenkins:

Because most of the paths that are used in Jenkins plugins are relative to the root workspace folder for the job, it is important all pipeline jobs to share the same custom workspace folder path. Thus, all artifacts can be placed under common accessible “Artifacts” folder and can be reused among all jobs in the pipeline.
Here are the pipeline jobs:

Build Stage – the source code is checkout from the given svn repository. The Visual Studio solutions are cleared and built again in the required order. Artifacts have been archived.

Building successfully your project might be complicated depending what 3^rd party software it relies on. Details for the build stage can be found here: “BUILD STAGE”.

Commit Stage – the artifacts from the upstream job, which indeed is the “Build Stage”, are copied in the dedicated folder called Artifacts, under a folder named with the SVN revision that triggered the pipeline instance.

That means svn revision number 19999 “produces” artifacts that are shared among Build, Commit, Acceptance stages and are located in C:\CI\Artifacts\19999. “C:\CI” is the custom workspace path.
After artifacts are copied, a set of unit tests are run. If one of the tests fails, the job instance is failed too. That automatically fails the entire instance of the delivery pipeline. Email notifications are sent to the people involved in changes that failed the build. NUnit report is generated and published to the job instance regardless of the tests executions.
After NUnit tests execution, NCover is used to measure the code coverage against defined threshold. Coverage report has been generated and published to the job instance, so it is accessible through the Jenkins GUI.
Details for commit stage can be found here: “COMMIT STAGE”.

Acceptance Stage

Acceptance stage can be split in two sub-tasks, which are implemented as psake tasks in my scenario.

First the web application is published to dedicated front-end server via powershell remoting. After that from the same revision, SSDT project is built and its output is used for publishing the database that corresponds to the web application source. The database is created on dedicated DB server again via powershell remoting.
After publishing is done, the selenium tests are executed through NUnit. Its config files are dynamically updated to point to the location where the application is published.

Details for the acceptance stage can be found here “ACCEPTANCE STAGE”.
Jenkins jobs definitions are located under “jobs” folder in its installation path. For each of the jobs in sub-folders are persisted the latest stable and successful builds, artefacts and instances history.

For most of the tasks you can find existing Jenkins plugins, which only have to be downloaded, installed and configured. I will review the plugins I use in the latter articles devoted to the different Jenkins jobs, which are part of the delivery pipeline

3. Integrate Psake and make related jobs exchange data you need

In my opinion the leading (most valuable) information related to a pipeline is the revision number that triggers the pipeline. It can be used for troubleshooting, for deciding which version to roll out on production, etc. That’s why, at the end of the pipeline the artifacts and deployment packages should be placed in a folder named with revision number that originates the build.
“Build Number” and “Build ID”, which are tightly related to the delivery pipeline instance and running jobs instances in it, are internal information that pertains more to the Jenkins as CI server rather than to delivery pipeline as “human meaning” process with expected output. At the end of the day “revision 1234” means more to the project team than “build 1534”.
Luckily the revision number in Jenkins is related as global variable for the jobs that are linked to polling subversion. You can see the list of global variables available in Jenkins on http://localhost:8080/env-vars.html/? (This is the default address accessible on the Jenkins server). According to the list the revision number is accessible in Jenkins as $SVN_REVISION, and it will be used as “shared” value between all jobs in one pipeline instance.
Well, back to the subject – how to install Psake and make it work with Jenkins?

Download Psake package and unzip it in the Workspace root folder C:\CI. Its name as you can see from the screenshot below is “psake-master”.
“Source” folder in the workspace root C:\CI is where your source will be checkout and updated by Jenkins. It’s been set up in the build job source code management settings panel.

Copy the script called “psake.ps1” from the “psake-master” folder and place in C:\CI. All future modifications on this file, related to Psake integrations with Jenkins will be done in this file.
Create one empty powershell file for each of the 3 jobs (stages) that will be linked in delivery pipeline - buildstage.ps1, commitstage.ps1, acceptancestage.ps1. Create one empty bat file that is going to be the entry point from Jenkins and will be the bridge between Jenkins jobs execution and Psake powershell scripts invocations. Name this file build.bat.
In each of the Jenkins jobs created in 2) add a build action which is “execute windows batch command” and put in the command name field “build.bat” that was just created.

In build.bat file call psake.ps1. The connection between Jenkins and Psake is done!

here is the body of the build.cmd

@echo off


powershell -NoProfile -ExecutionPolicy Bypass -Command "& '%~dp0\psake.ps1' -framework 4.0 -parameters @{revision='%REVISION%';stage='%JOB_NAME%'};if ($psake.build_success -eq $false) { exit 1 } else { exit 0 }"
exit /B %errorlevel%

here is the body of the psake.ps1

# Helper script for those who want to run psake without importing the module.
# Example:
# .\psake.ps1 "default.ps1" "BuildHelloWord" "4.0" 

# Must match parameter definitions for psake.psm1/invoke-psake 
# otherwise named parameter binding fails
param(

  [Parameter(Position=0,Mandatory=0)]
  [string]$buildFile = 'buildstage.ps1',
  [Parameter(Position=1,Mandatory=0)]
  [string[]]$taskList = @(),
  [Parameter(Position=2,Mandatory=0)]
  [string]$framework = '4',
  [Parameter(Position=3,Mandatory=0)]
  [switch]$docs = $false,
  [Parameter(Position=4,Mandatory=0)]
  [System.Collections.Hashtable]$parameters = @{},
  [Parameter(Position=5, Mandatory=0)]
  [System.Collections.Hashtable]$properties = @{}
)
try 
{

    Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start building project..."

    Write-Host "scripts params..."

    foreach ($h in $parameters.GetEnumerator()) {

    Write-Host "$($h.Name): $($h.Value)"

    }
    Write-Host -ForegroundColor "Green" -BackgroundColor  "White"  "Solution Configuration:" $parameters.Item('buildConfig');

    Write-Host -ForegroundColor "Green" -BackgroundColor "White"  "Framework version: " $framework

    remove-module .\psake-master\[p]sake
    Import-Module .\psake-master\psake.psm1
    $config = 'Release'
    $revision = ''
    $stage = ''
    if ($parameters.ContainsKey('buildConfig')) 
    {
        $config = $parameters.Item('buildConfig')
    }

    if ($parameters.ContainsKey('revision'))
    {

        $revision = $parameters.Item('revision')
    }
    if ($parameters.ContainsKey('stage'))
    {
        $stage = $parameters.Item('stage')
    }

    Write-Host $revision

    Write-Host $buildFile

    Write-Host $stage

    $psake.use_exit_on_error = $true

    switch ($stage) 
    { 
        'BUILD STAGE' 
        {
            $buildFile = 'buildstage.ps1'
        } 
        'COMMIT STAGE' 
        {
            $buildFile = 'commitstage.ps1'
        } 
        'ACCEPTANCE STAGE' 
        {
            $buildFile = 'acceptancestage.ps1'

        } 
        default 
        {
            "'{0}' is not a valid stage. No job execution will be triggered upon this input." -f  $stage
            throw "'{0}' is not a valid stage." -f  $stage

            $buildFile = ''

        }

    }

    invoke-psake $buildFile $taskList $framework $docs $parameters -properties @{config=$config; revision=$revision}

} 
catch {

  Write-Host -ForegroundColor "Red" -BackgroundColor "White" "Uexpected error."

  Write-Host  -ForegroundColor "Red" -BackgroundColor  "White" $error[0]

}
finally {

  remove-module .\psake-master\[p]sake -ea 'SilentlyContinue'

  Write-Host -ForegroundColor "Green" -BackgroundColor "White"  "psake module successfully unloaded..."

}

Here are additional details on the implementation:

Build.cmd calls the psake.ps1 with parameters

-parameters @{revision='%REVISION%';stage='%JOB_NAME%'}

We can outline one difference related to how Jenkins global variables are accessible in the bat files.

$JOB_NAME in Jenkins should be referred as %JOB_NAME% in bat files.

Once psake1.ps1 is called, it becomes the global entry point for all Jenkins jobs. From here depending on the job context $JOB_NAME - buildstage.ps1, commitstage.ps1, acceptancestage.ps1 are called with passing the svn revision value.

buildstage.ps1:

Framework "4.0"

# Framework "4.0x64"
properties {

  $code_directory = Resolve-Path .\Source\Src
  $tools_directory = Resolve-Path .\Tools
  $config = "Release"
  $revision = ""
}

task default -depends Rebuild

task Rebuild -depends Clean,Build

task Clean { 

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Clean task..."

}
task Build { 
    Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Build task..."
}

commitstage.ps1:

Framework "4.0"
# Framework "4.0x64"

properties {
  $code_directory = Resolve-Path .\Source\Src
  $tools_directory = Resolve-Path .\Tools
  $config = "Release"
  $revision = ""
}

task default -depends UnitTests

task UnitTests -depends CommitStage

task CommitStage{ 

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start CommitStage task..."
      If ($revision)
     {

     }
    else
    {
        Write-Host "Revision parameter for the Commit Stage job is empty. No artifacts will be extracted. Job will be terminated..."
        throw "Commit Stage job is terminated because no valid value for revision parameter has been passed."

    }
      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Complete CommitStage task..."

}

acceptancestage.ps1:

Framework "4.0"
# Framework "4.0x64"

properties {
  $code_directory = Resolve-Path .\Source\Src
  $tools_directory = Resolve-Path .\Tools
  $config = "Release"
  $revision = ""
}

task default -depends RunAcceptance

task RunAcceptance -depends DeployApplication, RunSeleniumNunitTests

task DeployApplication { 

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start DeployApplication task..."

      If ($revision)
    {
        Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start deploying application..."

    }
    else
   {
        Write-Host "Revision parameter for the Acceptance Stage job is empty. No artifacts will be extracted. Job will be terminated..."

        throw "Acceptance Stage job is terminated because no valid value for revision parameter has been passed."
    }

      Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Complete RunSelenium task..."
}

task RunSeleniumNunitTests { 

    Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start Selenium Tests task..."

}

All staging psake scripts have the same parameters presenting the source code folder, the mode of the build (Build|Release), revision number and tools root folder.

Here is some useful information on how to integrate psake with Jenkins that was very helpful to me.

https://github.com/JamesKovacs/psake/wiki/How-can-I-integrate-psake-with-Hudson%3F

4. Develop commit hook that triggers the jobs in the delivery pipeline. Set up access permissions

For developing the post-commit hook I used the explanations from subversion plugin. Here is the link:

https://wiki.jenkins-ci.org/display/JENKINS/Subversion+Plugin

A bat file calls the visual basic script hook.

SET REPOS=%1
SET REV=%2
SET CSCRIPT=%windir%\system32\cscript.exe
SET VBSCRIPT=C:\svnhooks\post-commit-hook-jenkins.vbs
SET SVNLOOK=C:\Program Files (x86)\VisualSVN Server\bin
SET JENKINS=http://jenkinsIP:8080/


"%CSCRIPT%" "%VBSCRIPT%" %1 %2 "%SVNLOOK%" %JENKINS%

repos   = WScript.Arguments.Item(0)
rev     = WScript.Arguments.Item(1)
svnlook = WScript.Arguments.Item(2)

jenkins = WScript.Arguments.Item(3)

Wscript.Echo "repos=" & repos

Wscript.Echo "rev=" & rev

Wscript.Echo "svnlook=" & svnlook

Wscript.Echo "jenkins=" & jenkins

svnlook = """C:\Program Files (x86)\VisualSVN Server\bin\svnlook.exe"""

repos = """C:\Visual SVN\Repositories\Project"""

Set shell = WScript.CreateObject("WScript.Shell")

Wscript.Echo svnlook & " uuid " & repos

Set uuidExec = shell.Exec(svnlook & " uuid " & repos)

Do Until uuidExec.StdOut.AtEndOfStream
  uuid = uuidExec.StdOut.ReadLine()
Loop

Wscript.Echo "uuid=" & uuid

Set changedExec = shell.Exec(svnlook & " changed --revision " & rev & " " & repos)

Do Until changedExec.StdOut.AtEndOfStream
  changed = changed + changedExec.StdOut.ReadLine() + Chr(10)
Loop

Wscript.Echo "changed=" & changed

Set authorExec = shell.Exec(svnlook & " author --revision " & rev & " " & repos)

author = authorExec.StdOut.ReadLine()

Wscript.Echo "author=" & author

Set dirsChangedExec = shell.Exec(svnlook & " dirs-changed --revision " & rev & " " & repos)
path = dirsChangedExec.StdOut.ReadLine()

Set objFSO=CreateObject("Scripting.FileSystemObject")

Wscript.Echo "path=" & path

nPos = InStr(1, path, "branches/branchName/SourceCode/", 1)

'todo: remove this section
'outFile="c:\test\test.txt"
'Set objFile = objFSO.CreateTextFile(outFile,True)
'objFile.Write "uuid=" & uuid & vbCrLf
'objFile.Write "changed=" & changed & vbCrLf
'objFile.Write "author=" & author & vbCrLf
'objFile.Write  "path=" & path & vbCrLf
'objFile.Write "nPos=" & nPos & vbCrLf

'remove the author name restriction
if nPos = 1 then
    Wscript.Echo "triggering build"

    url = jenkins + "crumbIssuer/api/xml?xpath=concat(//crumbRequestField,"":"",//crumb)"

    objFile.Write "url=" & url & vbCrLf
    Wscript.Echo "url=" & url

    Set http = CreateObject("Microsoft.XMLHTTP")
    http.open "GET", url, False
    http.setRequestHeader "Content-Type", "text/plain;charset=UTF-8"

    http.setRequestHeader "Authorization", "Basic dGVzdDp0ZXN0" 'change  the password (username:password) base64encoded; for example test:test ; no brackets
    http.send

    crumb = null
    if http.status = 200 then
      crumb = split(http.responseText,":")
      Wscript.Echo "crumb=" & http.responseText
    end if

    Wscript.Echo "http.status=" & http.status

    objFile.Write "http.status=" & http.status & vbCrLf

    url = jenkins + "subversion/" + uuid + "/notifyCommit?rev=" + rev

    Wscript.Echo url
    objFile.Write "url=" & url & vbCrLf

    Set http = CreateObject("Microsoft.XMLHTTP")

    http.open "POST", url, False

    http.setRequestHeader "Content-Type", "text/plain;charset=UTF-8"
    http.setRequestHeader "Authorization", "Basic dGVzdDp0ZXN0"

    if not isnull(crumb) then 
      Wscript.Echo "crumb(0)=" & crumb(0)
      Wscript.Echo "crumb(1)=" & crumb(1)
      http.setRequestHeader crumb(0),crumb(1)
      http.send changed

      if http.status <> 200 then

        Wscript.Echo "Error. HTTP Status: " & http.status & ". Body: " & http.responseText

      end if

      Wscript.Echo "HTTP Status: " & http.status & ". Body: " & http.responseText

      objFile.Write "http.status=" & http.status & vbCrLf

      objFile.Write "http.responseText=" & http.responseText & vbCrLf

    end if
end if

objFile.Close

The changes I did compared to the given vbs script in the plugin’s documentation is to add the authorization in the request. I used my scripts Jenkins user along with its password in format

Username:password

http.setRequestHeader "Authorization", "Basic dGVzdDp0ZXN0"

You can use online Base64 encode and decode tool like http://www.base64decode.org/

Another important thing that worths mentioning is that “Poll SCM” option in the build stage job must be checked in order the commit hook to trigger the build in Jenkins. You can put a pattern that runs very rare in the Schedule field.

At this point if you make a commit in the source control repository, the build job should be triggered. The rest of the jobs won’t be executed yet, because they are not related as downstream jobs to the build stage.

5. Relate all jobs from point 2. as upstream -> downstream jobs

Install the pipeline plugin and restart Jenkins. You can see that two other plugins are automatically installed:

JQuery plugin
Parameterized plugin

“Parameterized plugin” is used to link the jobs in a delivery pipeline.

Link “Build Stage” to “Commit Stage”

In the “Build Stage” settings, define post-build action “Trigger parameterized build on other projects”

In “Predefined parameters” section a new parameter “REVISION” is assigned with the value of global variable $SVN_REVISION.

Link “Commit Stage” with “Build Stage” and “Acceptance Stage”

To receive the value of the svn revision that triggered the “BUILD STAGE”, you have to define the job as parameterized and to define the name “REVISION” as string parameter. It should be the same as the “predefined parameters” section in “BUILD STAGE”.

You can link the “ACCEPTANCE STAGE” to “COMMIT STAGE” alike above: post build action “Trigger parameterized build on other projects”.

Link “Acceptance Stage” with “Commit Stage”

To receive the value of the svn revision that triggered the “Build Stage”, you have to define the job as parameterized and to define the name “REVISION” as string parameter. It should be the same as the “predefined parameters” section in “Commit stage”

At this point the relation between the jobs is defiend.

“Build stage” is upstream project to “Commit stage”.

“Commit stage” is downstream project to “Build stage”.

“Commit stage” is upstream project to “Acceptance stage”.

“Acceptance stage” is downstream project to “Commit stage”.

6. Create pipeline view and make it default for your user profile, so you can view all related jobs as sequence

Create a new view of type “Build Pipeline View”

Define its properties and set up initial job to be “Build stage”.

Make the new view default for your profile, so when you click to “My Views” it will be loaded by default. Just point the name of newly created pipeline in your profile “My Views” -> “Default View” field.

At this point the backbone of your delivery pipeline is built. You can commit a change, the source code is updated and empty Psake tasks are successfully executed in each of your Jenkins jobs. If you want you can trigger an instance of the pipeline only on scheduled time instead of commit hook integration.

Here is the list of the articles about each stage implementation:

Professional .NET Articles

Feb 9, 2014

Commit Stage in CI Jenkins and Psake

Build Stage in CI Jenkins and Psake

Feb 8, 2014

Continuous integration for .Net with Jenkins and Psake

Building delivery pipeline for .Net Web Applications with Jenkins and Psake

How to build the backbone of the delivery pipeline

About Me

Links

Visitor Map

Blog Archive