Feb 9, 2014

Acceptance Stage in CI Jenkins and Psake

Related posts:
The “ACCEPTANCE STAGE” is the third job in my delivery pipeline.
Please, refer the general article for setting up the delivery pipeline with Jenkins and Psake here.
You can take a look on the article dedicated to preceding jobs in my pipeline for build stage here and commit stage here.
The primary job of  acceptance stage is to execute acceptance testing, which in my case is based on Selenium. Its job can be summarized with 2 major activities:
  • Deploy the artifacts that are output of the  build stage instance, and that are tested by the  commit stage instance.
  • Executing the Selenium tests with NUnit
1. Deploy artifacts
Deployment task itself is comprised of 2 sub-tasks:
  • Deploying the Web Application to dedicated Front-End server, that is not the build-server.
  • Deploying the database with SSDT to dedicated database server, that is not the build-server.
To get the REVISION context from the upstream job in the pipeline instance (which is the commit stage), mark the job as parameterized and define the same name that is pointed in the build trigger for the “COMMIT STAGE” job.
AcceptanceStageDefinition

The “build” step is quite simple in this case. It contains only “build.cmd”, that was reviewed in the previous articles.
The functionality related to deploying web application and database are executed in “acceptancestage.ps1”.

If you don’t use “msbuild” command line for deploying your web packages to dedicated machine ( that is accessible in your network), you have to implement this with custom programmatically logic.
In my case I did it with Powershell WebAdministration module.

So, the cmdlets from the module that creates web site and applications have to be executed in the context of the machine that they will reside in. And this machine is different than the build machine that host Jenkins and triggers the instance of the “ACCEPTANCE STAGE” job.
One possible option to fulfill this is to build web service, responsible for deployment and web site creation. It can be invoked by the build machine. Unfortunately, this implies to more development efforts, integration and authentication concerns, which technically becomes a complication in the delivery pipeline.

The other approach (the one I chose) is to use PowerShell remoting, which will allow me to call the cmdlets from WebAdministration module in the context of the web server, where I want to deploy my web package.
There is a prerequisite for this. The front-end server (webhost in the script), should expose shared folder, in which the artifacts will be copied before executing the deploy DeployWebProject.ps1 with remoting.
The script uses WebAdministration module of IIS, loops through all published web sites on the front-end server in order to extract and calculate the port of new site that is to be published.
Then New-Website and New-WebApplication command are used. After the web application is deployed, its web config is modified , so the connection string has been properly set up.
DeployWebProject.ps1 is located on the Jenkins server, but if it refers other scirpts or resources they should be placed on the “remote” front-end server.

task DeployApplication { 

    $webhost = "\\webhostIP" 
    $webhostPassword = "webhostPassword"
    $dbhost = "\\dbhostIP"
    $dbhostPassword =  "dbHostPassword"
    Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start DeployApplication task..."
    If ($revision)
    {
        Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start deploying application..."
        $p = Resolve-Path .\       
        #Authenticate as User1 with needed privileges.
        $password = convertto-securestring $webhostPassword -asplaintext -force
        $credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist "webhost\User1",$password
        #copy artifacts
        Write-Host "Moving artifacts packages on the front-end server hard drive..."
        $artifacts_directory = Resolve-Path .\Artifacts\$revision\Source\Packages

        NET USE "$webhost\ci\$revision" /u:webhost\User1 $webhostPassword 
        robocopy $artifacts_directory  "$webhost\ci\$revision" WebProject.zip
        net use "$webhost\ci\$revision" /delete
        Write-Host "Moving artifacts done..."

        #copy deploy scripts
        $deployScriptsPath =  Resolve-Path .\"DeployScripts"    
        NET USE "$webhost\CI\powershell" /u:webhost\User1 $webhostPassword 
        robocopy $deployScriptsPath "$webhost\CI\powershell" PublishWebSite.ps1
        net use "$webhost\CI\powershell" /delete      
        $dbServer = "dbServerConnectionStringName"  
        $dbServerName = "dbServerName"
        $sqlAccount = "sqlAccount"
        $sqlAccountPassword = "sqlAccountPassword"  

        invoke-command -computername webhostIP -filepath  "$p\DeployScripts\DeployWebProject.ps1"  -credential $credentials  -argumentlist @($revision, $dbServer, $revision, $sqlAccount, $sqlAccountPassword)     

        Write-Host -ForegroundColor "Green" -BackgroundColor "White" "Start deploying database..."
        #...pretty much the same
    }
    else
    {
        Write-Host "Revision parameter for the Acceptance Stage job is empty. No artifacts will be extracted. Job will be terminated..."

        throw "Acceptance Stage job is terminated because no valid value for revision parameter has been passed."
    }
}

The link below gives explanation how the credentials might be stored encrypted in a file, rather than being used plain text in ps1 script.

http://blogs.technet.com/b/robcost/archive/2008/05/01/powershell-tip-storing-and-using-password-credentials.aspx

Below is the beginning of the DeployWebProject.ps1 script:
param(

 [string]$revision = $(throw "revision is required"),
 [string]$dbServer = $(throw "db server is required"),
 [string]$dbName = $(throw "db name is required"),
 [string]$sqlAccount = $(throw "sql acocunt is required"),
 [string]$sqlAccountPassword = $(throw "sql account password is required")
 )
    $p = Resolve-Path .\
    Write-Host $p    
    Set-ExecutionPolicy RemoteSigned –Force


robocopy with impersonation is used, because the context of the Jenkins jobs has not permission over the shared folder by default. The context is the user that runs the windows service.

robocopy copies the artifacts from C:\CI\Artifacts\$revision to $webhost\ci\$revision (this might be created dynamically by the deploy ps1 script).

After the web deployment is completed the shared folder content is cleared.

Deploying database uses pretty much the same approach. Artifacts that are need here are:

  • Dacpac file
  • Publish database profile file
  • Init.sql scripts that can be used for creating the initial data needed for the web application to be operational.
*.dacpac file is output file of your SSDT project and should be archived as artifact in the “Build Stage”, after database project is build. It is available in the bin\Debug|Release folder of the SSDT project.

In order to generate database publish profile, open up your ProjectDB.sln and right click on the SSDT project (named ProjectDB) -> Publish.


CreatePublishingDBProfileXml


The click “Save Profile As…” and save the file as ProjectDB.publish.xml. The file is stored on the Jenkins file system.

Below is sample content of the file:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <IncludeCompositeObjects>True</IncludeCompositeObjects>
    <TargetDatabaseName>4322</TargetDatabaseName>
    <DeployScriptFileName>4322.sql</DeployScriptFileName>
    <TargetConnectionString>Data Source=dbserver;Initial Catalog=xxx;User ID=xxx;Password=xxx;Pooling=False</TargetConnectionString>
    <ProfileVersionNumber>1</ProfileVersionNumber>
  </PropertyGroup>
</Project>

Artifacts are copied to shared folder $dbhost\CI\$revision on the database server. Then, again PowerShell remote execution is used alike for web site deployment.

The publish xml is copied on the $dbhost\CI\$revision folder from the Jenkins machine file system location. Its content and connection string are adjusted. The newly created database has the name of the revision number, so it can be easily recognized when troubleshooting is required. Also the publishing profile name is renamed to ProjectDB.publish.$revision.xml, after it is copied to the appropriate folder.

Below is the deployment script for the database:
param(
 [string]$revision = $(throw "revision is required"),
 [string]$dbServer = $(throw "db server is required"),
 [string]$sqlAccount = $(throw "sql acocunt is required"),
 [string]$sqlAccountPassword = $(throw "sql account password is required")
 )

    $pathToPublishProfile = "C:\CI\{0}\ProjectDB.publish.{0}.xml" -f $revision
    $dacpacPath = "C:\CI\{0}\ProjectDB.dacpac" -f $revision
    $remoteCmd = "& `"C:\Program Files (x86)\Microsoft SQL Server\110\DAC\bin\SqlPackage.exe`" /Action:Publish  /Profile:`"$pathToPublishProfile`" /sf:`"$dacpacPath`""

    $sqlInit = "sqlcmd -S {0}  -U {1} -P {2} -d {3} -i `"C:\CI\{3}\ProvideInitData.sql`"" -f $dbServer, $sqlAccount, $sqlAccountPassword, $revision

    Invoke-Expression $remoteCmd
    Invoke-Expression $sqlInit
    Write-Host "Creating database finished.."


As you can see the artifacts are referred as local resources to db server, regardless the DeployDB.ps1 is executed from the build server.

After successful deployment the temp content $dbhost\CI\$revision and $webhost\CI\$revision is cleaned up and the folders are deleted.

At this point you should be able to browse your recently deployed web application, to log in and to work with it. Dedicated database named with the revision number is created for each web deployment.

2. Execute Selenium tests

In the previous article I showed how unit tests can be executed and integrated in Jenkins with bat file.

Here is how i did it with PowerShell.

$nunitProjFile = "$p\Artifacts\{0}\Source\Automation\WebProject.SeleniumTests\SeleniumTests.FF.nunit" -f $revision
$outputFile = "$p\Artifacts\{0}\Source\Src\Automation\WebProject.SeleniumTests\console-test.xml" -f $revision
$nunitCmd = "& `"C:\Program Files (x86)\NUnit 2.6.3\bin\nunit-console-x86.exe`" $nunitProjFile /xml:$outputFile"
Write-Host $nunitCmd

Invoke-Expression $nunitCmd
Write-Host "exit code is " $LASTEXITCODE
 
if ($LASTEXITCODE -ne 0)
{
    throw "One of the selenium tests failed. The acceptance stage is compromised and the job ends with error"
}

Before executing the selenium tests, related connection strings must be programmatically modified to point the correct database and web address of the deployed in previous step web application.
Selenium server should be started on the Jenkins machine in order tests to be successfully executed.
When even one of the tests fail, the job is terminated completes as failed.

Related links:

https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin
http://technet.microsoft.com/en-us/magazine/ff700227.aspx - how to enable PS remoting

25 comments:

  1. Wonderful article, very useful and well explanation. Thanks a lot for offering this unique post with us. I really enjoyed by reading your blog post.

    SEO Training in Chennai

    ReplyDelete
  2. Excellent goods from you, man. I’ve understand your stuff previous to and you’re just too excellent. I actually like what you’ve acquired here, certainly like what you are stating and the way in which you say it. You make it enjoyable and you still take care of to keep it sensible. I can not wait to read far more from you. This is actually a tremendous site..

    SMO Services Chennai

    ReplyDelete
  3. Superb i really enjoyed very much with this article here. Really its a amazing article i had
    ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and
    please keep update like this excellent article.thank you for sharing such a great blog with
    us. expecting for your updation.
    seo company in chennai

    ReplyDelete
  4. wow amazing post.The key points you mentioned here related to maintenance of car is really awesome.Checking all fluid levels,changing oil and of course the regular service of the car which is necessary to maintain our vehicle.Thank you for the information.

    home spa services in mumbai

    ReplyDelete
  5. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.

    cat coaching in Chennai

    ReplyDelete
  6. good blog, All screenshots can explain the full blogs easily. This is very easy to understand. Using this screen shots know the details becomes easy..
    Back to original

    ReplyDelete
  7. Great site for these post and i am seeing the most of contents have useful for my Carrier.Thanks to such a useful information.Any information are commands like to share him.

    hadoop training in chennai

    ReplyDelete
  8. Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..
    PHP Training in Chennai

    ReplyDelete
  9. I just see the post i am so happy the post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be subscribing to your feed and I hope you post again soon.

    digital marketing company in india

    ReplyDelete
  10. This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.

    Home Spa Services in Mumbai

    ReplyDelete
  11. That was a great message in my carrier, and It's wonderful commands like mind relaxes with understand words of knowledge by information's.
    best GMAT coaching classes in chennai

    ReplyDelete
  12. Really nice information you had posted. Its very informative and definitely it will be useful for many people

    Manpower Consultancy in Chennai

    ReplyDelete
  13. I’ve been browsing on-line greater than three hours today, but I never discovered any attention-grabbing article like yours. It is beautiful worth sufficient for me. Personally, if all webmasters and bloggers made good content material as you did, the net will be a lot more helpful than ever before.
    Study in USA Consultants in Chennai | Overseas Education Consultants in Chennai | Australia education Consultants in Chennai

    ReplyDelete
  14. I am a regular reader of your blog. the blog is very interesting and will be much useful for us. I really enjoyed very much with this article here. Really its a amazing article I had ever read. I hope it will help a lot for all.
    SEO Training in Chennai

    ReplyDelete
  15. Nice it seems to be good post... It will get readers engagement on the article since readres engagement plays an vital role in every blog.. i am expecting more updated posts from your hands.
    iOS Training in Chennai
    Android Training in Chennai
    php Training in Chennai

    ReplyDelete
  16. Great and useful article. Creating content regularly is very tough. Your points are motivated me to move on
    SEO Company in Chennai

    ReplyDelete
  17. i have read fully about your posted information. i got some useful ideas from yours. as a web design supporter i got how you are managing effect of school developemt and manging puipil. keep follow more information from yours.
    Best CAT Coaching in Chennai

    ReplyDelete
  18. Learning new technolgy would help oneself at hard part of their career. And staying updated is the only way to survive in current position. Your content tells the same. Thanks for sharing this information in here. Keep blogging like this. iOS App Development Company in Chennai

    ReplyDelete
  19. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    Fresher Jobs
    Fresher Openings

    ReplyDelete
  20. Excellent .. Amazing .. I’m satisfied to find so many helpful information here within the put up, we want work out extra strategies in this regard, thanks for sharing..
    Architectural Firms in Chennai
    Architects in Chennai

    ReplyDelete

  21. Its very useful to me. Wonderful blog.. Thanks for sharing informative Post.

    Installment loans
    Payday loans
    Title loans

    ReplyDelete
  22. It’s the best time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I want to suggest you few interesting things or suggestions.You can write next articles referring to this article. I desire to read even more things about it..
    J2EE Training in Chennai
    Java Training in Chennai
    PHP Training in Chennai

    ReplyDelete
  23. Really Good blog post.provided a helpful information.I hope that you will post more updates like this.
    Digital marketing company in Chennai

    ReplyDelete
  24. I must thank you for the efforts you have put in penning this site. I am hoping to check out the same high-grade content by you later on as well. In truth, your creative writing abilities has inspired me to get my own, personal blog now..
    Office Interior Designers in Bangalore
    Office Interior Designers in Hyderabad

    ReplyDelete
  25. I’m really amazed with your posting skills as well as with the layout on your blog site. Is this a paid style or did you modify it yourself? Either way keep up the pleasant quality writing, it is rare to see a great site such as this one these days.
    Delivery service singapore
    Singapore delivery service
    24 hour delivery singapore

    ReplyDelete