Keeping Workbench trim and fit!

Under the covers the Workbench model data is stored in a set of RDF XML files using the Jena Semantic Web Framework.

At the lowest level, this means that a set of files exists (by default) under "$CTIER_ROOT/workbench/rdfdata" on your ControlTier server for each project you create in Workbench:


$ cd $CTIER_ROOT/workbench/rdfdata
$ ls -lh
total 16M
-rw-rw-r-- 1 anthony anthony 6.1M Dec 8 10:53 Arch_UModules_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 491K Dec 5 17:33 Arch_UObjects_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 6.1M Dec 8 10:53 Arch_UTypes_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 809 Dec 5 14:34 Arch_UXforms_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 490K Dec 8 10:53 Map_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 1022K Dec 8 10:53 Modules_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 56K Dec 5 17:33 Objects_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 1.2M Dec 8 10:53 Types_UPioneerCycling
-rw-rw-r-- 1 anthony anthony 1.4K Dec 5 14:35 Workbench
-rw-rw-r-- 1 anthony anthony 809 Dec 5 14:34 Xforms_UPioneerCycling


A given set of files has the project name appended (in this case "PioneerCycling") and is split into two sets: the primary files and their archives (prefixed with "Arch_").

This would all be largely academic if it were not that managing these files turns out to be critical to the responsive performance of anything but the most trivial projects. It turns out that Jena relies on file level locking to manage updates and in the process repeatedly copies the entire file to temporary "checkpoint" copies. Of course, at the OS level, performance copying files of even tens of MB in size is trivial.

However; streaming the same data through the Jena library turns out to be a significant performance bottleneck; so much so that it really pays to keep the ControlTier repository trim and fit!

The primary way to do this is to navigate to the Workbench administration page, find the "Model Administration (Advanced)" section and run the five file compaction tasks:

This process minimizes the size of the primary data files and can be run as frequently as makes a difference.

Dealing with the archive files is a little more complex.

In normal operation there is no need to track the history of changes to the model so it is reasonable to remove the archive files on a regular basis. The process for achieving this is straightforward:

1. Shutdown Workbench.
2. Remove the "Arch_" files from $CTIER_ROOT/workbench/rdfdata associated with the required project(s).
3. Restart Workbench.

There are a few points to note about this process:

• It is necessary to do this with Workbench stopped as the file set is cached in the JVM's heap and will simply be re-written otherwise.
• You may wish to skip the "Modules" archive file since removing it invalidates Workbench's notion of the most recent ("head") version of the packaged modules on the WebDAV requiring that you repackage all Deployment and Package modules - quite a lengthy process.
• With a project with a stable type model, it is really only "Objects" archive file that has an impact on performance and so it may only necessary to remove this file.
• As a rule of thumb, only worry about files that are > ~20MB.
• There have been cases where we've set process up in cron (since Jobcenter/Ctl/Antdepo requires Workbench to be available for normal operation).

Finally, I should note that this whole issue was much more of a problem under ControlTier 3.1 and that we've done a lot to mitigate its impact on performance under ControlTier 3.2 by eliminating unnecessary model versioning. We have not dealt with the fundamental scaling issues in Jena, so it still pays to be conscious of all this.

Anthony Shortland.
anthony@controltier.com

Package-centric CTL object data

Background

As larger and larger users begin to use ControlTier outside of non-production environments, issues rising from scale, network policy and change management begin to enter. One feature found in the typical ControlTier infrastructure is the immediate propagation of configuration model changes to affect running deployments. This is great when you want to reset a port setting, change paths, toggle app settings or whatever, and have these changes take immediate effect. Many development environments would come to a grinding halt if they couldn't have this instantaneous centralized control.
Large scale production environments require a different set of assumptions due to infrastructure and policy differences:
* Infrastructure: Production environments can span multiple data centers and subnets making a single data repository impractical. This means a centralized Workbench may not be accessible to all clients or scale sufficiently across 1000s of nodes.
* Policy: Operations managers often are uncomfortable with the idea of application management data being managed outside of a change control. This means ControlTier metadata should be released using the same policies as application code.


This blog post proposes an alternative to managing and releasing ControlTier data using a "build and release" methodology.

Build and Release process

ControlTier object data can be maintained and released using the same build and release lifecycle as any other application artifact.

• Build: Checkout object XML data, load it in Workbench for data validation, then archive the results in a package that can be distributed.
• Release: Retrieve object archive from software release repository, distributing it to CTL hosts then extract the archive.

This process can be driven from the "ProjectBuilder" module included in the base framework. Below is a diagram showing the basic elements:



Tool support

ProjectBuilder support

ProjectBuilder can be enhanced to prepare static metadata files in a "build" phase of the process life cycle. Specifically, ProjectBuilder can generate entity.properties data for a specified set of objects. The result of the generation is a set of files archived in a single file.

archive-objects: A new command

A new command should be added that takes object name and type parameters and generates the entity.properties file for each one. Below is the proposed option set:

usage: archive-objects [-file] [-object <>] [-type <>]

CTL support

CTL will include a new administrative command that can create and extract CTL object data files (eg, entity.properties). Below is the proposed option set:

usage: ctl-archive -a action -file archive [-p project] [-d dir]


Archive delivery

The diagram below describes how the process and tools drive the build and release of CTL object data:



This process diagram left the distribution tool open to user preference. The next diagram shows how RPM and Yum can distribute the data:



Alternatively, a simple ctl-exec script could push and invoke the extraction process:




Next steps: Your feedback
All the basic bits and pieces to support this new package-centric data management methodology exists in 3.2.3. What's left is formalizing it in a documented configuration.
What will be most helpful is feedback on the approach and any other considerations that should be factored in.

If it seems like the approach is on the right track, my next follow up should be the documented "howto".

Thanks, Alex

Package management nirvana

Managing the set of package objects and files has been a key challenge for some time now. Obviously, the trick is to keep enough packages around to meet reasonable requirements, but to delete them promptly enough once they are no longer required to avoid consuming too much of the repository's resources.

The "state of the art" since ControlTier 3.0 has been the Builder repoFind command which can (optionally) delete package objects and files (i.e. "purge") following these rules:

• Identify candidate packages by a type regex
• Apply a buildstamp/version regex
• Avoiding deleting any packages that are in use (i.e. have referrer relationships)

(Note that repoFind has a companion command, repoPurge, which it calls directly when the "-purge" option is specified).

I recently improved this state of affairs by adding the Purge workflow and the packagePurgeRegex to the Builder base type in order to add the following rule:

• Avoid deleting the most recently built package associated with the builder

This is achieved by using the attribute to provide an "inverse" regex of the builder's buildstamp attribute as repoFind's buildstamp option - i.e.:

^(?!^${entity.attribute.buildstamp}$).*$

This works well enough with a simple model that has a single builder creating packages of a distinct type. Trouble arises when - as is the case for standard branch, trunk, release development - there are multiple builders (usually of the same type) creating packages of a given type. Since each builder then has a distinct buildstamp attribute; running the Purge workflow for one builder will erroneously delete the most recent builds of any other builder that also manages the package type. Grrr ... the default inverse regex is too inclusive!

The solution for this is to generate a list of buildstamps to exclude based on all instances of the Builder type in question and use that to constrain the action of repoFind.

I have just revised the Builder type to achieve this by adding the "generateBuildstampExcludes" command to the Builder base type. If the Builder (or Builder sub-type) in question has a "BuilderPackagePurgeRegex" setting resource assigned, the command updates its value with the list of buildstamps to exclude.

The last piece of this particular puzzle, then was to add the new command to the Builder's standard Purge workflow, thus ensuring that "entity.attribute.packagePurgeRegex" is set to the appropriate value prior to the invocation of repoFind.

The Purge command's new usage is:

Purge [-buildertype <${entity.classname}>] [-packagetype <${entity.attribute.packageType}>]

... allowing packages of the selected type (defaulting to the assigned package type of the builder) to be purged excluding buildstamps associated will all instances of the specified builder type (defaulting to the type of the invoking builder).

Typical use cases would be to invoke Purge from an operational builder, or to use a "global" ProjectBuilder (co-deployed with the Workbench package repository) to manage purges for all builder and package types.

I've committed these enhancements to the ControlTier 3.1 support branch (since I implemented them for a particular client); however, I'll get them merged into the 3.2 development branch as soon as possible.

Anthony Shortland,
ControlTier Professional Services.

Build and deployment syzygy!

The combination of Subversion, CruiseControl, Ant & Maven with ControlTier has proven to be a powerful unification in the build & deployment universe!

Subversion's provision for committing sets of files labeled with a unique revision number (per repository) provides build numbering as part of a broader release version scheme.

CruiseControl's ability to format and present build output based on custom (Ant and Maven) builders provides a highly visible means of managing builds in the project context.

Ant & Maven deliver comprehensive build tooling providing the means to create packages from a given source base that can be used both for development and release.

To return to the astronomical metaphor, ControlTier is the fifth planet - the "Jupiter" of build and release tools if you like - in the syzygy that really makes this alignment of tools a powerful combination!

When ControlTier's "Elements 2.0" Java Server Module Library is used with version 3.1 of our framework, or "Elements 3.2" is combined with version 3.2 of the framework (due for general availability by the end of July) to automate the build and release cycle, the full power of the integrated tool-set is revealed:

1. The Elements "AntBuilder" and "MavenBuilder" modules use Subversion's "Last Changed Revision" numbers as "build" numbers that are combined with optional "major", "minor" and "release" version attributes to support automatic generation of "n.n.n.n" package "buildstamps". Automatic generation of package version numbers that meet site convention is critical to continuous integration based automated builds.
2. The modules include a command to generate a CruiseControl project definition for inclusion in "config.xml". Management of CruiseControl's configuration and operation is therefore completely automated by ControlTier.
   
3. The modules also include "shim" scripts that serve as adapters to allow the standard CruiseControl "maven2" and "ant" builders to indirectly invoke Antdepo/Ctl which, in turn, executes the module's (build) command (note that Antdepo/Ctl can also be directly called using CruiseControl's "exec" builder). This approach preserves CruiseControl's ability to format build output while allowing ControlTier to manage the release numbering scheme and guarantee predictable builds.
4. Since ControlTier's standard Build workflow (checkout, build, import) is "wedged" between CruiseControl and your Ant/Maven based build, we can both manage keeping the working files up to date ahead of each build as well as automatically finding packages produced by the build and importing them into our repository. The standard build workflow provides a solid bridge from build to deployment.
   
5. It is also possible to use ControlTier's standard Update workflow (build, upgrade, deploy) to include automatic deployment to a "smoketest" environment for testing with tools such as Selenium. ControlTier facilitates automation of post-build integration testing as part of the continuous integration process.
   

Consider the power of combining these build tools with the Elements library support for deploying to a growing set of application technologies such as ActiveMQ, Hsqldb, JavaServiceWrapper, Tomcat, Mysql, Apache HTTP, Mule, WindowsService, JBoss, and OpenLDAP. All supported on Linux/Unix (Solaris) and Windows.

On the commercial side of our business, deploying "out-of-the-box" solutions based on the Elements library has become the standard approach for customers using any of these technologies.

Within a couple of weeks we can establish a new ControlTier based build and deployment infrastructure and migrate existing applications to it giving customers the benefit of Jobcenter for self-service deployment and application management, Reportcenter for visibility into build and deployment activity and the Workbench configuration and process automation modeling tool.

Anthony Shortland
anthony@controltier.com

ControlTier on the road

Starting with BarCampESM back in January, we've been hitting the road showing off our tools and engaging in conversations about IT operations and application deployment. Our travels have included presenting our new CTL framework to a variety of systems management user groups as well as exhibitor gigs at SaaSCon and JavaOne.



Next up... O'Reilly's Velocity 08 (June 23 -24). Velocity is a new conference dedicated to operations and performance for web-based businesses. The speaker lineup and the early attendee list bodes well for plenty of interesting and lively conversations. ControlTier will have a booth in the exhibit hall so if you attend Velocity stop by for a demo of our latest tools or to swap war stories with our hard working engineers. Not registered for Velocity? Use the registration code "vel08js" for 20% off.



Whether it be at conferences, user groups, or other forums... we love meeting other folks in our field. If you have any suggestions, leave a comment or send us an email at automation@controltier.com.

Wrangling the package repo!

Once you're using a builder from day-to-day frequently adding new packages to the ControlTier (WebDAV) repository, it's alarming how quickly their numbers and the amount of disk space they consume can rise!

A couple of new Builder base type commands were introduced with ControlTier 3.1 to facilitate managing the package repository: "repoFind" & "repoPurge".

The "repoFind" command's "-purge" option comes to the rescue by providing the antidote to "repoImport":

repoFind [Builder] [-buildstamp <.*>] [-packagetype <>] [-session ] [-version <.*>] [-purge]
Find package objects in the repository.

Essentially, the command queries the object model matching package objects by "buildstamp" and "packagetype" regex patterns to create a list of candidate packages, the (optionally) calling the "repoPurge" command which invokes each package's "purge" command:

purge -url <>
removes the file from the repository

... which in turn deletes the package file using the Ant "davdelete" task. Finally, the "repoPurge" command executes a batched deletion of all the selected package objects. Note that only package that do not have referrer relationships are deleted (i.e. packages that are not in use).

Here are a couple of examples:

• Purge all packages regardless of their type associated with a given buildstamp:
  

  $ ad -p rhbc -t AtgDasModuleBuilder -o babyandchild -c repoFind --
     -buildstamp 8734 -purge
  

• Purge all packages of a given type regardless of their buildstamp (the "packagetype" option value can be a regex too):
  

  $ ad -p rhbc -t AtgDasModuleBuilder -o babyandchild -c repoFind -- -buildstamp '.*'
     -packagetype AtgModuleJar  -purge
  

By the way, having just run through using "repoFind" with one of our client's projects, I can report a subtle bug that I've just fixed in the "repoPurge" command. It turns out that the command assumes that the required Package module is installed in order to be able to run the "purge" commands. This is always the case if you invoke the "repoPurge" command from a node which has previously been used for a "repoImport" (e.g. your build system), but if you choose another system (e.g. the ControlTier server), the module is not available and the command fails.

Here's the fix using the same logic used in "repoImport":

$ svn diff repoPurge.xml
Index: repoPurge.xml
===================================================================
--- repoPurge.xml       (revision 8087)
+++ repoPurge.xml       (working copy)
@@ -88,6 +88,13 @@
    <session-get session="${session.command}" resultproperty="pkg.mapref-uri" key="purge.package.@{pkgType}.@{pkgName}.mapref-uri"/>
    <session-get session="${session.command}" resultproperty="pkg.repo-url" key="purge.package.@{pkgType}.@{pkgName}.repo-url"/>
    <propertyregex property="filetype" override="true" input="@{pkgName}" regexp="[^\.]*.([^\.]*)$" select="\1"/>
+         <controller>
+           <execute>
+             <context/>
+             <command name="Install-Module" module="Managed-Entity" depot="${context.depot}"/>
+             <arg line="-depot ${context.depot} -module @{pkgType} -version head"/>
+           </execute>
+         </controller>
    <controller>
      <execute>
        <context/>

The fix will make the next 3.1 point release and will be merged into the 3.2 development brach and posted to Sourceforge soon.

Thanks,

Anthony.

See you at JavaOne

This year marks the first we go to JavaOne as an exhibitor. It's about time, given our deep experience in automating development and operations for our predominantly Java-using customer base. Our Java Server Library, "Elements", has been evolving over the last couple years, and has really been an instrumental part of improving customer's life cycle reliability and speed.
Java is front and center at ControlTier. Our new CTL control dispatcher has special documentation areas for Ant and Maven users. There's still a lot of functionality useful to Java users that we are in the midst of packaging and documenting.
Of course, all the Open.ControlTier software is Java-based, and as one of ControlTier's developers, it's exciting to be at JavaOne to see what's new.
If you are going to be there, swing by the ControlTier booth and say hello.

Example JBoss deployment screencasts

My previous post on deploying a sample application to Tomcat proved so popular that I've added this set of screencasts that describe how to do a similar deployment to JBoss. (Note that the screencasts assume that you've been though enough of the original set to have a couple of nodes setup with the Demonstration project).



Thanks,

Anthony Shortland
anthony@controltier.com

A sheep in wolf's clothing

My recent post regarding configuring OpenSSH on Windows using Cygwin was written from the perspective of users wanting to exploit ControlTier in a broadly Windows based environment.

In this post, I'm going to document a Unix-centric OpenSSH/Cygwin installation designed to make a Windows server look as much like a Unix system as possible when accessed from the network in order to simplify managing a few Windows based systems in largely Unix based environment.

Cygwin software installation

• Create a local or domain Windows administrator account that has a POSIX user name (I use the "build" account for these notes).
• Download and run the Cygwin installer.
• The cleanest way to install Cygwin in the root of its own dedicated partition since it is absolutely necessary that the Cygwin root directory is synonymous with the Windows file system root for that drive so that Java's platform agnostic path management will work equally well with the Unix or Windows versions of key paths. Using a separate partition is desirable in order to separate the application installation (under ControlTier and Cygwin) from the Windows OS installation (typically on drive C:):
  

  $ df -k
  Filesystem           1K-blocks      Used Available Use% Mounted on
  E:\bin                20964348    213848  20750500   2% /usr/bin
  E:\lib                20964348    213848  20750500   2% /usr/lib
  E:                    20964348    213848  20750500   2% /
  c:                     8377864   7003552   1374312  84% /cygdrive/c
  $ cd /
  $ ls
  Cygwin.bat  RECYCLER                   bin       dev  home  proc  usr
  Cygwin.ico  System Volume Information  cygdrive  etc  lib   tmp   var

• If this is not feasible then ignore the warnings and select "C:\" as the installation root directory to create a "hybrid" directory structure:

  $ pwd
  /
  $ ls
  AUTOEXEC.BAT            MSDOS.SYS                  WINDOWS   home          tmp
  CONFIG.SYS              MSOCache                   bin       lib           usr
  Cygwin.bat              NTDETECT.COM               boot.ini  ntldr         var
  Cygwin.ico              Program Files              cygdrive  pagefile.sys
  Documents and Settings  RECYCLER                   cygwin    proc
  IO.SYS                  System Volume Information  etc 

• Beyond the base package set make sure you include "openssh" (and hence its dependencies). Of course, there are many many other useful packages that you'll probably like to include for a practical installation of Cygwin (e.g. "rsync", "unzip", "zip", "vim", etc).

SSH server configuration

• Cygwin includes a script to configure the SSH service, run from a "Cygwin Bash Shell" (Note the value given to the CYGWIN environment variable. Note also my comment to the original posting regarding W2k3 Server complications) :
  

  $ ssh-host-config
  Generating /etc/ssh_config file
  Privilege separation is set to yes by default since OpenSSH 3.3.
  However, this requires a non-privileged account called 'sshd'.
  For more info on privilege separation read /usr/share/doc/openssh/README.privsep
  .
  
  Should privilege separation be used? (yes/no) yes
  Generating /etc/sshd_config file
  
  
  Warning: The following functions require administrator privileges!
  
  Do you want to install sshd as service?
  (Say "no" if it's already installed as service) (yes/no) yes
  
  Which value should the environment variable CYGWIN have when
  sshd starts? It's recommended to set at least "ntsec" to be
  able to change user context without password.
  Default is "ntsec".  CYGWIN=binmode ntsec tty
  
  The service has been installed under LocalSystem account.
  To start the service, call `net start sshd' or `cygrunsrv -S sshd'.
  
  Host configuration finished. Have fun!

• Start the SSH service:
  

  $ net start sshd
  The CYGWIN sshd service is starting.
  The CYGWIN sshd service was started successfully.

Java installation

• Naturally, you can use the Windows system default Java installation so long as its either Java 1.4 or 1.5. However, it may be preferable to install a version of Java specifically for the use of ControlTier. By convention this is installed into "$CTIER_ROOT/pkgs" (usually "$HOME/ctier/pkgs" of the account used to run ControlTier).
• Note that although Sun distributes its JDK in Windows (graphical) installer format, there's nothing stopping you creating a Zip file of a "reference" installation and using that to setup Java across the network.
• Wherever Java is installed, set up the JAVA_HOME environment variable ahead of the ControlTier installation.
  

ControlTier installation

• As of ControlTier 3.1.5 the Unix install script ("install.sh") is not compatible with Cygwin (possibly due to assumptions built into Sun's JDK on Windows).
• For this reason, installing the ControlTier software over the network still follows the Windows pattern.
• Setup the key environment variables with Windows style values:
  

  $ export CTIER_ROOT=~/ctier
  $ export JAVA_HOME=~/ctier/pkgs/jdk1.5.0_14 
  

• The key thing is to run the "install.bat" command shell from the Cygwin Bash shell:
  

  $ cmd.exe /C install.bat -client
  
  -check-prereqs:
       [echo] Using compatible Java version: 1.5
  
  -load-props:
       [echo] Using CTIER_ROOT: /home/build/ctier
  .
  .
  .
       [echo]     if [ -f ~/.ctierrc ]; then
       [echo]     . ~/.ctierrc
       [echo]     else
       [echo]     echo ~/.ctierrc not found 1>&2
       [echo]     fi
  
  install-client:
       [echo] Install Complete
  

• Next, manually setup the ".ctierrc" file in the Cygwin user's home directory to ensure the correct shell environment is available:
  

  $ pwd
  /home/build
  $ cat .ctierrc
  # this file was generated by ControlTier installer.
       
  export CTIER_ROOT=~/ctier
  
  export ANTDEPO_HOME=~/ctier/pkgs/antdepo-1.3.1
  export ANTDEPO_BASE=~/ctier/antdepo
  
  # Server settings
  export JOBCENTER_HOME=~/ctier/pkgs/jobcenter-0.7
  export CATALINA_HOME=~/ctier/workbench
  export CATALINA_BASE=~/ctier/workbench
  
  export JAVA_HOME=~/ctier/pkgs/jdk1.5.0_14
  
  export PATH=$JOBCENTER_HOME/bin:$ANTDEPO_HOME/bin:$CATALINA_HOME/bin:$PATH
  
  if [ -n "$BASH" ] ; then
  . $ANTDEPO_HOME/etc/bash_completion.sh ;
  if [ -t 0 -a -z "$ANTDEPO_CLI_TERSE" ]
  then
  ANTDEPO_CLI_TERSE=true
  export ANTDEPO_CLI_TERSE
  fi
  fi

• Finally, override the "depot-setup" and "ad" scripts to invoke their Windows counterparts:
  

  $ cat $ANTDEPO_HOME/bin/depot-setup
  #!/bin/sh
  
  exec cmd.exe /C depot-setup.bat "$@"
  $ cat $ANTDEPO_HOME/bin/ad
  #!/bin/sh
  
  exec cmd.exe /C ad.bat "$@"
  

With this "sleight of hand" in place, it is possible to manage Windows systems on the network in the same way as their Unix/Linux counterparts taking full advantage of the Cygwin and Java/Ant abstractions of the underlying OS facilities.

(By the way, a future version of ControlTier will resolve the script and JDK compatibility issues that result in the customizations in this posting).

Anthony Shortland,
anthony@controltier.com

Example Tomcat deployment screencasts

I've created the following set of screen-casts that take you through the process on installing ControlTier in a multi-node environment and deploying a sample Tomcat based web application using the Elements 2.0 module library.

Sort the list of screen-casts by date and start with the three box installation:



These screen-casts are something of an experiment for me. Let me know what you think of this approach to providing tutorial-style documentation.

Anthony Shortland.
anthony@controltier.com