ControlTier 3.4.3 is now available

ControlTier 3.4.3 is now available:
http://sourceforge.net/project/showfiles.php?group_id=151079

Here are the release notes:
http://sourceforge.net/project/shownotes.php?group_id=151079&release_id=678230

And don't forget to check out the ControlTier Wiki for documentation:
http://apps.sourceforge.net/mediawiki/controltier/index.php?title=Main_Page

Brand new examples/tutorials:
http://apps.sourceforge.net/mediawiki/controltier/index.php?title=Examples

ControlTier all Apache 2 licensed in 3.4.3

The next release of the ControlTier distribution will switch to an all Apache 2 license agreement. All of ControlTier 3.x was already Apache 2 licensed except for the Workbench server web app so this really just makes it all consistent. Workbench had been licensed under GPL 2 due to previous business arrangements that have since been re-worked. 

How does this impact end users? Well, not much except for the knowledge that you only have one open source license to be aware of and that Apache 2 provides the most freedom of use. 

New Whitepaper: Achieving Fully Automated Provisioning

Working with our friends at Reductive Labs, we've just finished a whitepaper called "Web Ops 2.0: Achieving Fully Automated Provisioning"(pdf).

There is a live example of our joint ControlTier and Puppet solution, naturally. However, the main point of the paper is much broader than that. What we are proposing is an outline of an open source toolchain for fully automated provisioning.

What's the criteria for "fully automated provisioning"?

That details are in the whitepaper but here's the list we came up with:

1. Be able to automatically provision an entire environment -- from "bare-metal" to running business services -- completely from specification
2. No direct management of individual boxes
3. Be able to revert to a "previously known good" state at any time
4. It’s easier to re-provision than it is to repair
5. Anyone on your team with minimal domain specific knowledge can deploy or update an environment

Here's the proposed open source toolchain (with ControlTier and Puppet highlighted):

Read the whitepaper and let us know what you think. I imagine that we'll get some objection to either the criteria or the toolchain... but that's fine. Sparking a conversation is actually what we want.

The point of all of this is to encourage discussion on how open source provisioning tools fit together to provide real world solutions. It's all too often that we meet people looking to improve their operations but they are befuddled by all of the various open source choices that don't seem to have much relation to each other. Hopefully this paper will be a first step towards greater public discussion. Educating the public lifts all boats.

All of the content in the whitepaper is licensed under the Creative Commons (Attribution - Share Alike) license.

Update: See this post on the dev2ops.org for more information and progress on the next version

ControlTier 3.4.2 posted to SourceForge.net

ControlTier 3.4.2 is now available:
http://sourceforge.net/project/showfiles.php?group_id=151079

Here are the release notes:
http://sourceforge.net/project/shownotes.php?group_id=151079&release_id=671303

And don't forget to check out the ControlTier Wiki for documentation:
http://apps.sourceforge.net/mediawiki/controltier/index.php?title=Main_Page

New ControlTier Screencasts

New ControlTier Screencasts are now up on the wiki:
http://apps.sourceforge.net/mediawiki/controltier/index.php?title=Screencasts

There you'll find:

• Demo screencasts (JBoss deployment, build automation, using Jobcenter, etc...)
• Automation development screencasts (Using ProjectBuilder, creating CTL modules, defining commands, etc...)
• HowTo screencasts (importing packages, deploying packages, defining services, etc...)
• Presentation on using ctl-exec

Additional screencasts will be added as they are produced. If anyone would like to make a screencast and have it included it in the list, send either the file or the link our way.

ControlTier 3.4 Posted to SourceForge.net

I'm pleased to announce that ControlTier 3.4 is now available

For more info see the message group post and release notes:
http://groups.google.com/group/controltier/browse_thread/thread/69c91cf9e0a92e4a

To download ControlTier:
http://sourceforge.net/projects/controltier/

And don't forget to check out the new ControlTier Wiki (also hosted by SourceForge):
http://apps.sourceforge.net/mediawiki/controltier/index.php?title=Main_Page

New Wiki for ControlTier's Open Source Tools

Things are still a bit under construction, but the new ControlTier Wiki (also hosted by SourceForge) is now the definitive source for ControlTier documentation.

As always, any feedback or suggestions are highly welcome.

Abstracted Administration

[The latest version of this post can be found on the ControlTier Wiki]

The idea that drives the ControlTier project is "abstracted administration". In this paradigm, you think about and develop management processes using an abstracted view, one that is independent of any particular physical node or software deployment. What drives the need for abstracted administration is rising scale and complexity . Abstracted administration is paramount in a world where

• the host environment is varied in kind and size (e.g., different scales in heterogeneous environments and also the growing use of "elastic" virtual machine infrastructure)
• management processes are becoming more distributed and more easily impacted by application and environmental differences (e.g, multi-step procedures that execute across the network and across different tools).

This project supports the idea of an abstracted administration paradigm, one where you manage distributed application services and their management processes through a simplified, more standardized and abstracted view. Within this paradigm, an administrator has the choice to focus on managing operations from a higher level, and let the underlying framework coordinate operations across the actual physical environment. Of course, one can also choose to manage things at a much finer level of granularity, performing management activity on a particular host, which still remains important.

Note:
* The ControlTier project is not a VM technology. ControlTier is a technology that lets you deploy and control services hosted on operating system instances (virtualized or not).

Abstracted administration framework

Within this paradigm, the framework

• Lets you manage many deployments through an abstracted, logical structure or at any distinct point by objectifying your process within the context of the abstracted deployment model
• Wraps around your scripts to produce workflow endpoints that can be combined together to execute distributed multi step processes.

Within the administration framework, your operations processes become more uniform and therefore more reusable, with environment specific parameters and settings externalized in a collaboratively maintained model, shared across support teams. Within this paradigm exists a hierarchy of types that let you break down operation processes into several standard layers, each focused on a specific aspect. Aspects exist for package building, staging and deployment, service run state control, mediated execution, and more.

Abstracted Nodes

For command execution, this project aims to help you abstract the physical Nodes in your environment. Commands ultimately execute on some host, but in large scale environments, it's cumbersome to specify the particular hosts. The ControlTier software provides a couple ways to abstract node infrastructure:

• Node tags and attributes: The ctl-exec command lets you execute ad hoc commands by addressing target Nodes using tags and attributes rather than lists of hosts. This is a convenient way to manage host groups and ctl-exec supports inclusion and exclusion filters to identify any subset of hosts. ctl-exec also supports parallel execution important when you need to execute actions simultaneously across a large number of hosts.
• Command dispatching': The ctl command lets you define reusable commands targeted to individual service management control actions or target coordinated distributed actions, logically via a Site. The command dispatcher looks up the nodes on which commands should be executed and invokes them remotely when necessary. In this case, one stays focused on managing service action without having think about nodes.

Abstracting nodes from your procedures is the first step towards abstracted administration. When nodes vary between environments or when they are based on VMs and can be rescaled at any time depending on conditions, your scripts will not have to be changed to redefine node targets.

Abstracting the Service

One of this project's primary goals is to provide a service management interface that lets you forget about Nodes during operation.

• Long running application components are called Services in ControlTier. You abstract your services by exposing all the physical environment differences in the Service's object model. Doing this lets you define your service management code in an abstracted way. During execution your procedures are bound to environment specific views.
• ControlTier's Site, provides the management interface that lets you logically control a set of services be they one machine or many. Application components combine together to form a distributed application. Where these components are hosted depends on the environment. For example, in development or QA they may all reside on one node, while in production they may be spread over many.

Exposing logical control of the many parts of a service is a further step towards abstracted administration, since at this level of abstraction not only are Nodes abstracted but so are the individual application deployments that comprise the integrated service.

Abstracting the process

There are several service management life cycle activities common to any application service: build, stage, install, update, stop, start, configure, check, roll back, etc. Of course these activities vary depending on operating system, application platform, or environment. The last aim of the ControlTier project is this: simplify operations by obscuring environment differences that impact procedures.

• ControlTier includes a standard set of types, each responsible for carrying out each of the life cycle steps. You can also expose your procedures in place of the standard implementation.
• Service management processes are carried out over multiple steps across different machines. Again depending on where the process runs, process execution can occur on different machines. ControlTier workflows allow you to define and execute processes independent of the environment making them more reusable.
• Besides abstracting location, service management processes can also be executed sequentially or in parallel without any code modifications. ControlTier workflows allow you to define a thread count in the object model to control parallel or sequential execution.

By exposing life cycle activities as service management workflows reusable across environments, another level of abstracted administration is achieved.

What drives the ideas behind abstracted administration are the successive layering of abstractions:

• Abstract the nodes, for better visibility into the services.
• Abstract the services, to gain better visibility of the management processes.
• Abstract the processes, for standardized reusable life cycle steps and workflows

Through the process of abstraction, all the specifics become maintained in an object model and the procedural code consolidates into common libraries. Less code means less maintenance, better re-usability, and further elimination of procedural variation, often the root cause of service management problems.

Focus for 2009

Hello All,

We are winding down 2008, after a good year's development that culminated in the current 3.2 release. The features of 3.2 both evolved and matured the 3.1 functionality, but also include several new fundamental capabilities. We are excited to see new 3.2-based solutions.

Besides working with consultants in the service group, we also work with community members and have acknowledged several areas where focussed effort should be made for the coming year:

Vastly improve our documentation. The docs are currently in a woeful state. They are spread out in a hard to navigate structure and are written too abstractly and are not helpful to new users. Here's how we are going to improve them:

• Consolidate all the docs into one medium and one site.
  
• Use a Wiki instead of Forrest. A Wiki is a much more fluid way to keep docs up to date. Also we can easily add community members to help contribute.
  
• Make the documentation "How To" oriented. These are short focussed explanations on using ControlTier software for a typical use case
  

Merge "Elements" and the ControlTier "base" libraries into a single source code module and build artifact. You may not know this, but Elements is a library of ready to use modules interfacing with J2EE and other application infrastructure. It is currently hosted at Moduleforge. To improve out-0f-the box productivity we'll:

• Consolidate Elements and Base into a single controltier project "seed" and CTL extension
  
• Write documentation that will assume the Elements library is already installed
  
• Write tutorials revolving around Elements use cases
  

Improve release process. I admit it... our release process was very erratic. 2008 releases were all driven by some external project schedule and did not serve the community well. Here's how we'd like to change:

• Roadmap and milestone based scheduling
  
• Regular release points
  
• Publicize releases better including mailing list, news, freshmeat, etc
  
• Change lists
  
• Standard use of Duke's Bank demo for QA testing
  

ControlTier Demo shall work out of the box. ControlTier software is a pretty general purpose process automation system which sounds great until you want to see it actually do something. In 2008, we began using the J2EE tutorial's sample application, "Duke's Bank" in our demos. This demo shows quite a breadth of ControlTier use cases and helps make its functionality and applicability more concrete. It should work out of the box. Here's what we want to do:

• Simple demo setup that is well documented and quick to set up
  
• Tutorial documentation revolves around the Duke's Bank use cases to establish a consistent set of examples
  
• Supporting slide presentation that lets you run your own demos when you want to show other people in your own groups
  

Increase community involvement and support

• Take better advantage of the Sourceforge features to allow contributions of all kinds. We barely scratch Sourceforge's surface now and there's some useful tools to take advantage of
  
• The community really is the best authority on how to move the project forward so we want to better facilitate comments from the community discussion areas
  
• Run a ControlTier IRC. Sometimes a person wants to ask a quick question or just bounce some ideas. 
  

Of course, development is not frozen for 2009 but we want to shift priorities to make sure the ControlTier project is healthy and the software usable on its own. We are looking forward to driving these improvements and welcome any feedback or helping hands.

Installing ControlTier 3.2 on Windows XP

[Note: These instructions are for installing 3.2 and they point to an archived version of the 3.2 manual. If you are installing a newer version of ControlTier go to the ControlTier wiki]

Here's my cheat sheet for installing the ControlTier framework on Windows XP (soon to be folded into the standard documentation!):

• Use Firefox with the ControlTier web applications since AJAX compatibility issue preclude using IE
• Follow the 3.2 framework installation notes on Open.ControlTier
  
• Choose a %CTIER_ROOT% directory that doesn't contain spaces since this is know to break certain standard modules (e.g. "C:\ctier")
  
• So far as dependencies are concerned:

1. ControlTier requires Java 1.5. I keep a zipped up version of Sun's JDK that I can unpack into "%CTIER_ROOT%\pkgs" to avoid disturbing the system-wide installation. (There are some issues associated with attempting to install multiple versions of Java on the same system using Sun's Microsoft Installer).
   
2. You can find the Graphviz Windows installer on their download page (the latest version when I looked was 2.20.3. I also install Graphviz into "%CTIER_ROOT%\pkgs" since it usually only required by ControlTier. (Note that I've found myself having to manually sort out the Path system environment variable in order to ensure the correct "dot.exe" is in %PATH% once Graphviz has been installed a couple of times).
   

• Download the Zip release of the latest 3.2 release of the ControlTier framework from Sourceforge. This was version 3.2.4 at the time of writing this posting. (When installing from a local Windows desktop or from an RDP session I prefer to use the Jar installer, for all other circumstances I use the Zip installer).
• In preparing these notes I ran through the installation (taking default values) using the Zip installation method successfully, however ...
  
• ... with this release the Jar installer (which attempts to establish Jetty as a Windows service) is broken.

• In order to start the ControlTier server:

1. I started a new command shell and ran the "ctier.bat" script that the installer placed in my user's home directory ("C:\Documents and Settings\Anthony Shortland").
2. I executed "%JETTY_HOME%\bin\start.bat"
3. I picked up the ControlTier server's "Welcome" page at "http://localhost:8080" and from there launched Jobcenter, Workbench, ReportCenter and Jackrabbit ...
4. ... authenticating as "default/default" as required.

• In order to populate the default Workbench project with a useful set of modules, I downloaded the latest (3.2.4) release of the Elements Module Library seed Jar from the Sourceforge "Moduleforge" project ...
• ... and loaded it via Workbench's Admin page "import seed" dialog.

At this point I had a working ControlTier framework installation ready to push ahead and try both some of the tutorials on Open.Controltier.

Anthony Shortland.
anthony@controltier.com