zDevOps

Ever since ZIGI V1R0 was released there’s been options on how to do the initial install. Straight via git clone (had to have git installed), or via the ZIP download from GitHub (no git required for install) or via CBTTAPE.org ... Seeing as GIT is now only installable via conda, I’ve added this option for ZIGI installs too.. Via https://anaconda.org/zdevops/zigi you can get the ‘short’ instructions on how to install zigi via conda. The good news is, it also installs git and it’s prerequisites. The basic installation steps (after installing miniconda obviously) are: Logon to your USS environment (preferably via an SSH session, not TSO OMVS) and make sure you’re in the ‘conda environment’ you want ZIGI and git to be installed. Assuming you’re creating a neat separated environment for just the goodness of git and ZIGI execute the following commands (the environment used in this example is ‘ispfgit’) to create and activate this environment:

conda create -n ispfgit conda activate ispfgit

Also make sure the zoss-appdev channel (rockets channel with all the goodies) is in the conda config via:

conda config --add channels zoss-appdev

Up next: decide where you want ZIGI to be installed on your z/OS environment. The current version of the ZIGI conda installer will copy the ZIGI.EXEC and ZIGI.PANELS to z/OS datasets prefixed with the environment variable ZPREFIX.

Make sure your user has the required authorisation to allocate datasets with that name. Setting this prefix is done via:

export ZPREFIX=HLQ.PREFIX

With the above example HLQ.PREFIX.ZIGI.EXEC and HLQ.PREFIX.ZIGI.PANELS will be the allocated (and populated) datasets.

Then install ZIGI (and git and the rest) via:

conda install -c zdevops zigi

Conda will then show you what all it’s going to do and ask you if you want to continue:

Solving environment: done ## Package Plan ##   environment location: /zdo/conda/envs/ispfgit   added / updated specs:    - zigi The following packages will be downloaded:    package                    |            build    ---------------------------|-----------------    zigi-3.10                  |                2         182 KB  The following NEW packages will be INSTALLED:    bash:    4.3.48-2        zoss-appdev              curl:    7.66.0-1        zoss-appdev              expat:   2.2.9-4         zoss-appdev              git:     2.26.2-pl524_78 zoss-appdev              libtag:  1.4-1           zoss-appdev              openssl: 1.0.2k-5        zoss-appdev              perl:    5.24.4-4        zoss-appdev              zigi:    3.10-2          zdevops    zlib:    1.2.11-4        zoss-appdev Proceed ([y]/n)? 

Once you reply ‘y’ and conda is done doing what it needs to do you’ll see:

All setup.. you can run ZIGI from HLQ.PREFIX.ZIGI.EXEC(ZIGI) An example environment file (might want to add to ~/.profile?) prepared in /zdo/conda/envs/ispfgit/.zigienv For ZIGI these files need to be in EBCDIC format, keep that in mind (the .zigienv file is in IBM-1047 format already)

This /zdo/conda/envs/ispfgit/.zigienv file (if not copied to your .profile) needs to be entered in the ZIGI dialog upon first start, so ZIGI knows where to find git ... If you (as a lot I hear do) then want to ‘distribute’ the git environment (and ZIGI) to another LPAR/SYSPLEX by copying the files (or the whole ZFS and mounting it at a different spot) make sure to update the environment file and/or the users .profiles (or /etc/profile if you’re REALLY setting things up properly lol)

If you don’t want to mess with that autogenerated env-file, or want to setup your own, the rquired envvars stay the same in regards to the ‘older way’ of installing git...

Here’s an example based on our dev-lpar (where conda is installed in /zdo/conda)

export PATH=/zdo/conda/envs/ispfgit/bin:$PATH export MANPATH=$MANPATH:/zdo/conda/envs/ispfgit/man export PERL5LIB=/zdo/conda/envs/ispfgit/lib/perl5:$PERL5LIB export LIBPATH=/zdo/conda/envs/ispfgit/lib/perl5/5.24.0/os390/CORE:$LIBPATH export GIT_SHELL=/zdo/conda/envs/ispfgit/bin/bash export GIT_EXEC_PATH=/zdo/conda/envs/ispfgit/libexec/git-core export GIT_TEMPLATE_DIR=/zdo/conda/envs/ispfgit/share/git-core/templates export GIT_PAGER=more export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)" export _BPXK_AUTOCVT=ON export _TAG_REDIR_ERR=txt export _TAG_REDIR_IN=txt export _TAG_REDIR_OUT=txt

So, you want to add the Let’s Encrypt CA into RACF so you can call services via HTTPS that use the (awesome and free) Let’s Encrypt Certificates?

Here’s how I’ve done it ...

First, download the Let’s Encrypt Authority X3 certificate via https://letsencrypt.org/certificates/

It’s a basic ‘text-file’ that should look something similar to the following:

-----BEGIN CERTIFICATE----- MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/ KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg== -----END CERTIFICATE-----

Now alloate a dataset in z/OS that's 80 bytes wide and has a RECFM of VB. Then stick the contents of that downloaded certificate into that dataset (ZDO.LETSENC.CA.VB in this example) as a datasetname.

Our trusty IKJEFT01 is well equipped to exeute RACF commands 'via JCL', so let's whip up some JCL to import this certificate into RACF.

//IMPORT EXEC PGM=IKJEFT01 //SYSTSPRT DD SYSOUT=* //SYSPRINT DD SYSOUT=* //SYSTSIN DD * RACDCERT + ADD('ZDO.LETSENC.CA.VB') + CERTAUTH + HIGHTRUST + WITHLABEL('Lets Encrypt CA') SETROPTS RACLIST(DIGTCERT) REFRESH

After submitting that, the Let's Encrypt CA is available on your system. You can verify this via the RACDCERT LIST CERTAUTH TSO command.

You'll see the following information:

Label: Lets Encrypt CA Certificate ID: 2QiJmZmDhZmjgdOFo6JAxZWDmaiXo0DDwUBA Status: HIGHTRUST Start Date: 2016/03/17 12:40:46 End Date: 2021/03/17 12:40:46 Serial Number: >0A0141420000015385736A0B85ECA708< Issuer's Name: >CN=DST Root CA X3.O=Digital Signature Trust Co.< Subject's Name: >CN=Let's Encrypt Authority X3.O=Let's Encrypt.C=US< Signing Algorithm: sha256RSA Key Usage: HANDSHAKE, CERTSIGN Key Type: RSA Key Size: 2048 Private Key: NO Ring Associations: *** No rings associated ***

So, last night I ‘launched’ a MatterMost instance on https://mainframe.community. To summarize MatterMost (via wikipedia) it is: an open-source, self-hostable online chat service with file sharing, search, and integrations. It is designed as an internal chat for organisations and companies, and mostly markets itself as an open-source alternative to Slack[7][8] and Microsoft Teams. In this post I wanted to quickly explain how and why I did this. Let’s start with the why first. But Why? Last week, while working for one of my clients, I discovered they are starting to implemen MatterMost as a cross-team collaboration tool. And I discovered it has integrations, webhooks and bots. Being the Mainframe nerd I am, I quickly whipped up some lines of REXX to call CURL so I could ‘post’ to a MatterMost channel straight from The Mainframe. It was also quite easy in the wsadmin scripts to have jython execute an os.system to call curl and do the post... Now I wanted to take it a step further and create a “load module” that did the same, but could be called from a regular batch-job to, I donno, post messages when jobs failed, or required other forms of attention. Seeing as I was going to develop that on my own ZPDT/ZD&T I needed a sandbox environment. Running MatterMost locally from docker was a breeze, yet not running as “https” (something I wanted to test to work from the still to be made load-module. So, seeing as I already had the “mainframe.community”-domain, I thought, why not host it there, and use that as a sandbox....turned out that was easier done than imagined. But How? The instructions provided at https://docs.mattermost.com/install/install-ubuntu-1804.html were easy enough to follow and should get you up and running yourself in under an hour.

Seeing as there already ‘some stuff’ running at the local datacenter here I already had an nginx-environment up and running. I started with creating a new VM in my ProxMox environment (running Ubuntu 18.04) and made sure this machine got a static IP. From there on I did the following:

sudo apt update sudo apt upgrade sudo apt install postgresql postgresql-contrib

That then made sure the VM had a local database for all the MatterMost things. Initializing the DB environment was as easy as;

sudo --login --user postgres psql postgres=# CREATE DATABASE mattermost; postgres=# CREATE USER mmuser WITH PASSWORD 'x'; postgres=# GRANT ALL PRIVILEGES ON DATABASE mattermost to mmuser; postgres=# \q exit

Of course the password is not ‘x’ but something a bit more secure...

Then, make a change to the postgres config (vi /etc/postgresql/10/main/pg_hba.conf) changing the line

local   all             all                        peer

to

local   all             all                        trust

Then installing mattermost was basically these next commands:

  systemctl reload postgresql   wget https://releases.mattermost.com/5.23.1/mattermost-5.23.1-linux-amd64.tar.gz   mv mattermost /opt   mkdir /opt/mattermost/data   useradd --system --user-group mattermost   chown -R mattermost:mattermost /opt/mattermost   chmod -R g+w /opt/mattermost   vi /opt/mattermost/config/config.json   cd /opt/mattermost/   sudo -u mattermost ./bin/mattermost   vi /opt/mattermost/config/config.json   sudo -u mattermost ./bin/mattermost   vi /lib/systemd/system/mattermost.service   systemctl daemon-reload   systemctl status mattermost.service   systemctl start mattermost.service   curl http://localhost:8065   systemctl enable mattermost.service   restart mattermost   systemctl restart mattermost

So “they” installed a smart-meter at your place and now you want to se eyour data in Splunk? Here’s a little how-to... Step 1. Get a P1-DataUSB Cable or craft your own. Just google “P1 DatCable” and you should be allright.

Step 2. Find one of those RapberryPi’s we both know you have lying around somewhere. Step 3. Connect the cable to the Pi :) Step 4. Get this little “gist” I left at GitHub and run it once. (python metrics.py) You should see some output like the following:

{ "currentTariffIndicator": 1.0, "dateTime": "2020-05-23 21:41:00", "equipmentIdentifier": "xxxxxxxxxxxxxxxxxxxxx", "gasM3Consumed": 1807.74, "kWhConsumedHighTariff": 6339.325, "kWhConsumedLowTariff": 2106.207, "kWhDeliveredLowTariff": 0.002, "longPowerFailures": 42.0, "powerFailures": 801.0, "wattsConsumed": 1.052, "wattsConsumedL1+P": 0.0, "wattsConsumedL1-P": 0.0, "wattsConsumedL2+P": 0.0, "wattsConsumedL2-P": 0.0, "wattsConsumedL3+P": 1.052, "wattsConsumedL3-P": 0.0, "wattsDelivered": 0.0 }

Step 5. Get the UniversalForwarder for Splunk for ARM (the default link on the Splunk website was broken for me, looked like a outdated or ill-updated link) so go here (https://www.splunk.com/page/previous_releases/universalforwarder#armlinux) and pick a version. Get it to your Pi then:

tar xvfz thatfileyoujustgot.tgz splunkforwarder/bin/splunk start --accept-license splunkforwarder/bin/splunk add forward-server <yourIP>:<receiveDataPort> splunkfowarder/bin/splunk add monitor /some/path/on/your/pi

Step 6. Add a little crontab to say:

* * * * * /usr/bin/python /path/to/metrics.py >> /some/path/on/your/pi/data.json

Step 7. You now have your smart-meter data in Splunk.

If you’ve been following me on twitter you might have heard about “zigi”: The z/OS ISPF Git Interface, or as we like to call it: “Probably the best source code management utility since the rubber band!“. You can read all about it at https://zigi.rocks.

It’s an awesome tool to manage your traditional z/OS datasets in a local git repo and do all the funky stuff like pull from, and push to, remote repositories like GitHub, BitBucket and the likes.

Getting real tired of constantly creating and deleting repos at GitHub I thought it would be handy to have a ‘local’ git repo server running. This brought me to the Community Edition of GitBlit (https://gitblit.com). A free available, java-based git-repo server.

How hard would it be to run that on my precious z/OS? Not all that hard it seems...

Here’s a short 8 step how-to, in case you want to set that up too. Or for future me, when I need to lookup how I did this again...

Step 1. Download GitBlit GO for Linux/OSX via http://dl.bintray.com/gitblit/releases/gitblit-1.8.0.tar.gz Step 2. Upload this (binary!!!) to your USS environment Step 3. Make sure you have the gunzip installed via Rocket Software’s Ported Tools and gunzip that file. (or unzip locally then upload the tar-file if you’re really in a hurry lol) Step 4. 

tar xvf gitblit-1.8.0.tar

Step 5. Get into that freshly made directory called gitblit-1.8.0 and

chtag -tc ISO8859-1 gitblit.sh

Step 6. Edit that file (vi, ISPF-edit, whatever rocks your boat) to make it say the following:

#!/bin/bash java -Dfile.encoding=UTF-8 -jar gitblit.jar --baseFolder data

Step 7.

sh gitblit.sh

Then go to https://<ip_of_your_mainframe>:8443 and logon with the default user admin (with password admin)

Step 8.

Don’t worry about the NoClassDefFoundError for com.sun.jna.Native

[WARN ] FAILED org.eclipse.jetty.server.Server@1b0455d7: java.lang.NoClassDefFoundError: com.sun.jna.Native (initialization failure) java.lang.NoClassDefFoundError: com.sun.jna.Native (initialization failure)

Caused by: java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/z/os-s390x/libjnidispatch.so) not found in resource path

The awesome talk by @Voulnet at the Northern Security Conference (nsc.frl) discussed various techniques that are used in malware to ‘obfuscate’ the real inner workings of said malware.

One of the techniques I learned about is “doing a DNS lookup, parse the IP as characters (4 bytes per IP) and inject that into the code”.

The post-talk discussion I had with him got me wondering if this would work on The Mainframe too. Just take IPv4 addresses that are the integer representation of EBCDIC characters and bind them to a dns-entry. 

My idea was that once the correct ip-addresses (that resolve to EBCDIC chars) were ‘DNS-queryable’ I could just load that into the REXX and execute it.

A some of you might or might not know, there is a tiny program that is being run on all the mainframes in the world. It’s called IEFBR14 and is one of the few programs I know with such a humongous “changes/lines of code”-metric.

This post is titled “The Smart Contract that doesn’t do anything...” so it must be about hip, modern Blockchains and not about old and dusty Mainframes.

So you want to use GIT to manage your SYS1.PARMLIB, or your USERID.COBOL.SRCE but you are (just like me) getting really tired of ftp-ing the files to a Linux-box, adding them to a git repo and commit/pushing them over and over (and over) again?

Imagine how awesome it would be to just update your git repository “straight outta ISPF”

We’ve tackled this with some ISPF-Rexx, a little panel and the ported git for z/OS available via awesome peeps at Rocket Software.

TL;DR -> https://streamable.com/qa2th 

After making changes to a member all you need to do is type “ZGADD” in front of it and the REXX will work it’s magic, copying the member over to USS, and adding it to a git repo. If no git repo is present at the requested directory it will even create one and populate it with a minimal “.gitattributes” to ease the ASCII-EBCDIC issues :)

The pop-up window will then ask you where in USS your git-repo is (or where you want to create it) and let’s you enter a short commit message.

After pressing enter you will a directory with the same name as the PDS :) It’s should be no suprise that the member in question (DFNSTRT) is there too...

In keeping the promise of a full git interface from ISPF let’s take a look at git log (yeah I know, still need to do some work on ‘less’)

There are still some ‘sharp edges’ that need to be sorted before this stuff can be released into the wild. It’s already sitting in a neat little private repo on GitHub (updated directly from ISPF of course).

Stay tuned.....it won’t be long until the repo goes public :)

If your company has a mainframe chances are it’s never been given it’s proper day in the sun. We’ve heard all the excuses ranging from “potential system outage” to “we don’t know how”. This training aims to tackle the excuses by demonstrating that mainframes are just computers like everything else, providing the attendees with the language and knowledge to start testing their own mainframes. Arming them with the appropriate responses and tools to tackle every excuse in the book.

This two day training, including real world attack scenarios, will be located in The Netherlands (Van der Valk Breukelen). 

We’re almost certain this is training the first of it’s kind in Europe. 

If you register before Christmas the price is EUR 1075. If you’re part of a Guide Share Europe organisation, like most organisations that have a Mainframe, the price is EUR 995.

If you’re not spending the night (why would you not?) there’s an undocumented EUR 175 discount ;) This also excludes you from dinner though....

You can play the teaser video below.  

Over 200 Mainframe professionals gathered in Almere last Thursday to attend the “National Conference 2017″ organised by the Dutch region of Guide Share Europe. This year’s theme was Digital Transformation and zdevops was one of the many sponsors.

Besides the known suspects from the various Dutch Enterprises there were also Belgian Students who attended a “Master the Mainframe”-meetup.

Above a picture of the ‘main room’. This is where the keynotes where presented. 

Richard Gamblin & Frank van der Wal (both working at European level in the domain of Digital Transformation) kicked off with a marvellous session titled “Gluren bij de buren - How organisations are adopting Digital Transformation”  sharing real-life examples of the projects they are involved in. Their success factors, pitfalls and the motivation for major enterprises whose business critical processes are centred around IBM Z.

After eight breakout sessions and a lovely lunch-break Peter Korsten, Global Thought Leader for IBM took us one step further and talked about the next phase of Digital Transformation in his keynote titled : “Digital ReinventionTM, the next phase of using your digital assets and processes” sharing the possibilities that arise once an enterprise goes ‘all-in’ on digitalization.

Eight breakout sessions further into the day we were treated to an awesome keynote, presented by Zoe Chilton of Red Bull Racing about they way they run their IT. Very enlightening to hear how, in the Formula-1 sphere,  there’s only a “one-week-sprint” to deliver a ‘brand new car’ tuned up to the bolts for the next race. Awesome stuff!

All in all a very awesome day. If you want to see what other people had to say, go and #natconf17 on twitter :) 

For those who missed it: here’s some pictures...

And yes, that was a real Formula-1 car standing at the entrance!!!

Keep an eye out for next years conference, as this event is getting better every year!!  If you want to stay connected to GSE-NL, and their upcoming events, make sure to leave your email address at their MailChimp signup page.

Grrrr!

We really need to update this z1090 environment :(

So yeah, you’ve heard about this marvelous function in TSO called ‘OGET’. So instead of going ‘online’ to ‘read the docs’ you decide to read the manual straight from your TSO session.....

You type: TSO HELP OGET and are presented with page after page of text. I hate to bring it to you guys at IBM but there’s a little typo there.

Did you spot it too?

If you can’t find the typo, there’s a very simple mehtod to find it :)

If you don’t wanna read all the docs (and hunt for more typos) the TL;DR for using OGET is:

OGET /path/to/file ‘SOME.PDS.OR(MEMBER)’ 

This will copy the contents of that file into a dataset or member. There are various options to set the “Access mehtod” of the copy operation.

Convert ASCII/EBCDIC with or without custom translate tables, you name it OGET has it all...

Slight caveat might occur when you use the resulting dataset in your JCL. It will most likely be written in variable blocked 251 bytes records.

So we’ve been trying to implement a little service for validating email addresses on webforms.

Nothing special, nothing ‘rocket-science’-like from the service point of view. A very straightforward use of a brilliant python module ‘validate-email’ by syrusakbary (http://github.com/syrusakbary/validate_email) would do the job just fine!

Calling the service from a webpage without ‘server-side’ hassle neither was the biggest hassle. All that needed done was setting up a simple Flask-app listening to /checkmail called via AJAX with a JSON input for the email address returning valid=yes or valid=no. 

In order to check an email address on a web-form we can add a checkmail(this) call to the onChange of input-field containing the email address (usually ‘#email’) ....

function checkmail(mail) { $.ajax({ type: "POST",    crossDomain: true,    url: "https://path.to.flask/endpoint",    data: JSON.stringify({    mail: mail.value }),    contentType: "application/json; charset=utf-8",    dataType: "json",    success: function(data) {      if (data.result.valid == 'no') {        alert(”email error”);      }  }); }

But then we had to try to solve the ‘annoying’ part of opening this service up for other consumers as well.  And making sure that the service would not be hammered-down.

We had to bring in some RATE-limit or even a pay-for-it service (just like the guys over at http://verify-email.org/ do....

Their implementation of adding this functionality in the backend of your webapp was not something we wanted to go for. The service should be implementable via an Ajax call to our service from the client page.So how did we implement this?

APIKEYS in Ajax request

In order to ‘use’ the service from any webpage you need some APIKEYS in the Ajax request so the service can identify the webpage hosting the service and the end-user using the service. For this we use an API-KEY (identifying the webpage) and a USE-KEY (identifying) the user.

These (obviously) cannot be ‘static’ as that would allow any ‘third-party’ to use the keys (easily obtained via the DOM) on their own page, thus effectively stealing api use counters from potentially paying customers...

To circumvent this behaviour we generate ‘one-time-use’-keys that will be send to the JavaScript-runtime within the calling page via the illusive endpoint ‘/gimmekeys.js’ :)

This will return two lines of JavaScript :

apikey = ‘89a40fdc-e9fb-4e4b-9013-60cef1e03d11′; usekey = ‘27b6171d-fb31-45e2-a5df-3f2161bc2448′;

The apikey is generated for every unique request.environ.get('HTTP_REFERER') requesting /gimmekeys.js

The usekey is somewhat trickier. It too is generated for every client calling the page using our service via 

<script src="https://checkmailtool.herokuapp.com/gimmekeys.js"> </script>

This usekey is then stored as a Mongo Document, defined via MongoEngine as:

class USEKEY(Document):    created     =   DateTimeField(default=datetime.datetime.now)    usekey      =   StringField(unique=True)    forpage     =   ReferenceField(ClientPage)    meta = {                    'indexes': [                                    {'fields': ['created'], 'expireAfterSeconds': 60}                                            ]                        }

Allowing for usekeys to be valid for one minute. These keys can then be used in the webpage for that client (and every other webpage being able to get a hold of these two keys within one minute). We’re not that well schooled on maths but deem this situation very unlikely.....

One of these days the full service will go online (or be sold to verify-email.org lol), for now you can start using it for free (max. 5 calls per day, per webpage).

If you leave a nice reply below and/or give us some traction you might just get a free ‘20K’-api key for your own webpage :)

https://github.com/wizardofzos/reacl 

Sometimes you just don't have the time to mess about.

And then there's no -R option for the setfacl command. Mainly this is a good thing. If somehow you decide your ACL's should be diffirent than from what you've set them initially you should ask yourself the "7 W's" about 3 times?

But when you realise this needs to be done, it's a major drag to provide all existing files with the correct ACL's. Sure, one can 'reallocate' all the files into a new directory (with correct ACL's) then delete the old directory and rename the new one to the old one.

This reacl.sh script has a somewhat other approach :)

Finally I get to post some stuff about WebSphere, The Mainframe and general awesomeness. 

Now, we are being presented with 8.5.5 and the Liberty Profile has JMS support. This means the old complaint of "yeah, but can it talk to MQ?" has been eradicated without any question.

Ever since my March visit to the labs in Boblingen I hoped this 'feature' would be available as it would give way to an even more portable application runtime for the "Enterprise IT Environments" out there. 

Why would one still run a "quote unquote" expensive ND-environment where a 'simple' Liberty Instance would do just as fine? 

If one doesn't really trust the Operating System a total Operating System Isolation Level (IL5) is the preferred modus-operandi, meaning everything is local. Once one moves upwards on the 'Gartner Maturity Scale' Nirvana can be reached where applications and data are shared and thus 'world-useable'. 

Nirvana is a multi-location, multi-server, redundant-locally, application-synchronous environment, but we will not get into the details of that at this point in time.

An environment (like z/OS) is more than capable of running quite some 'Liberty Profile Enabled Applications™' on a single footprint yet still providing an EAL5 certification level. (

Day-to-Day operations for these applications are really easy and pretty awesome once one makes sure an application is uniquely identified by a four-letter-word.

The deployment system for the world-usable environment (for arguments sake let's just say the place where all the customers are) will have the following tools at it's disposal : 

run <app>

stop <app>

update <app>

Where every <app> is a customer application. 

Just imagine this <app> is nothing more than a fully packaged Liberty Profile. (s <app>, f <app>,stop, f <app>,update=xx)

Of course we will be needing a stop-runtime, update-runtime and start-runtime function as well. Being able to retrieve updates, test them and activate them might be a dead-on requirement too. Seeing as all of these these are really nothing more than another Liberty Profile this is considered trivial and thus not further explained ;)

That being said, it might be useful to know the 's receive', 's test' and 's stop' commands :)

Any developer can download the application's production version and populate a local development data store. From there on he (or she) can change anything (alternating from application code to data-definitions and/or operator-instructions) and package this.

This package can then be 'deployed' to a test-environment where it will (or will not) be certified to run on an AnyPlatform™-certified platform.

Once the application has been certified, any customer can decide to try-and-see what this package will do and either deploy or destroy it. Upon deployment the package will go 'live' (world-usable). Server reboots are done solely upon request and without these any application updates can and will be done in-flight without any serious impact to customer experience.