#runbook

Orchestrator Runbook Examples

Runbook for Active Directory User Account Provisioning

Self-Service Active Directory User Provisioning Using System Center Orchestrator and SharePoint

Runbook for Active Directory User Password Reset

Runbook for Automatic System Log Archiving

Runbook for Workstations Power Saving

Runbook for Weekend Maintenance Routine

Server patching - maintenance mode, add to collection, wait for compliance, remove, error handling

Scheduled file transfers

Sharepoint administration

Support ticketing integration

Restart any Windows Service (on SCOM Alert) From a Single Orchestrator Runbook

Alert Remediation - E.g. Free space issues on application servers

Exchange Automation using Orchestrator 2012

Exchange Mailbox Management

Exchange Mailbox Database Management

Exchange Mailbox High Availability Management

Exchange Distribution Group Management

Exchange Address List Management

Exchange Routing Management

Automatic mailbox creation, modification or deletion as part of the User Provisioning Process for new, existing or retired employees

Automatic database maintenance for reorganizational tasks during the night or weekend

Automatic recovery tasks for mail routing issues

Areas for Possible Automation

Patching

User and Group Maintenance

Security Sweeps

Disk Usage Scans

Performance Monitoring

File Transfers

Code Promotion

High-Level Administration

Reboots

Sources

[0] https://www.packtpub.com/mapt/book/Virtualization-and-Cloud/9781785287589/5

[1] https://www.reddit.com/r/sysadmin/comments/4virzj/system_center_orchestrator/

[2] http://www.systemcentercentral.com/4-orchestrator-runbooks-every-scom-administrator-should-know-about/

[3] http://www.msexchange.org/articles-tutorials/exchange-server-2010/management-administration/exchange-automation-using-orchestrator-2012.html

[4] https://automys.com/library/asset/self-service-active-directory-user-provisioning-using-system-center-orchestrator

[5] https://blogs.technet.microsoft.com/antoni/2016/04/07/system-center-service-manager-scsm-and-orchestrator-scorch-integration-and-using-the-scsm-self-service-portal-ssp-as-a-front-end-to-orchestrator/

New Post has been published on http://www.eventenrichment.org/event-enrichment-cisco-switch-duplexmismatch/

Event Enrichment: Cisco : Switch : DUPLEXMISMATCH

Next up in our ongoing Event Enrichment series; a generic enrichment providing triage steps for duplex mismatch errors for Cisco routers and switches.

While enjoying your shift in the quiet solitude of the NOC ;), you suddenly receive an alert from PagerDuty or your NMS. Depending on your level of expertise, you would typically need to open a runbook or Ops Wiki to determine how to handle the event.

Instead, let’s explore a different method, Event Enrichment, using the following syslog entry as our reference.

Jan 14 11:16:45 %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1/0/1 (not half duplex), with TBA04251226 3/2 (half duplex).

Clearly this event has some useful information in it but requires user intervention to investigate the problem. Let’s assume that this same event again arrives at the NOC, this time enriched with the procedure necessary to handle the event.

In that scenario, remediation time will decrease as the information required to properly triage the problem is already embedded in the initial alert.

Cisco : Switch : DUPLEXMISMATCH Event Enrichment

Remediation

A mismatch on the duplex settings between two devices ( a switch and a router / two switches / a switch and server / etc ) is a common problem. This situation generally occurs when the speed and duplex modes between two devices is left to be set by auto-negotiation rather than specific configuration instruction.

To determine the status of the port on each device, log into the device using either SSH, if available, or telnet.

Next, issue the Cisco Discover Protocol (CDP) command to check the device on the far side of the interface:

sw-01-br#sh cdp neighbors fastEthernet 1/0/1 detail ------------------------- Device ID: sw-06-gr Entry address(es): IP Address: 10.10.11.6 Platform: cisco WS-C3750-48TS, Capabilities: Router Switch IGMP Interface: FastEthernet1/0/1, Port ID (outgoing port): FastEthernet1/0/33 Holdtime : 120 sec Version : Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(55)SE5, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2012 by Cisco Systems, Inc. Compiled Thu 09-Feb-12 18:59 by prod_rel_team advertisement version: 2 Protocol Hello: OUI=0x00000C, Protocol ID=0x0112; payload len=27, value=00000000FFFFFFFF010221FF000000000000001562643300FF0000 VTP Management Domain: 'cisco' Native VLAN: 1 Duplex: half Management address(es): IP address: 10.10.11.6

Finally, issue the ‘sh interface’ command to check the interface on this switch:

sw-06-br#sh interfaces fastEthernet 1/0/1 status | b Duplex Port Name Status Vlan Duplex Speed Type Fa1/0/1 connected 1 a-full a-100 10/100BaseTX

Conclusion: There is a duplex mismatch (half versus full) between switch sw-01-gr and sw-06-br. Furthermore there is a port set to auto negotiate which is against best current practice for Cisco interface configurations.

ESCALATION

Provide the NetEng team with the following information:

Original Event Summary:

[Jan 14 11:16:45 %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1/0/1 (not half duplex), with TBA04251226 3/2 (half duplex).]

Verified Findings:

Switch Side A Name [sw-01-br] Switch Side A Interface [Fa/1/0/1] Switch Side A Duplex Value [Full]

Switch Side B Name [sw-06-gr] Switch Side B Interface [Fa/1/0/33] Switch Side B Duplex Value [Half] <== This port is set to auto-negotiate

Escalate your findings to the On-Call NetEng team using the PagerDuty NetEng Service (or other alerting mechanism).

Adopting the Event Enrichment methodology enhances the standardization and scalability of your NOC and on-call processes.

What are some of the remediations that you use in your infrastructure?

New Post has been published on http://www.eventenrichment.org/event-enrichment-unix-rsyslogd-imuxsock-message-drops/

Event Enrichment : Unix : Rsyslogd : IMUXSOCK MESSAGE DROPS

This is the first in our sample Event Enrichment series!

While enjoying your shift in the quiet solitude of the NOC ;), you suddenly receive an alert from PagerDuty or your NMS. Depending on your level of expertise, you would typically need to open a runbook or Ops Wiki to determine how to handle the event.

Instead, let’s explore a different method, Event Enrichment, using the following syslog entry as our reference.

It all starts with an alert…

Jan 28 12:01:20 zenoss-mon rsyslogd-2177: imuxsock lost 5 messages from pid 1169 due to rate-limiting

This event has some useful information (we are losing messages which could be important, or event critical), but requires user intervention in order to investigate the problem. Now, assume that this same event arrives at the NOC already enriched with the steps required to handle the event. Mean Time to Repair (MTTR) would decrease given that the information required to properly triage the problem is already included in the initial alert.

Event Enrichments are composed of two components: remediation and escalation. Remediation consists of the steps necessary to rectify the problem, beginning with troubleshooting. The escalation includes the information to pass along as well as the intended recipient of said information (team or individual engineer)

REMEDIATION 

The first step in investigating this alert is to log into the device / server generating the error.

ops@noc-jump:~$ ssh ops@zenoss-mon Last login: Tue Jan 28 17:38:54 2014 from 172.25.230.5 [ops@zenoss-mon ~]#

The next step is to determine the process associated with the PID referenced in the alert.

Example

[ops@zenoss-mon ~]# ps aux | grep 1169 root 1169 0.2 0.1 203312 10368 ? S Jan23 22:48 /usr/sbin/snmpd -LS0-6d -Lf /dev/null -p /var/run/snmpd.pid

From the result of this command we can conclude that snmpd is generating more events than the configured rate-limiting threshold for rsyslogd (default is 200 events in a 5 second interval). On-call systems engineering will need to investigate the cause of this message suppression.

ESCALATION

Escalate to the on-call SysEng team using the PagerDuty SysEng Service (or other alerting mechanism) and include the following information:

Original Event Summary:  [Jan 28 12:01:20 zenoss-mon rsyslogd-2177: imuxsock lost 5 messages from pid 1169 due to rate-limiting]

Verified Findings:

This error is being generated due to an issue with /usr/sbin/snmpd.

[ops@zenoss-mon ~]# ps aux | grep 1169 root 1169 0.2 0.1 203312 10368 ? S Jan23 22:48 /usr/sbin/snmpd -LS0-6d -Lf /dev/null -p /var/run/snmpd.pid

      Adopting the Event Enrichment methodology enhances the standardization and scalability of your NOC and on-call processes.

New Post has been published on http://www.eventenrichment.org/it-operations-wo-a-runbook-is-like-slapping-murphy-with-a-rotten-fish/

IT operations w/o a Runbook is slapping Murphy with a rotten fish.

flickr/cverdier

It’s the middle of the night, probably around 03:18 in the morning. You are sound asleep after a fantastic night out on the town. All of a sudden, you hear the dreaded “bzzz bzzz bzzz” of your phone on Do Not Disturb Mode (you do have your phone configured to DND at night except for Pagerduty right?)

Groggily rubbing your tired eyes, you look at the screen and see a PagerDuty alert from your NOC. “Damn” you think, “just my luck”. You fire up the VPN and ssh to the host. As it turns out, this is a recently provisioned front line Apache server for a new web application that rolled out to the company last week. You wearily start trying to track down the relevant information regarding the credentials, critical components, affected users, etc. associated with this server. Get ready for a long night…

Compare and contrast that with the same scenario but upon firing up the VPN you go directly your company’s runbook and look up the type of event (Network or Systems), class of server (web server, database, etc), followed by the specified server. You find the relevant entry which contains the information you need and then immediately start the remediation process.

Run books are critical for IT Operations. Without a runbook, your company remains in the un-scaleable, fraught with error, and frustrating, morass of oral tradition based IT Operations. By allowing critical information to remain in the heads of a few (or in some cases, one) senior engineering resources, you leave yourself exposed to increased downtime in the event that the critical resource is unavailable.

There are many options for setting up Runbooks including: using a Wiki (works very well), a Google site, or even a three-ring binder (not recommended). Anything that you can do to convert the oral traditions of your team(s) into written procedures becomes a boon to both your company and your hard-working staff.

As a bonus, once you have those written procedures for troubleshooting IT operations events, you are 90% of the way to setting up Event Enrichments and really turbo charging your IT Operations Mean Time To Repair (MTTR).

If you appreciated this article, you should read the next one:

Implement the Event Enrichment process

デフォルトだと Runbook の最大同時実行数は 50。

c:\Program Files (x86)\Microsoft System Center 2012\Orchestrator\Management Server\aspt.exe <RunbookServerName> <MaximumRunningRunbooks>

<RunbookServerName> : 変更対象の Runbook Server のホスト名を指定します。すべての Runbook を変更する場合には "*" を指定します。 <MaximumRunningRunbooks> : 設定する Runbook 同時実行数の上限を指定します。最大で 1000 まで指定できます。