Event ID 10303 in Operations Manager Event Log

Posted in Uncategorized on 2014/09/22 by CRCerr0r

Background:

Every once in a while you get an Error event in the Operations Manager event log, with event ID 10303. Here is an example:

Log Name: Operations Manager
Source: Health Service Modules
Date: 9/18/2014 9:16:51 AM
Event ID: 10303
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: yourcomputer.domain.com
Description:
The Microsoft Operations Manager Expression Filter Module failed to process a data item and dropped it.

Error: 0x80004005

One or more workflows were affected by this.

Workflow name: UIGeneratedMonitor73d5bd10284341c0936c93f5f43409cf
Instance name: YourApplicationName
Instance ID: {F4ED789B-F363-A9F6-F4F5-09CC14FB9CDC}
Management group: YourManagementGroup

Resulting Behavior/Symptoms:

What ends up happening is you do not get alerts triggered based on monitors watching an Event Log for specific events, after the above event is logged. All other monitors work, just the specific one mentioned in Workflow name (you can figure out which one it actually is, if you export your MPs and search for that string, although I am sure there probably is a PowerShell command or SQL query you can run to get that as well, like one of these, I just have not needed to)

Cause:

There is a bug in the filtering module’s parser that chokes on certain events. That happens on SCOM 2007 and 2012 (pre-R2, have not tested on R2, but that may have it as well). When the parser chokes, it unloads the monitor and you effectively get a silent death of a monitor. This happens when you have the following in place:

You are monitoring event log for events, with a filter such as “Event Level Equals Error AND Event Source Equals MyService AND Parameter 1 Matches Wildcard *some string here*”. The key word here is Matches Wildcard. Apparently that is where the parser chokes.

Workaround/Solution:

You can do one of two things (not both, although it probably won’t cause an issue if you did):

1. Modify your monitor to read (as in the above example) “Event Level Equals Error AND Event Source Equals MyService AND Parameter 1 Contains ‘some string here'”

2. Edit the registry on the agent computer:

Create the following key: HKLM\Software\Microsoft\Microsoft Operations Manager\v3\Modules\Global\ExpressionFilter

Then create a DWORD value under this key: MaxExpressionDepth

This value can be between 500 and 100000, the default is/should be 2000.

Test.

Hope this helps someone… 🙂

Advertisements

PowerShell to get remote website’s SSL certificate expiration

Posted in Monitoring, PowerShell, Scripting, Windows on 2014/02/04 by CRCerr0r

I recently needed to put together a PowerShell script that would check the expiration of some external and internal certificates for my company and let me know when they are close to expiring. Since some of the hosts were IP addresses, and some certs were not trusted by the machine running the check, I had to have a way to disable certificate chain validation (equivalent to the curl option -k). There are many ways to get web content in PowerShell, and some are more flexible than others… After some poking around, I put together the script below, combining examples from this post and this post.

$minimumCertAgeDays = 60
$timeoutMilliseconds = 10000
$urls = @(
"https://www.website.com/Login.aspx",
"https://10.1.1.10/myTestPage.aspx"
) #disabling the cert validation check. This is what makes this whole thing work with invalid certs...
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true} foreach ($url in $urls)
{
Write-Host Checking $url -f Green
$req = [Net.HttpWebRequest]::Create($url)
$req.Timeout = $timeoutMilliseconds try {$req.GetResponse() |Out-Null} catch {Write-Host Exception while checking URL $url`: $_ -f Red} [datetime]$expiration = $req.ServicePoint.Certificate.GetExpirationDateString()
[int]$certExpiresIn = ($expiration - $(get-date)).Days $certName = $req.ServicePoint.Certificate.GetName()
$certPublicKeyString = $req.ServicePoint.Certificate.GetPublicKeyString()
$certSerialNumber = $req.ServicePoint.Certificate.GetSerialNumberString()
$certThumbprint = $req.ServicePoint.Certificate.GetCertHashString()
$certEffectiveDate = $req.ServicePoint.Certificate.GetEffectiveDateString()
$certIssuer = $req.ServicePoint.Certificate.GetIssuerName() if ($certExpiresIn -gt $minimumCertAgeDays)
{Write-Host Cert for site $url expires in $certExpiresIn days [on $expiration] -f Green}
else
{Write-Host Cert for site $url expires in $certExpiresIn days [on $expiration] Threshold is $minimumCertAgeDays days. Check details:`n`nCert name: $certName`nCert public key: $certPublicKeyString`nCert serial number: $certSerialNumber`nCert thumbprint: $certThumbprint`nCert effective date: $certEffectiveDate`nCert issuer: $certIssuer -f Red} rv req
rv expiration
rv certExpiresIn
}

Hope it saves someone some time… 🙂

How to change the IP address of the cluster nodes hosting a SQL Server instance

Posted in Microsoft SQL Server with tags , on 2012/03/30 by CRCerr0r

For whatever reason, there doesn’t seem to be a good post on how to change the IP addresses of the physical cluster nodes hosting a clustered SQL instance (or multiple clustered SQL instances). So after piecing together some articles and testing, here are the simple steps:

Setup:

2-node Microsoft Cluster, each node is running Windows Server 2008 R2. In my case I had two SQL 2008 instances, each normally running on one of the nodes, in an ‘Active-Active’ setup.

SQL Instance A – the applications and services belonging to one of the SQL instances (IPs, names, drives, services, etc.)

Node n – each of the nodes

  1. Take SQL Instance A offline on Node 1 (do not fail over, just take it offline)
  2. Change IP address on Network Adapters (in Windows)
  3. In Cluster Administrator, change the IP address of the all SQL resources (Services and applications) of Instance A to new IPs. You have to do this twice for some reason (click Apply, then OK, then open the properties again and re-set the IP – you will know you need to do it, when the IP under the Resource Name says “”IP Address: Address on Cluster Network x” instead of the actual IP address, and when you open the properties, the mask is set to 0.0.0.0). Confirm a new network has been created under Networks and it is setup as Enabled
  4. Bring Instance A online
  5. Repeat 1-4 for the second instance on Node 2 for Instance B
  6. Failover to test

Simple. 🙂

Exit code -1073741502 when you run an Opalis policy with a Run Program module on a Windows Server 2008 R2

Posted in Microsoft, Opalis with tags on 2012/02/18 by CRCerr0r

Running an Opalis policy with a Run Program module against a Windows 2008 R2 server you may get an exit code of -1073741502 when the Run Program module executes. The full output of the module looks like this:

 

Connecting with OpExec service on SERVERNAME… Starting cmd.exe on SERVERNAME…

Executing cmd.exe on SERVERNAME…

cmd.exe started on SERVERNAME with process ID 3496. Waiting for completion…

Process completed. Obtaining the remote execution status… Disconnecting from SERVERNAME… Disconnected Return value: -1073741502; Log status: 16 (Process exited on SERVERNAME with return code -1073741502.)

 

The reason for this is that the Opalis Remote Execution service is setup, by default, to be able to interact with the desktop session (a check mark on the Log On tab of the service). The error code means:

The application failed to initialize properly. Usually indicates that the application has been launched on a Desktop to which current user has no access rights. Another possible cause is that either gdi32.dll or user32.dll has failed to initialize.

 

Unchecking the “Allow service to interact with desktop” corrects the issue.

PowerShell Invoke-SQLcmd cmdlet “Could not find stored procedure”

Posted in PowerShell, Scripting with tags on 2011/07/16 by CRCerr0r

If, in the process of writing a script using Invoke-SQLcmd cmdlet from the SQL 2005/2008 shell you get “Could not find stored procedure” back, but you know what you are executing IS there, or it is not even a stored procedure, check the input. In my case the input T-SQL had quotes around it, which is what choked the command. Remove the quotes from the input string – the issue goes away.

SCOM 2007 Regular Expressions

Posted in Microsoft, Monitoring, SCOM on 2011/06/24 by CRCerr0r

One of the hardest thing to deal with in SCOM over the years (for me) has been understanding how SCOM interprets RegEx. It has been frustrating, to say the least. And for whatever reason, everything else to do with SCOM has been very well documented, expanded by the community and freely available. Not RegEx info. So today, working on a non-related issue, I came across this post that had some really good, concise info on how SCOM handles RegEx. In case the post gets deleted or lost or whatever, here is the excerpt (thanks to Dan Rogers):

Regular expression support in SCOM 2007

Many teams that are authoring management packs may need to include regular expression matching in their discoveries and groups, as well as for pattern matching in expression criteria in monitors and rules.

There are two different types of regular expression support in the SCOM product, and you have to know which element you are working in to choose the correct one.  Specifically, Group membership calculation and expression filters use distinctly different syntaxes for pattern matching.

Group Calculation matching criteria

Group calculation uses PERL regular expression syntax.  By default, the matching is case insensitive, but in the XML you can specify that an expression needs to be case sensitive by way of a special attribute dedicated to specifying that the expression content should be evaluated in a case sensitive way.

Group Calculation is found in your MP whenever you are using the Group Calc module.

The GroupCalc expression has an operator called MatchesRegularExpression that is used to create dynamic group membership based on pattern matching expressions.  The implementation of this operator passes the expression found in the MP XML to the SQL call name dbo.fn_MatchesRegularExpression.  If this call returns 0, the match is false.  If the expression returns 1, the match is true.

GroupCalc also supports two special sub-elements that abstract away a couple of common regex style queries.

GroupCalc sub element

Regex Equivalent

ContainsSubstring ^*{O}.*$                ({O} is replaced by the substring)
MatchesWildcard MP expression Regex Equivalent
? .
* .*
# [0-9]

Table 1:  GroupCalc special functions

Note:  If either of these two special operators are used, the evaluation will always be case sensitive.

Expression Filter matching criteria

Expression filters used in management packs use .NET Regex expression syntax.  A summary of the .NET regular expression syntax elements appears below.  Expression filters are present in your management pack whenever you are using the Expression Eval module.

Construct

SCOM Regex

Any Character

.

Character in Range

[ ]

Character not in range

[^ ]

Beginning of Line

^

End of Line

$

Or

|

Group

( )

0 or 1 matches

?

0 or more matches

*

1 or more matches

+

Exactly N matches

{n}

Atleast N matches

{n, }

Atmost N matches

{ , n}

N to M Matches

{n, m}

New line character

\n

Tab character

\t

Regular expressions via SDK

The SCOM SDK has a Matches criteria operator for filtering objects. This operator use the same functionality as MatchesCriteria in the GroupCalc case explained above.

When using the SDK to construct a criteria expression to find objects in the Ops Manager database, the following syntax elements are valid (see below).  This syntax is useful when creating a criteria expression that includes any of the following elements:

  • Comparison operators
  • Wildcard characters
  • DateTime values
  • Integer to XML Enumeration comparisons

Comparison operators

You can use comparison operators when constructing a criteria expression. The valid operators are described in the following table:

Operator Description Example(s)
=, == Evaluates to true if the left and right operand are equal. Name = ‘mymachine.mydomain.com’
!=, <> Evaluates to true if the left and right operand are unequal. Name != ‘mymachine.mydomain.com’
> Evaluates to true if the left operand is greater than the right operand. Severity > 0
< Evaluates to true if the left operand is less than the right operand. Severity < 2
>= Evaluates to true if the left operand is greater than or equal to the right operand. Severity >= 1
<= Evaluates to true if the left operand is less than or equal to the right operand. Severity <= 3
LIKE Evaluates to true if the left operand matches the pattern that is defined by the right operand. Use the characters in the wildcard table later in this topic to define the pattern. Name ‘LIKE SQL%’Evaluates to true if the Name value is “SQLEngine.”Name LIKE ‘%SQL%’

Evaluates to true if the Name value is “MySQLEngine.”

MATCHES Evaluates to true if the left operand matches the regular expression defined by the right operand. Name MATCHES ‘SQL*05’Evaluates to true if the Name value is “SQL2005.”
IS NULL Evaluates to true if the value of the left operand is null. ConnectorId IS NULLEvaluates to true if the ConnectorId property does not contain a value.
IS NOT NULL Evaluates to true if the value of the left operand is not null. ConnectorId IS NOT NULLEvaluates to true if the ConnectorId property contains a value.
IN Evaluates to trueif the value of the left operand is in the list of values defined by the right operand.

Note
The IN operator is valid for use only with properties of type Guid.
Id IN (‘080F192C-52D2-423D-8953-B3EC8C3CD001’, ‘080F192C-53B2-403D-8753-B3EC8C3CD002’)Evaluates to true if the value of the Id property is one of the two globally unique identifiers provided in the expression.
AND Evaluates to true if the left and right operands are both true. Name = ‘SQL%’ AND Description LIKE ‘MyData%’
OR Evaluates to true if either the left or right operand is true. Name = ‘SQL%’ OR Description LIKE ‘MyData%’
NOT Evaluates to true if the right operand is not true. NOT (Name = ‘IIS’ OR Name = ‘SQL’)

Table 3: SDK comparison operators

Wildcards

The following table defines the wildcard characters you can use to construct a pattern when using the LIKE operator:

Wildcard Description Example
% A wildcard that matches any number of characters. Name LIKE 'SQL%'Evaluates to true if the Name value is “SQLEngine.”Name LIKE '%SQL%'

Evaluates to true if the Name value is “MySQLEngine.”

_ A wildcard that matches a single character. Name LIKE 'SQL200_'Evaluates to true for the following Namevalues:”SQL2000″

“SQL2005”

Note
The expression evaluates to false for “SQL200” because the symbol _ must match exactly one character in the Name value.
[] A wildcard that matches any one character that is enclosed in the character set.

Note
Brackets are also used when qualifying references to MonitoringObject properties. For more information, see Defining Queries for Monitoring Objects.
Name LIKE 'SQL200[05]‘Evaluates to true for the following Namevalues:”SQL2000″

“SQL2005”

The expression evaluates to false for

“SQL2003.”

[^] A wildcard that matches any one character that is not enclosed in the character set. Name LIKE 'SQL200[^05]'Evaluates to truefor”SQL2003.”

The expression evaluates to false for

“SQL2000” and

“SQL2005.”

Table 4:  Wildcard operators used with LIKE operator

DateTime comparisons

When you use a DateTime value in a query expression, use the general DateTime format (“G”) to convert the DateTime value to a string value. For example,

C#

string qStr = “TimeCreated <= ‘” + myInstant.ToString(“G”) + “‘”;

ManagementPackCriteria mpCriteria = new ManagementPackCriteria(qStr);

All date values need to be converted to the G format (GMT) so that valid string comparisons can be made.

Integer value comparison to enumerations

When you use an integer enumeration value in a query expression, cast the enumeration value to an integer. For example,

C#

string qStr = “Severity > ” + (int)ManagementPackAlertSeverity.Warning;

MonitoringAlertCriteria alertCriteria = new MonitoringAlertCriteria(qStr);

Restore TMG 2010 Exported config from one server to another

Posted in ForeFront TMG 2010, Microsoft on 2011/03/04 by CRCerr0r

Imagine this situation:

  • You have a TMG 2010 box (I have tested this with the Enterprise Edition with SP1, Rollup 1 for SP1 and Rollup 2 for SP1)
  • The TMG box is dead or for whatever other reason you have to bring up a brand new TMG server to replace the old one
  • You don’t want to recreate all objects and rules.

This is what you can do:

You need to do a rule export (or if your box is dead, hopefully you did one before) that includes all sensitive information and user permissions. To do this:

  1. Right-click on ‘Forefront TMG (SERVER_NAME)’ and choose Export (Backup)
  2. Click Next
  3. Check BOTH boxes to Export confidential information and user permission settings, put a password in
  4. Give it a path to save to and complete the wizard

 

After you have the config file (it will be pretty large) copy it to the new server. The new server needs to be identical to the old one, down to the naming of the NICs. If you are putting the new server side-by-side with the old one, and both will be online for a bit, the new server must have different IPs, for obvious reasons. On mine, the old one’s IPs ended with .1 on every subnet (except the External one) so I made the new one end with .2 on every subnet.

The next thing you need to do is install TMG. Make sure it is isntalled the same way, including paths (if you installed the old one on D:\Program Files\… the new one must be the same) and patched with the same patches (TMG ones, not so much OS).

Open the XML export file from the old server in Notepad. Do a ‘find and replace’ for every OLD_SERVER_NAME and OLD_SERVER_IP values and replace them with their NEW_SERVER_NAME, NEW_SERVER_IP counterparts.

Import the config file into the new server choosing the ‘Overwrite’ option in the wizard. Restart the ‘Microsoft Forefront TMG Storage’ service (that will restart all TMG services)

If you had Site-to-Site VPN you may have to re-create it on the new server. If you get an error in the event log “The user SYSTEM dialed a connection named xxxxx which has failed. The error code returned on failure is 789.” check the IPs on each endpoint of the Site-ti-Site VPN. If you get an error “The user SYSTEM dialed a connection named xxxxx which has failed. The error code returned on failure is 812.” check the Dial-In properties of the account used for the Site-ti-Site VPN and make sure it is set to Allow.