Background:
Every once in a while you get an Error event in the Operations Manager event log, with event ID 10303. Here is an example:
Log Name: Operations Manager
Source: Health Service Modules
Date: 9/18/2014 9:16:51 AM
Event ID: 10303
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: yourcomputer.domain.com
Description:
The Microsoft Operations Manager Expression Filter Module failed to process a data item and dropped it.
Error: 0x80004005
One or more workflows were affected by this.
Workflow name: UIGeneratedMonitor73d5bd10284341c0936c93f5f43409cf
Instance name: YourApplicationName
Instance ID: {F4ED789B-F363-A9F6-F4F5-09CC14FB9CDC}
Management group: YourManagementGroup
Resulting Behavior/Symptoms:
What ends up happening is you do not get alerts triggered based on monitors watching an Event Log for specific events, after the above event is logged. All other monitors work, just the specific one mentioned in Workflow name (you can figure out which one it actually is, if you export your MPs and search for that string, although I am sure there probably is a PowerShell command or SQL query you can run to get that as well, like one of these, I just have not needed to)
Cause:
There is a bug in the filtering module’s parser that chokes on certain events. That happens on SCOM 2007 and 2012 (pre-R2, have not tested on R2, but that may have it as well). When the parser chokes, it unloads the monitor and you effectively get a silent death of a monitor. This happens when you have the following in place:
You are monitoring event log for events, with a filter such as “Event Level Equals Error AND Event Source Equals MyService AND Parameter 1 Matches Wildcard *some string here*”. The key word here is Matches Wildcard. Apparently that is where the parser chokes.
Workaround/Solution:
You can do one of two things (not both, although it probably won’t cause an issue if you did):
1. Modify your monitor to read (as in the above example) “Event Level Equals Error AND Event Source Equals MyService AND Parameter 1 Contains ‘some string here'”
2. Edit the registry on the agent computer:
Create the following key: HKLM\Software\Microsoft\Microsoft Operations Manager\v3\Modules\Global\ExpressionFilter
Then create a DWORD value under this key: MaxExpressionDepth
This value can be between 500 and 100000, the default is/should be 2000.
Test.
Hope this helps someone… 🙂
Are you on the bus? 