Skip to main content

SCOM 2012 AppAdvisor Kerberos Issue

Update Sept 23 2012
********************************

This may not work as i have started to see this issue again

*******************************************



I am facing issues while trying to access the AppAdvisor after Installing SCOM 2012
After troubleshooting for a while now i have seen different behaviors and errors that have changed over times.

First Error:


While Trying to access the http://servername/AppAdvisor or http://servername/AppDiagnostics site locally or remotely it gave an error


The event log on the SCOM Server Showed the following


Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          20-08-2012 11:27:41
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SCOMSERVER.domain.com
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 20-08-2012 11:27:41
Event time (UTC): 20-08-2012 08:27:41
Event ID: 5af4a6e0bddd4d7ebf8b78f0870321b9
Event sequence: 6
Event occurrence: 1
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/1/ROOT/AppAdvisor-1-129899248334061251
    Trust level: Full
    Application Virtual Path: /AppAdvisor
    Application Path: E:\Program Files\System Center 2012\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\
    Machine name:

Process information:
    Process ID: 6224
    Process name: w3wp.exe
    Account name: IIS APPPOOL\OperationsManagerAppMonitoring

Exception information:
    Exception type: OleDbCommandException
    Exception message: Exception has been thrown by the target of an invocation.
Command text: SELECT apm.IsInstallCompleted ()
Connection: Provider=SQLOLEDB;Server=;database=OperationsManagerDW;Integrated Security=SSPI;

   at Avicode.AX5.Data.OleDb.OleDb.ExecuteScalar(OleDbCommand oleDbCommand)
   at Avicode.Intercept.SEManager.Core.DBAccess.DB.ExecuteScalar(String query)
   at Avicode.Intercept.SEManager.WebViewer.Pages.Authenticate.Page_Load(Object sender, EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Exception has been thrown by the target of an invocation.
   at Avicode.AX5Base.IdentityThread.Execute(Action action)
   at Avicode.AX5.Data.OleDb.OleDb.ExecuteScalar(OleDbCommand oleDbCommand)

Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'.
   at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
   at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
   at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
   at System.Data.OleDb.OleDbConnection.Open()
   at Avicode.AX5Base.IdentityThread.ActionItem.ExecutionBody(Object actionItem)



Request information:
    Request URL: http://localhost/AppAdvisor/Pages/Authenticate.aspx?ReturnUrl=/AppAdvisor
    Request path: /AppAdvisor/Pages/Authenticate.aspx
    User host address: 127.0.0.1
    User:
    Is authenticated: False
    Authentication Type:
    Thread account name: IIS APPPOOL\OperationsManagerAppMonitoring

Thread information:
    Thread ID: 3
    Thread account name: IIS APPPOOL\OperationsManagerAppMonitoring
    Is impersonating: False
    Stack trace:    at Avicode.AX5.Data.OleDb.OleDb.ExecuteScalar(OleDbCommand oleDbCommand)
   at Avicode.Intercept.SEManager.Core.DBAccess.DB.ExecuteScalar(String query)
   at Avicode.Intercept.SEManager.WebViewer.Pages.Authenticate.Page_Load(Object sender, EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)


Custom event details:

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ASP.NET 4.0.30319.0" />
    <EventID Qualifiers="32768">1309</EventID>
    <Level>3</Level>
    <Task>3</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-08-20T08:27:41.000000000Z" />
    <EventRecordID>3227</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SCOMServer.domain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>3005</Data>
    <Data>An unhandled exception has occurred.</Data>
    <Data>20-08-2012 11:27:41</Data>
    <Data>20-08-2012 08:27:41</Data>
    <Data>5af4a6e0bddd4d7ebf8b78f0870321b9</Data>
    <Data>6</Data>
    <Data>1</Data>
    <Data>0</Data>
    <Data>/LM/W3SVC/1/ROOT/AppAdvisor-1-129899248334061251</Data>
    <Data>Full</Data>
    <Data>/AppAdvisor</Data>
    <Data>E:\Program Files\System Center 2012\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\</Data>
    <Data></Data>
    <Data>
    </Data>
    <Data>6224</Data>
    <Data>w3wp.exe</Data>
    <Data>IIS APPPOOL\OperationsManagerAppMonitoring</Data>
    <Data>OleDbCommandException</Data>
    <Data>Exception has been thrown by the target of an invocation.
Command text: SELECT apm.IsInstallCompleted ()
Connection: Provider=SQLOLEDB;Server=
;database=OperationsManagerDW;Integrated Security=SSPI;

   at Avicode.AX5.Data.OleDb.OleDb.ExecuteScalar(OleDbCommand oleDbCommand)
   at Avicode.Intercept.SEManager.Core.DBAccess.DB.ExecuteScalar(String query)
   at Avicode.Intercept.SEManager.WebViewer.Pages.Authenticate.Page_Load(Object sender, EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Exception has been thrown by the target of an invocation.
   at Avicode.AX5Base.IdentityThread.Execute(Action action)
   at Avicode.AX5.Data.OleDb.OleDb.ExecuteScalar(OleDbCommand oleDbCommand)

Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'.
   at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
   at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
   at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
   at System.Data.OleDb.OleDbConnection.Open()
   at Avicode.AX5Base.IdentityThread.ActionItem.ExecutionBody(Object actionItem)

</Data>
    <Data>http://localhost/AppAdvisor/Pages/Authenticate.aspx?ReturnUrl=/AppAdvisor</Data>
    <Data>/AppAdvisor/Pages/Authenticate.aspx</Data>
    <Data>127.0.0.1</Data>
    <Data>
    </Data>
    <Data>False</Data>
    <Data>
    </Data>
    <Data>IIS APPPOOL\OperationsManagerAppMonitoring</Data>
    <Data>3</Data>
    <Data>IIS APPPOOL\OperationsManagerAppMonitoring</Data>
    <Data>False</Data>
    <Data>   at Avicode.AX5.Data.OleDb.OleDb.ExecuteScalar(OleDbCommand oleDbCommand)
   at Avicode.Intercept.SEManager.Core.DBAccess.DB.ExecuteScalar(String query)
   at Avicode.Intercept.SEManager.WebViewer.Pages.Authenticate.Page_Load(Object sender, EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
</Data>
  </EventData>
</Event>




When checked in the SQL Server event Log  which hosts the Databases for SCOM found


Log Name:      Application
Source:        MSSQLSERVER
Date:          20-08-2012 11:27:40
Event ID:      18456
Task Category: Logon
Level:         Information
Keywords:      Classic,Audit Failure
User:          ANONYMOUS LOGON
Computer:      SQLServer01.domain.com
Description:
Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors. [CLIENT: X.X.X.X]

These events were generated every time a user tried to access the AppAdvisor or AppDiagnostics Site

As i looked online i found this
http://social.technet.microsoft.com/Forums/en-IN/scomapm/thread/1fd63bda-b5ca-4767-823a-606cad1cc11b

Which described some thing like

if user that installed Web Console did not have administrative permissions on DB, security was not correctly configured (you should have got warning about this). You will need to configure it manually.
Run this SQL statement: EXEC [apm].GrantRWPermissionsToComputer N'[LOGIN] 
The statement must be run twice against both the Operations Manager Database and the Data Warehouse Database. Running the SQL statement against the Operations Manager Database is required for Application Diagnostics to work correctly and running the SQL statement against the Data Warehouse Database is required for Application Advisor to work correctly.
If the Operations Manager Database and the Data Warehouse Database are on different servers and the Web Console is installed on one of these servers, the parameters will be different for each server. Local installation means that the Web Console is installed on same server as the database. A remote installation means that the Web Console is installed on a different server than the database. The local and remote parameters are as follows:
  • For local installation, the LOGIN is: IIS APPPOOL\OperationsManagerAppMonitoring
  • For remote installation, the LOGIN is: Domain\MachineName$



But in my case the User who installed SCOM had Full Admin rights on servers and was sysadmin on SQl
So i dint try this.

I tried IISRESET etc but still kept getting the error

I left at that point .

But now when i try to acces the AppAdvisor and AppDignostics site i get a different Event description on the SCOM Server.

I am not sure why the event changed (Only thing i think of was the servers were rebooted for Maintenance)

Anyways the event now is


Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 26-08-2012 13:39:47
Event time (UTC): 26-08-2012 10:39:47
Event ID: 92527397ffd641e0880471bb99d03816
Event sequence: 30
Event occurrence: 1
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/1/ROOT/AppAdvisor-1-129904509814210000
    Trust level: Full
    Application Virtual Path: /AppAdvisor
    Application Path: E:\Program Files\System Center 2012\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\
    Machine name: SCOM01

Process information:
    Process ID: 5428
    Process name: w3wp.exe
    Account name: IIS APPPOOL\OperationsManagerAppMonitoring

Exception information:
    Exception type: WebException
    Exception message: The request failed with HTTP status 401: Unauthorized.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.ReportingService.ListChildren(String Item, Boolean Recursive)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.ReportServiceProvider.Avicode.Intercept.SEManager.Core.Services.ReportingServices.IReportPopulate.PopulateList(State state, SemCore semCore)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.StateAdapter.GetItemsInternal(Func`2 match)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.StateAdapter.GetListDisplayedItems()
   at Avicode.Intercept.SEManager.WebViewer.Modules.ReportServiceSelector.InitListOfReports()
   at Avicode.Intercept.SEManager.WebViewer.Modules.ReportServiceSelector.InitOrUpdateParameters()
   at Avicode.Intercept.SEManager.WebViewer.Modules.ReportServiceSelector.Page_Load(Object sender, EventArgs e)
   at System.EventHandler.Invoke(Object sender, EventArgs e)
   at Avicode.Intercept.SEManager.WebViewer.SemBase.SemPage.OnLoad(EventArgs e)
   at Avicode.Intercept.SEManager.WebViewer.Pages.ReportServices.ReportServicePage.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)



Request information:
    Request URL: http://SCOM01/AppAdvisor/Pages/ReportService/ReportServicePageImpl.aspx?notdefined=1&_r=&_c=g&_pg=312c0194-f693-45d1-a3b9-bd6758dc7e4c&_s=C761E252
    Request path: /AppAdvisor/Pages/ReportService/ReportServicePageImpl.aspx
    User host address: X.X.X.X
    User: ABC\Adminuser
    Is authenticated: True
    Authentication Type: Forms
    Thread account name: IIS APPPOOL\OperationsManagerAppMonitoring

Thread information:
    Thread ID: 3
    Thread account name: IIS APPPOOL\OperationsManagerAppMonitoring
    Is impersonating: False
    Stack trace:    at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.ReportingService.ListChildren(String Item, Boolean Recursive)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.ReportServiceProvider.Avicode.Intercept.SEManager.Core.Services.ReportingServices.IReportPopulate.PopulateList(State state, SemCore semCore)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.StateAdapter.GetItemsInternal(Func`2 match)
   at Avicode.Intercept.SEManager.Core.Services.ReportingServices.StateAdapter.GetListDisplayedItems()
   at Avicode.Intercept.SEManager.WebViewer.Modules.ReportServiceSelector.InitListOfReports()
   at Avicode.Intercept.SEManager.WebViewer.Modules.ReportServiceSelector.InitOrUpdateParameters()
   at Avicode.Intercept.SEManager.WebViewer.Modules.ReportServiceSelector.Page_Load(Object sender, EventArgs e)
   at System.EventHandler.Invoke(Object sender, EventArgs e)
   at Avicode.Intercept.SEManager.WebViewer.SemBase.SemPage.OnLoad(EventArgs e)
   at Avicode.Intercept.SEManager.WebViewer.Pages.ReportServices.ReportServicePage.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)



I asked the Query on technet
http://social.technet.microsoft.com/Forums/en-IN/operationsmanagergeneral/thread/193361cc-361b-4f88-b749-86b11d392d73

Graham pointed to IIS Kerberso double hop Issue

http://weblogs.asp.net/owscott/archive/2008/08/22/iis-windows-authentication-and-the-double-hop-issue.aspx

As i checked the Authentication settings for the AppAdvisor site in IIS , it looks like



I made the changes as per the suggestion from Graham and now the settings look like:




After that AppAdvisor did work for me on the Server Locally and remotely using IP , hostname ,FQDN
Although i have an extra step of Form based Authentication, so you need to enable Pop ups so that you can get the Authentication form shown below.



Hope this Helps

Comments

Popular posts from this blog

Group Policy Object did not apply because it failed with error code '0x80070534 No mapping between account names and security IDs was done

One of our servers was losing the Local Admin settings as we control them using GPO (restricted Groups) We were receiving the following event on this particular server : Event ID 4098 Application Event Log The computer 'Administrators (built-in)' preference item in the 'Servers Local Admins {odjd9DBD-22AF-48EA-ADF5-F42ADE4182hst}' Group Policy Object did not apply because it failed with error code '0x80070534 No mapping between account names and security IDs was done.' This error was suppressed. To fix the issue we deleted all the folders from the following location and rebooted the server C:\ProgramData\Microsoft\Group Policy\History Hope this helps.

iDRAC 7 Shows no Signal in Virtual Console Preview

Recently i logged on to iDRAC for one of our Dell R720 servers,however somehow the console redirection did not work at all. I kept on seeing No Signal on the Virtual console Preview. I tried Rebooting the server but that did not help. Finally i clicked on Reset iDRAC and this did the trick. It may take around 2 -5 minutes before you can access iDRAC page again. I had to power on the server through iDRAC before anything showed up on console. Hope this Helps.

Close Open Files in Isilon

To close Open files in Isilon Sometimes we may have to Close open files on Isilon , This is how we can accomplish it First find all the Open Files using the Following Command: isi_for_array  "isi smb openfiles list" To Narrow the search  we can grep the result  as an example isi_for_array  "isi smb openfiles list" | grep -i abc Once you find the files the 6 digit number is the id for that open file. To close the open file run the following command isi_for_array isi smb openfiles close "123456" -f Hope this helps