System Center Orchestrator 2022 - sync with System Center Service Manager 2022 fails with RsaProtectedConfigurationProvider problem after installing the 2019 web service

 after upgrading System Center Orchestrator from 2019 to 2022, orchestrator itself was working fine.

however, SCSM failed to synchronize after the upgrade. it turns out that SCSM (2019 AND 2022) do NOT work with the 2022 orchestrator web API....
The current solution is to install the orchestrator 2019 console and web service - alongside the orchestrator 2022 install - pointing to the orchestrator 2022 database. and because the standard orchestrator install uses port 81 and 82, I installed the 2019 webservice with port 5001 and 5002 for the console.

sidenote: this was in my DEV environment. I exported my production orchestrator database to the DEV environment, and when upgrading to orchestrator 2022 - pointed to the DB from my production environment. obviously also make sure the service master key is exported and imported into the DEV SQL environment. So the orchestrator console was working fine at this point.

but then I was confronted with a sync error - SCSM could still not talk to the orchestrator webservice.

Unable to decrypt Orchestrator config file. Failed to decrypt using provider RsaProtectedConfigurationProvider Error message from the provider The parameter is incorrect.

the way to test is using aspnet_regiis.exe:

first create a copy of the web.config - as the following command will decrypt the connectionStrings section

"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_regiis.exe" /pdf "connectionStrings" "C:\Program Files (x86)\Microsoft System Center\Orchestrator\Web Service\Orchestrator2012"

that command failed with the same error.

so - the connectionStrings section is encrypted/decrypted using the key stored in NetFrameworkConfigurationKey.

using a utility keypal (project can be downloaded via github) you can list the keys
https://github.com/doggy8088/KeyPal

ultimately the solution was to export the NetFrameworkConfigurationKey from my production system and then restore it on my DEV server.

so on both prod and dev I ran this command to export:
aspnet_regiis.exe /px "NetFrameworkConfigurationKey" "c:\temp\NetFrameworkConfigurationKey_pcname.xml"

then on the DEV system delete the original key:
aspnet_regiis.exe /pz "NetFrameworkConfigurationKey" 

and import the key from the PROD environment:

aspnet_regiis.exe /pi "NetFrameworkConfigurationKey" "c:\temp\NetFrameworkConfigurationKey_prodpcname.xml"

it puzzles me that it shows 1024 here...?!? but it does not seem to be a problem

I ran the iisreset command and also later restarted the server, but now when I open the webpage of the web service I get a correct response.

and the web api via port 81 is still working too - you can use this powershell script

https://techcommunity.microsoft.com/t5/system-center-blog/a-brand-new-web-console-for-orchestrator-2019/bc-p/3256805/highlight/true#M3526

$OrchURI = 'http://OrchServerFQDN:5001' #Replace 5001 by the WebService port
$JobID = '00000000-0000-0000-0000-000000000000' # Correct job ID of course, you can list all jobs to search by name later on or maybe there can be filtered...

# To list all Runbook Servers
$RunbookServers = @(Invoke-RestMethod -Uri ('{0}/api/RunbookServers' -f $OrchURI) -UseDefaultCredentials | Select -ExpandProperty value | Select -ExpandProperty Name)

# To list all Jobs currently running
$CurrentJobs = Invoke-RestMethod -Uri ('{0}/api/Jobs?$filter=Status%20in%20(%27Running%27,%27Pending%27)&$expand=Runbook($select=Name),RunbookInstances' -f $OrchURI) -UseDefaultCredentials | Select -ExpandProperty value

this works fine !

so you can use the 2022 web api via powershell to kick off a runbook.

Get-SCSMObjectHistory commandlet obsolete in SCSM 2016 - an alternative

I found out my script to figure out the "Created By" value from a Service Request in SCSM was no longer functional.


it used to be :

$req = get-scsmclassinstance -class $cl -Filter "Id -eq $id"
$inf = Get-SCSMObjectHistory -Object $req
$inf.History[0].UserName

The command SCMSObjectHistory does no longer exist in SCSM


here is how to achieve the same again:

$cl = get-SCSMClass -displayName "Service Request"
$req = get-scsmclassinstance -class $cl -Filter "Id -eq 'SR2577'"
$Relations = Get-SCSMRelationshipInstance -SourceInstance $req -TargetInstance $req
foreach ($Relation in $Relations)
{
 if ((Get-SCSMRelationship ($Relation.RelationshipId)).DisplayName -eq "Created By User")
 {
  $ReqByName = ($relation.TargetObject).DisplayName
  break  #no need to continue the loop since there is only one Created By relation !
 }
}

so $req  contains the details of the service request again
and then you find all the relations for that request.
now step through all relations (all I get is a list of GUID's) but you can show the RelationsshipID, and if you feed that into the get-SCSMRelationship, you can see what relationship it is.
in my case I am only interested in the Created By User, so I step thru it till i find that one, and then I extract the display name of the user who created the service request.


you can do $Relations.TargetObject  and it will show a list of all related target objects - it may help you to find what you're looking for :)
I had to figure this out by trial on error as I could not find anything about it.


actually,

 

Reinstall SCOM 2016 agent without APM as task in SCOM

in some occasions you don't want to have the APM module installed at all because it can break IIS - if one of the Application Pools is using .NET 2.0 (Technet article)


pushing the agent via SCOM will always install also the APM module.
so I created a script which will uninstall the SCOM agent, and then reinstall it without APM.
reason for the uninstall is, I tried to do a repair as they suggest, however, the repair fails - complaining it is missing some data - I presume this is related to the fact that after installing the agent UR1 was applied, so the MOMAgent.msi that I was using is the base version without UR1.


obviously if you use the SCOM agent to reinstall itself, it has to launch a script which is independent of the SCOM agent, so the process uses a VBscript which creates a commandfile and then registers a runonce scheduled task to run 2 minutes later....


I have registered the task against the Agent class



The script task requires 2 parameters :

PrimaryManagementGroupName \\ManagementServername\Agentmanagement\amd64

(I have shared out the AgentManagement folder to everyone read-only)





first find in the registry where the uninstall string is located, and /I or /X are replaced with /qn /X
then scan the registry for the installed management groups, and store them into an array


MGNames(x), MGservers(x), MGports(x)


so the script collects that information, and then creates a batch file (first as txt file).
you have to create it as txt because security does not permit to create a .cmd directly....


the batchfile contains the following steps (each separated by a ping command as "sleep")

1. copy Agent install files
2. uninstall agent (UninstallString)
3. Install agent with PrimaryManagementgroup
   the management server is pulled from the registry prior to uninstall
4. install URx update
5. if detected, a VBscript is executed to add it to the extra management groups
6. delete the folder (rd /s /q folder) and it's contents

besides the batchfile, if the agent was registered to more than 1 management group, for every extra management group, it creates a VBscript to register the extra management group


so this is what the batchfile looks like that is created by the task


xcopy \\MGserver\Agentmanagement\amd64\* C:\windows\temp\amd64 /Y /R
ping 1.1.1.1 -n 10 /w 1000 > nul
MsiExec.exe /qn /X {742D699D-56EB-49CC-A04A-317DE01F31CD}
ping 1.1.1.1 -n 70 /w 1000 > nul
msiexec.exe /qn /i c:\windows\temp\amd64\Momagent.msi USE_SETTINGS_FROM_AD=0 ACTIONS_USE_COMPUTER_ACCOUNT=1 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=MGgroup MANAGEMENT_SERVER_DNS=MGServer NOAPM=1 SECURE_PORT=MGport AcceptEndUserLicenseAgreement=1
ping 1.1.1.1 -n 70 /w 1000 > nul
msiexec.exe /p c:\windows\temp\amd64\KB3190029-amd64-Agent.msp
ping 1.1.1.1 -n 70 /w 1000 > nul
cscript.exe C:\windows\temp\amd64\AddMGgroup0.vbs
ping 1.1.1.1 -n 20 /w 1000 > nul
rd /s /q C:\windows\temp\amd64




and this is what the VBscript looks like that is created by the task:


Set objMSConfig = CreateObject("AgentConfigManager.MgmtSvcCfg")
Call objMSConfig.AddManagementGroup(MGname,MGServer,MGport)
If Err.number <> 0 Then
 wscript.echo ("Failed to add SCOM")
Else
 Set oShell = WScript.CreateObject("WScript.Shell")
 Set oAPI = CreateObject("MOM.ScriptAPI")
 set oShellEnv = oShell.Environment("Process")
 computerName = oShellEnv("ComputerName")
 strCommand = "cmd /c net stop HealthService & cmd /c net start HealthService"
 Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2")
 Set objProcess = objWMIService.Get("Win32_Process")
 Set objProgram = objProcess.Methods_("Create").InParameters.SpawnInstanceobjProgram.CommandLine = strCommand
 Set strShell = objWMIService.ExecMethod("Win32_Process", "Create", objProgram)
 wscript.echo ("Management Group SCOM has been added")
 WScript.echo "Restarting SCOM Health Service on " & computerName
End If








so the above 2 files are created by the following task in SCOM (run at the agent)


on error resume next
Const HKLM = &H80000002 'HKEY_LOCAL_MACHINE
Dim oParams, strComputer, strKey, sourcefolder, Destinationfolder, VBfilename, CMDFile, delay
Dim FSO, objReg, strSubkey, arrSubkeys, wshShell,MGServer, MGServer1
Dim DisplayName, UninstallString, InstallString
Set oParams = WScript.Arguments
PrimaryMG = trim(oParams(0)) 'Management Group that should be registered again
AgentSourcePath = trim(oParams(1)) 'path to share and folder where Agent Install files can be found
Set oShell = wscript.CreateObject ("wscript.Shell")
set fs = CreateObject("Scripting.FileSystemObject")
strWindows = oShell.ExpandEnvironmentStrings("%windir%")
OutFolder = strWindows & "\temp\amd64"
Outputfile = strWindows & "\temp\amd64\OpsNoAPM.txt"
oShell.LogEvent 4, "OpsNoAPM.cmd: Going to create " & Outputfile
strComputer = "."
strKey = "SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\"
'Get WMI object
Set objReg = GetObject("winmgmts://" & strComputer & "/root/default:StdRegProv")
objReg.EnumKey HKLM, strKey, arrSubkeys
'Loop registry key to find uninstall string for SCOM Agent
For Each strSubkey In arrSubkeys
 objReg.GetStringValue HKLM, strKey & strSubkey, "DisplayName" , DisplayName
 If Instr(DisplayName,"Microsoft Monitoring Agent") > 0 Then
  objReg.GetStringValue HKLM, strKey & strSubkey, "UninstallString", UninstallString
  if (instr(UninstallString,"/I") > 0 ) then
   UninstallString = replace(UninstallString,"/I","/qn /X ")
  Elseif (instr(UninstallString,"/X") > 0 ) then
   UninstallString = replace(UninstallString,"/X","/qn /X ")
  Else
   wscript.echo "no uninstall string found"
   wscript.quit
  end if
 End If
Next
'loop registry to retrieve all management groups
strKey = "SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Agent Management Groups\"
Set objReg = GetObject("winmgmts://./root/default:StdRegProv")
objReg.EnumKey HKLM, strKey, arrSubkeys
Dim arrMGGroups()
Dim arrMGservers()
Dim arrMGPorts()
size = Ubound(arrSubkeys)
Redim arrMGGroups(size)
Redim arrMGservers(size)
Redim arrMGPorts(size)
cnt = 0
For Each strSubkey In arrSubkeys
 arrMGGroups(cnt) = strSubkey
 objReg.GetStringValue HKLM, strKey & strSubkey & "\Parent Health Services\0", "NetworkName", MGServer
 objReg.GetDWORDValue HKLM, strKey & strSubkey & "\Parent Health Services\0", "Port", MGPort
 arrMGservers(cnt) = MGServer
 arrMGPorts(cnt) = MGPort
 cnt = cnt + 1
next

if NOT isnull(UninstallString) then
 if fs.FolderExists(OutFolder) then
  if fs.FileExists(Outputfile) then fs.DeleteFile(OutputFile)
  if fs.FileExists(ResultFile) then fs.DeleteFile(ResultFile)
 Else
  fs.CreateFolder(OutFolder)
 End if
'Create batchfile to uninstall+install+patch+register extra MG group(s)
 Set out = fs.CreateTextFile(Outputfile)
 out.writeline "xcopy " & AgentSourcePath & "\* " & outFolder & " /Y /R"
 out.writeline "ping 1.1.1.1 -n 10 /w 1000 " & chr(62) & " nul"
 out.writeline UninstallString
 out.writeline "ping 1.1.1.1 -n 70 /w 1000 " & chr(62) & " nul"
 'register back with the same management server
 for x = 0 to size
  if arrMGGroups(x) = PrimaryMG then
   MGServer1 =  arrMGservers(x)
   MGPort = arrMGPorts(x)
  end if
 next
 out.writeline "msiexec.exe /qn /i c:\windows\temp\amd64\Momagent.msi USE_SETTINGS_FROM_AD=0 ACTIONS_USE_COMPUTER_ACCOUNT=1 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=" & PrimaryMG & " MANAGEMENT_SERVER_DNS=" & MGServer1 & " NOAPM=1 SECURE_PORT=" & MGPort & " AcceptEndUserLicenseAgreement=1"
 out.writeline "ping 1.1.1.1 -n 70 /w 1000 " & chr(62) & " nul"
 out.writeline "msiexec.exe /p c:\windows\temp\amd64\KB3190029-amd64-Agent.msp"
 out.writeline "ping 1.1.1.1 -n 70 /w 1000 " & chr(62) & " nul"
 'create VBscript that will register any other management groups and add line to commandfile to execute the VBscript
 for x = 0 to size
  if arrMGGroups(x) <> PrimaryMG then
   VBfilename = outfolder & "\AddMGgroup" & x
   'create unique VBfile to register extra MGgroup
   AddMGgroup arrMGGroups(x), arrMGservers(x), arrMGPorts(x), VBfilename
   out.writeline "cscript.exe " & VBfilename & ".vbs"
   out.writeline "ping 1.1.1.1 -n 20 /w 1000 " & chr(62) & " nul"
  end if
 next
 out.writeline "rd /s /q " & OutFolder
 out.close
 wscript.sleep(1000)
 'rename text file to .CMD
 CMDFile = Replace(right(Outputfile,len(Outputfile) - instrrev(Outputfile,"\")),"txt", "cmd")
 oShell.run "CMD /C rename " & Outputfile & " " & CMDFile, 0, true
 wscript.echo "Renamed " & outputfile & " to " & CMDFile
 delay = 3
 oShell.run "cmd /c %windir%\system32\schtasks.exe /create /RU SYSTEM /RL HIGHEST /V1 /Z /F /sc ONCE /st " & formatdatetime(dateadd("N",delay,now()),VBShortTime) & " /TN SCOMAgentUpdate /tr " & OutFolder & "\" & CMDFile
 info = "OpsNoAPM.cmd: created OpsNoAPM.cmd - it's scheduled to run at " & formatdatetime(dateadd("N",delay,now()),VBShortTime)
 oShell.LogEvent 4, info
 wscript.echo info
Else
 info = "OpsNoAPM.cmd: no registry entry found with the install location of the SCOM agent"
 oShell.LogEvent 4, info
 wscript.echo info
end If
Set oShell = Nothing
set out = Nothing
set fs = Nothing


'subroutine to create VBscript file to register an extra management group
sub AddMGgroup (MGName, MSNAme, MGPort,VBfile)
 dim VBf
 Set VBf = fs.CreateTextFile(VBfile & ".txt")
 VBf.writeline "Set objMSConfig = CreateObject(""AgentConfigManager.MgmtSvcCfg"")"
 VBf.writeline "Call objMSConfig.AddManagementGroup(" & MGName & "," & MSNAme & "," & MGPort & ")"
 VBf.writeline "If Err.number <> 0 Then"
 VBf.writeline " wscript.echo (""Failed to add " & MGName & """)"
 VBf.writeline "Else"
 VBf.writeline " Set oShell = WScript.CreateObject(""WScript.Shell"")"
 VBf.writeline " Set oAPI = CreateObject(""MOM.ScriptAPI"")"
 VBf.writeline " set oShellEnv = oShell.Environment(""Process"")"
 VBf.writeline " computerName = oShellEnv(""ComputerName"")"
 VBf.writeline " strCommand = ""cmd /c net stop HealthService & cmd /c net start HealthService"""
 VBf.writeline " Set objWMIService = GetObject(""winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2"")"
 VBf.writeline " Set objProcess = objWMIService.Get(""Win32_Process"")"
 VBf.writeline " Set objProgram = objProcess.Methods_( _"
 VBf.writeline " ""Create"").InParameters.SpawnInstance_"
 VBf.writeline " objProgram.CommandLine = strCommand"
 VBf.writeline " Set strShell = objWMIService.ExecMethod(""Win32_Process"", ""Create"", objProgram)"
 VBf.writeline " wscript.echo (""Management Group " + MGName + " has been added"")"
 VBf.writeline " WScript.echo ""Restarting SCOM Health Service on "" & computerName"
 VBf.writeline "End If"
 VBf.close
 vbFile1 = right(VBFile,len(VBFile) - instrrev(VBFile,"\")) & ".vbs"
 wscript.echo "CMD /C rename " & VBfile & ".txt " & VBfile1
 'Rename text file to .VBS
 oShell.run "CMD /C rename " & VBfile & ".txt " & VBfile1 , 0, true
end sub