Object Replication Manager (objreplmgr) on child site failed to insert .CID and .SDM files

Posted 27th June by Vincent Goh

I was presented with an issue whereby software updates (configuration items) were not replicating from the central parent site down to a particular child primary site. The symptoms of this being that if a certain update list, update package, or update deployment contained one of the missing updates (CI_items) then the effected object(s) would not appear in the console, and therefore I cannot deploy the effected update(s) at the child site

In this scenario I found a backlog of .CID files in the inbox folder below, each .CID file representing an update which couldn’t be processed.

\inboxes\objmgr.box\INCOMING\retry

The retry is attempted 100 times and then the .CID is place in the ‘bad’ folder

\inboxes\objmgr.box\INCOMING\retry\BAD
When SMS_OBJECT_REPLICATION_MANAGER attempts to process the effected CI’s the following is logged to the objreplmgr.log which indicates the failure:
Processing replication file d:\Microsoft Configuration Manager\inboxes\objmgr.box\INCOMING\RetryGL_42586.CID in retry.

Failed to insert Object 6298a02f-0a6a-4f34-b832-08059b682b63 from replication file d:\Microsoft Configuration Manager\inboxes\objmgr.box\INCOMING\RetryGL_48796.CID.

On to the resolution, I found that running the following 6 SQL queries on the effected child sites resolved the issue:

Delete from CI_ConfigurationItems Where CIType_ID in (1, 6, 8);
Update CI_SDMPackages set IsDeleted = 1 where SourceSite = ‘ZZZ’;
Delete from CI_ComplianceHistory where isdetected = 1
Delete from CI_Compliancehistory where isdetected = 0
Delete from CI_SDMPackages where is deleted =1 and sourecesite = ‘ZZZ’
Exec sp_DeleteOldSDMPackageData 0;

* note: replace ZZZ with the site code of your central site (or the active SUP which is the furthest upstream’)

Running the above queries will purge Software Updates data from the effected child site.

You should then wait about 30 minutes and then restart the following services on the effected child site:

SMS_EXECUTIVE
SMS_COMPONENT_MANAGER

The final step is to initiate a full site replication, this can be done using the heirarchy maintenance tool (syncchild option) or by place a file called .SHA * in the objmgr.box on the parent site.

* .SHA – replace with the 3 digit site code of the desired child site you wish to replicate.

Allow up to 24 hours for the site replication to complete, when it finishes you should have a fully matching compliment of updates on the central site and child primary site.

You can monitor the objreplmgr.log for successful insertion of .CID and .SDM items, and you should also find that the folder \inboxes\objmgr.box\INCOMING\retry does not contain any items for retry.

Posted 27th June by Vincent Goh

References
http://eskonr.com/2012/12/sccm-configmgr-failed-to-insert-sms-package-because-sdm-type-content-is-not-present-in-the-ci_contents-table-will-try-later/

http://blogs.msdn.com/b/steverac/archive/2011/04/10/software-updates-internals-mms-2011-session-part-i.aspx

http://blogs.msdn.com/b/steverac/archive/2011/04/16/software-updates-internals-mms-2011-session-part-ii.aspx

Sending content to remote DP failing and seeing Retrying rename in smsdpprov.log

Remote DP failing and seeing Retrying rename when distributing a package to the DP

Resolution – Symantec Endpoint Protection was casing the files to be locked. Updated to latest version of SEP

[10CC][Sun 08/25/2013 19:56:35]:Retrying rename of D:\SCCMContentLib\FileLib\TCC100000A to D:\SCCMContentLib\FileLib\AB2D\AB2D1AB92B3943B66AAC8538C8C655BEC2B75093E6B59B9AEC9287A476D3384C
[10CC][Sun 08/25/2013 19:56:39]:Retrying rename of D:\SCCMContentLib\FileLib\TCC100000A to D:\SCCMContentLib\FileLib\A11E\A11E3E94237BECC1EBB3C4CEC5AB80AA9AD40728446770D478D59D04A52FB216
[10CC][Sun 08/25/2013 19:56:42]:Retrying rename of D:\SCCMContentLib\FileLib\TCC100000A to D:\SCCMContentLib\FileLib\A644\A6446BEB9786181D3C328F07A3DD20AC1DCA30BD5A750A851A70EA1EF13DCC30
[10CC][Sun 08/25/2013 19:56:46]:Retrying rename of D:\SCCMContentLib\FileLib\TCC100000A to D:\SCCMContentLib\FileLib\39F3\39F3A89C269B4FF50937BABEB664810EF4B68144532C5E52B5E4B4EB7352C282
[10CC][Sun 08/25/2013 19:56:49]:Retrying rename of D:\SCCMContentLib\FileLib\TCC100000A to D:\SCCMContentLib\FileLib\B0E4\B0E4CF7E8B30EC1197208B806C8FE8ACFFE728319BFF0D4F956D6FB91C17FCC0
[10CC][Sun 08/25/2013 19:56:52]:Retrying rename of D:\SCCMContentLib\FileLib\TCC100000A to D:\SCCMContentLib\FileLib\E393\E393BB581F000202DEF3E23635515CFBE79DDAD3EA54E995008CCA08C364BED5
[10CC][Sun 08/25/2013 19:56:56]:Content ‘FB7B41B9-08FF-40BE-9CFA-83664BF41A25’ for package ‘CAS00363’ has been added to content library successfully
[10CC][Sun 08/25/2013 19:57:14]:Failed to delete package file(s) from \\server*\SMSPKGD$\CAS00363. Error = 145
[10CC][Sun 08/25/2013 19:57:22]:CreateDirectoryW failed for \\?\D:\SMSPKGD$\CAS00363\E70CE9B3-91BE-4009-927C-F42CB7771917\amd64
[10CC][Sun 08/25/2013 19:57:22]:CreateFolder failed; 0x800700b7
[10CC][Sun 08/25/2013 19:57:22]:CContentDefinition::ExpandContentDefinitionItems failed; 0x800700b7
[10CC][Sun 08/25/2013 19:57:22]:Failed to expand content ‘FB7B41B9-08FF-40BE-9CFA-83664BF41A25’ for package ‘CAS00363’ to share ‘\\server*\SMSPKGD$\CAS00363’. Error code: 183

DRS Troubleshooting

Articles which are useful for learning and troubleshooting DRS issues

How to Check Inbox Backlog in SCCM ConfigMgr 2012

SCCM ConfigMgr 2012 Site to Site replication and SQL Replication Guide

SQL Replication Troubleshooting Guide for ConfigMgr SCCM 2012

Tips for troubleshooting SC 2012 Configuration Manager Data Replication Service (DRS)

DRS Initialization In Configuration Manager 2012

Custom SQL Replication reports for System Center 2012 Configuration Manager

DRS Unleashed 

ConfigMgr 2012: DRS and SQL service broker certificate issues

The “Install Application” task sequence fails in System Center 2012 Configuration Manager SP1

Consider the following scenario:•You install a Microsoft System Center 2012 Configuration Manager Service Pack 1 (SP1) primary site server.
•You use the Install Application task sequence step to install two applications by using the Install applications according to dynamic variable list setting.
•You create a deployment for the sequence and then deploy the sequence to the target collection.

In this scenario, the task sequence fails.

Link to hotfix

Branch DP’s and Hash Values

How can I find the package hash value in WMI on a BDP:

1. Open Webemtest and connect to root\ccm

2. Select Enum Classes, click the recursive radio button press OK

3. Scroll down to ccm_peerdpagent and double click

4. select ccm_peerdp_job and click instances.

5. click a package packageID listed.

6. Scroll Down to the hash value.

Compare this value in WMI to the Hash value that you find on the Distribution Point for the same package. To get the hash value for the distribution point content use a tool called hashdir.exe. (This is an internal tool so you would need to contact your PFE DSE for it). Compare the two hashes above with the 40 character Sha1 hash that is listed in the “NewHash” column of the SMSPackages Table of the database. Obviously these should all be the same.

What is the BDPTmpWrkFldr?

There are three folders that come into play here, they are listed below in this order:

Needs Folder

Assemble Folder

Content Folder

The flow from one folder to the next is like this:

Flow—–>Needs Folder—–>Assemble Folder—->Content Folder

1. BDPTmpWrkFldr\PDP5DB1.tmp – AKA “Needs” folder used by the Content Transfer Manager where deltas and needs files are downloaded and saved to, before being compiled and sent to the Assemble folder, this folder is deleted after it has been utilized.

2. BDPTmpWrkFldr\PDP5D91.tmp – AKA “Assemble” folder, Delta and needs files are brought in to the Assemble folder once downloaded to the Needs folder and then merged (or assembled). Once we have hash successes here we then copy to SMSPKG$.

3. SMSPKG$ – AKA “Content” folder, this is where the content is actually stored for client retrieval after the deltas have been merged and hash has been verified from content in the Assemble folder.

In this Short example from CTM log we will find the Needs folder is referenced first then the Assemble folder. The merge delta operation occurs in the next log entry in the Assemble folder. Both folders are created with a naming convention of PDP****.tmp

CCMRDC] GenerateTargetFile() called with OldFile as “F:\SMSPKGF$\UP10004A\Patches\WINDOWS_VISTA\Windows6.0-KB960225-x86.msu”,

NeedsFile as “F:\BDPTmpWrkFldr\PDP5DB1.tmp\work\Patches\WINDOWS_VISTA\Windows6.0-KB960225-x86.msu .needs”,

0 Deltafiles and AssembledFile as “F:\BDPTmpWrkFldr\PDP5D91.tmp\Patches\WINDOWS_VISTA\Windows6.0-KB960225-x86.msu”

CMergeDeltaOperation::_ProcessFile – Created merged file F:\BDPTmpWrkFldr\PDP5D91.tmp\Patches\WINDOWS_VISTA\Windows6.0-KB960225-x86.msu. ContentTransferManager 7/28/2009 5:18:16 PM 7380 (0x1CD4)

What’s the difference between “Hash failed for package” and a “PDPHashMismatchEvent”?

“Hash failed for package” is shown when the current version of the ALREADY LOCALLY stored package does not match the stored package hash. If we get this message it indicates for the BDP to NOT download a delta but instead download the full current needed version of the package. Redownloading content is not so much as a “retry” but instead an indicator that we need the full content and not just the deltas.

Example of Hash Failed for Package:

Checking provisioning status for package = UP10004A, version = 3. PeerDPAgent 2/7/2009 1:20:46 PM 7660 (0x1DEC

Hash failed for package UP10004A. Redownloading content. PeerDPAgent 2/11/2009 5:43:33 PM 6312 (0x18A8)

Package UP10004A in state ‘UpdatingPackage’. PeerDPAgent 2/11/2009 5:43:33 PM 6312 (0x18A8)

Raising event:

[SMS_CodePage(437), SMS_LocaleID(1033)]

instance of PDPDownloadStartedEvent

{

ClientID = “GUID:BF41F222-3D9A-4AD6-ABDF-A8E301068C76”;

DateTime = “20090211234333.305000+000”;

MachineName = “WAK-FS-01”;

PackageID = “UP10004A”;

ProcessID = 2212;

SiteCode = “UP1”;

SourceVersion = 4;

ThreadID = 6836;

};

PeerDPAgent 2/11/2009 5:43:33 PM 6836 (0x1AB4)

Package UP10004A in state ‘Downloading’. PeerDPAgent 2/11/2009 5:43:33 PM 6836 (0x1AB4)

Download complete for CTM job {C2D4DFC8-670F-44A6-8332-88A0F1D3710B}, downloaded KB 8291 PeerDPAgent 2/11/2009 5:44:36 PM 7396 (0x1CE4)

Package UP10004A in state ‘DownloadComplete’. PeerDPAgent 2/11/2009 5:44:41 PM 7396 (0x1CE4)

Raising event:

[SMS_CodePage(437), SMS_LocaleID(1033)]

instance of PDPDownloadSuccessEvent

{

ClientID = “GUID:BF41F222-3D9A-4AD6-ABDF-A8E301068C76”;

DateTime = “20090211234450.011000+000”;

MachineName = “WAK-FS-01”;

PackageID = “UP10004A”;

ProcessID = 2212;

SiteCode = “UP1”;

SourceVersion = 4;

ThreadID = 6028;

};

PeerDPAgent 2/11/2009 5:44:50 PM 6028 (0x178C)

Package UP10004A in state ‘HashContentSuccess’. PeerDPAgent 2/11/2009 5:44:50 PM 6028 (0x178C)

Package UP10004A in state ‘HostingComplete’. PeerDPAgent 2/11/2009 5:44:58 PM 7396 (0x1CE4)

Package UP10004A in state ‘Succeeded’. PeerDPAgent 2/11/2009 5:44:58 PM 1800 (0x0708)

Note the version number above is “3” Before we update the package above from 3 to 4 we rehash the content. The Hash failed for content so instead of grabbing a Delta update we download the full 4 content.

A “PDPHashMismatchEvent” is shown when the content that we downloaded from the DP fails the hash match immediately after the content was downloaded. In the example below we did in fact attempt to redownload the package however the hash once again failed.

Here is an example of “PDPHashMismatchEvent”:

Package UP100035 in state ‘Downloading’. PeerDPAgent 4/27/2009 7:30:28 PM 6028 (0x178C)

Download complete for CTM job {772B7EAF-D420-4D6B-B421-45B7226A2D0B}, downloaded KB 4961 PeerDPAgent 4/27/2009 7:30:50 PM 1908 (0x0774)

Package UP100035 in state ‘DownloadComplete’. PeerDPAgent 4/27/2009 7:30:50 PM 1908 (0x0774)

Raising event:

[SMS_CodePage(437), SMS_LocaleID(1033)]

instance of PDPHashMismatchEvent

{

ClientID = “GUID:BF41F222-3D9A-4AD6-ABDF-A8E301068C76”;

DateTime = “20090428003051.129000+000”;

MachineName = “WAK-FS-01”;

PackageID = “UP100035”;

ProcessID = 672;

SiteCode = “UP1”;

SourceVersion = 5;

ThreadID = 5792;

};

PeerDPAgent 4/27/2009 7:30:51 PM 5792 (0x16A0)

Package UP100035 in state ‘HostingIncomplete’. PeerDPAgent 4/27/2009 7:30:51 PM 5792 (0x16A0)

Raising event:

[SMS_CodePage(437), SMS_LocaleID(1033)]

instance of PDPPkgDeleteEvent

{

ClientID = “GUID:BF41F222-3D9A-4AD6-ABDF-A8E301068C76”;

DateTime = “20090428013514.996000+000”;

MachineName = “WAK-FS-01”;

PackageID = “UP100035”;

ProcessID = 672;

Share = “”;

SiteCode = “UP1”;

SourceVersion = 5;

ThreadID = 4724;

};

PeerDPAgent 4/27/2009 8:35:14 PM 4724 (0x1274)

We did in fact retry again and failed the hash again at 10:40:51

Package UP100035 in state ‘DownloadComplete’. PeerDPAgent 4/27/2009 10:40:51 PM 5244 (0x147C)

Raising event:

[SMS_CodePage(437), SMS_LocaleID(1033)]

instance of PDPHashMismatchEvent

{

ClientID = “GUID:BF41F222-3D9A-4AD6-ABDF-A8E301068C76”;

DateTime = “20090428034051.189000+000”;

MachineName = “WAK-FS-01”;

PackageID = “UP100035”;

ProcessID = 672;

SiteCode = “UP1”;

SourceVersion = 5;

ThreadID = 4168;

};

PeerDPAgent 4/27/2009 10:40:51 PM 4168 (0x1048)

Package UP100035 in state ‘HostingIncomplete’. PeerDPAgent 4/27/2009 10:40:51 PM 4168 (0x1048)

Why we still do MD5 hash in Config Manager as the SHA1 hash has not been used for content security since SMS 2003 RTM?

We use this value as a quick integrity check as it’s very fast. Something to note is that the older MD5 Hashes are always 32 characters in length and corresponds to the “Hash” column in the SQL Database. Newer (as of SMS 2003 SP1 and Config Mgr) and more secure SHA1 hashes are always 40 characters in length and correspond to the “New Hash” column in the SQL DB.