Skype for Business Cloud Connector Edition (CCE) - We couldn't reach +1XXXXXXXXXX (404 Not Found)

Today I’m shedding light on the SIP 404 response status code while working with Skype for Business Online’s CloudPBX and Cloud Connector Edition (CCE). The “404 Not Found” error is masked in the Skype for Business Client as “We couldn’t reach +1XXXXXXXXXX” during an outbound PSTN call.

Always view the ms-diagnostics details specific to the SIP response.

This can be done by opening the .UccApilog file from the client’s Tracing folder ( https://support.office.com/en-us/article/Turn-on-error-logs-in-Skype-for-Business-4892fe33-ef40-48e2-9070-5308d2a57c7e) with a text editor or Snooper. Snooper is part of the Skype for Business Server 2015 Debugging Tools ( https://www.microsoft.com/en-us/download/details.aspx?id=47263).
Alternatively, an Administrator could pull the same info using the Get-CsUserSession Cmdlet ( https://technet.microsoft.com/en-us/library/mt715516.aspx ) or even better by using my PowerShell GUI for the same Cmdlet ( /2016/10/get-csusersessiongui)

Let’s look at some scenarios that cause the 404 Not Found response:

First, don’t be alarmed when you see many 404 Not found in the .UccApiLog. These are expected to occur when the Skype for Business client is attempting presence subscription or communication to contacts on external domains that don’t have federation configured or for objects that simply don’t exist in a domain.

User doesn’t exist in your domain: This will give you an ms-diagnostics message: “User does not exist”;destination=" user@domain.com";source="sip.domain.com"
User doesn’t exist in a federated domain: This will give you an ms-diagnostics message: “Previous hop shared address space peer did not report diagnostic information”;Domain="federateddomain.com"; PeerServer="sip.federateddomain.com";source="sip.domain.com"
User from an external domain and that domain is not configured for federation: This will give you an ms-diagnostics message: “Unable to resolve DNS SRV record”;domain="externaldomain.com";dns-srv-result=“NegativeResult”;dns-source=“WireQuery”;source="sip.domain.com"
Inbound PSTN called number that is either unassigned or doesn’t exactly match a User’s Line URI(E.164) in Skype for Business Online: This will give you an ms-diagnostics message: “Previous hop shared address space peer did not report diagnostic information”;Domain="domain.com"; PeerServer="sipfed.online.lync.com";source="sipcce.domain.com"

The above is all expected, but what if everything is configured correctly and you experience a “404 Not Found” during an outbound PSTN Call when using CCE?

With a successful outbound PSTN Call to CCE, there will be a SIP “101 Trying” followed by a SIP “101 Progress Report”. The Progress Report will highlight communication between Skype for Business Online and the CCE, showing the call is being passed to “OutboundRouting” indicated by the “Trying next hop” message. From this point on, we know the Call is at the CCE and any SIP Errors would be between the CCE and your PSTN connectivity.

With the “404 Not Found” the call flow is quite a bit different. There is the same SIP “101 Trying”, but this is immediately followed by the SIP “404 Not Found”. What we find is Skype for Business Online is doing a Reverse Number Lookup (RNL) for the dialed number within your domain, of course this doesn’t match as we’re trying to call the PSTN and the call is terminated. The call never attempts Outbound Routing and is not sent to the CCE. That last statement is very important, don’t burn support cycles on the wrong rabbit hole. Most CCE issues would be indicated by a SIP “504 Server time-out”: /2017/05/sfb-cce-504/.

Digging into the ms-diagnostics message: “User does not exist”;domain="domain.com";source="sipfed.online.lync.com"

Possible fixes:

Confirm the specific Skype for Business Online user has the HybridPSTNSite configured pointing to CCE, using the Get/Set-CsUserPstnSettings: https://technet.microsoft.com/en-us/library/mt670902.aspx. Original CCE documentation just had you set the CCE at the tenant level using: Set-CsTenantHybridConfiguration -PeerDestination. I believe this parameter is going to be deprecated and the better experience is setting the CCE at the user level. Back in November 2016, a couple customers with working users experienced the CSUserPstnSettings was cleared for all their users.
User is classified as Hybrid, CCE does not support hybrid users. I use the following command to list hybrid users on a tenant:

Get-CsOnlineUser | ? {$_.interpretedusertype -like 'hybrid*'} | select displayname,userprincipalname,interpretedusertype

Newly licensed for Cloud PBX, it can take up to 24 hours for a user to be enabled fully: https://support.office.com/en-us/article/Assign-Skype-for-Business-licenses-fd41934d-f2eb-4a1b-89d8-32cb37702b33?ui=en-US&rs;=en-US&ad;=US. I have seen new users have Inbound calling from CCE works within minutes but Outbound calling takes hours to a full day after the Cloud PBX license was applied.
Internal issues in Skype for Business Online, everything looks correct, but some internal replication or communication problem within Skype for Business Online is preventing the settings from being used. Best course of action is to check the Service Health dashboard for known issues and to create an Office 365 Support ticket.