SharePoint 2013 Configuring Search to Crawl Web Applications Using Claims and ADFS 2.0

A Web Application can have multiple “Authentication Providers” configured for each zone (Default, Intranet, Extranet, Internet, Custom).  In order for the SharePoint Search to crawl content we need to configure “Windows Authentication” on at least one zone in the Web Application.  While it is possible to configure multiple “Claims Authentication Types” for each zone, doing so raises usability concerns with the authentication process as well as features in SharePoint such as the People Picker.

This article will try to demonstrate how you can use an ADFS 2.0 Trusted Identity Provider for user authentication and still be able to index content in the Web Application with the SharePoint Search Service.

Usability Issues When Configuring Multiple Claims Authentication Types

If multiple “Claims Authentication Types” are configured for a particular zone the user will be presented with a “Sign In” page when first visiting a site hosted in that zone of the web application.  For example the default zone is configured to use both “Windows Authentication” and a “Trusted Identity Provider” in the following image:

image

If you leave this setting in place there are two undesirable side effects:

Side Effect #1

When using the People Picker users will see entries from both authentication providers.  This can get confusing because a user might logon with their ADFS identity, but the content owner may have inadvertently setup permissions with the Windows AD credentials.

  • If both providers are selected you’ll see two entries in the People Picker as it tries to resolve the names:

image

  • With just the “ADFS20” trusted identity provider selected in the “Claims Authentication Types” section, there’s only one entry:

image

Side Effect #2

With two authentication types configured users will be prompted with an additional “Sign-In” page when browsing sites in the web application.  The “Sign In” page will display a drop down box listing both authentication types.  If this is left in place then users will need to know which option to select and what format the logon identity will be in based on the selected option (i.e. <domain>\<user> for “Windows Authentication” and <emailaddress> for “ADFS20”)To complicate matters even further, permissions would need to be setup for both types otherwise “Access Denied” messages might be displayed if the wrong account is selected.

image

How To Resolve the People Picker and Sign In Page Behavior

Both issues can be avoided by configuring the “Default” zone of the web application to use only one authentication provider.  In this example we want to use just the “ADFS20” provider and can do so by unchecking the “Enable Windows Authentication” and “Integrated Windows authentication” check boxes for the “Default” zoneWe are then left with only one authentication provider so the “Sign In” page will no longer be displayed.

image

Search Issue

Notice that when we uncheck the two options for windows authentication a red message is displayed indicating that search will not be able to crawl the web application.  Search requires windows authentication to be selected on at least one zone in the web application and in the current state only the ADFS20 trusted identity provider is configured.  If we were to run a crawl on the web application several “Access denied” errors would be displayed:

image

So at this point we know it’s necessary to have only one authentication provider configured on the default zone in order to get rid of the “Sign In” page and to only show values in the people picker related to the ADFS 2.0 trusted identity provider.  We also know that without windows authentication being setup on at least one zone, that the crawl account will be unable to index the web application.

To work around these issues we can extend the web application to any of the other remaining 4 zones and configure windows authentication for that particular zone (i.e. Intranet).  This will involve the following steps:

  1. Create a DNS A-record for the extended web app
  2. Create a SSL Certificate
  3. Extend the Web Application
  4. Bind Certificate to Extended Web Application
  5. Configure Search Content Source
  6. Configure Server Name Mappings in Search
  7. Execute a Full Crawl
Create a DNS A-record

I always use DNS A-records instead of CNAME’s when working with SharePoint Web Applications just in case Kerberos is configured now or later as an authentication option.

  • Open the DNS console from the “Administrative Tools” folder
  • Expand the “Forward Lookup Zones”
  • Right Click the domain where you want to create the entry and select the “New Host (A or AAAA)…” option

image

  • Enter the Name and IP Address
  • Click Add Host

image

Create SSL Certificate

This step is optional, but typically you want to encrypt all traffic going to the web app even though users will be accessing content through https://portal.2008r2.local instead of extended web address of https://portalsearch.2008r2.local:444 .

The certificate can be self-signed or from an internal/external certificate authority.  Since we outlined earlier how to setup “Active Directory Certificate Services”, the example below will outline how to request a certificate from our internal CA through the Certificates MMC Snap-In.

  • Logon to the WFE server (SP-WFE-1 in our example)
  • Open the MMC console (Start –> Run –> MMC)
  • Add the Certificates Snap-in (File –> Add/Remove Snap-in…)
  • Select “Certificates” followed by the “Add >” button
  • Select the “Computer account” radio button and click “Next >”
  • Leave the “Local computer:” radio button selected and click “Finish”
  • Click “OK”

You should now have the Certificates Snap-In loaded in the MMC and are ready to request the certificate.

  • Navigate to the “Personal” –> “Certificates” folder.
  • Right click the “Certificates” folder and select the “All Tasks” –> “Request New Certificate…” option
  • Click “Next” on the “Before You Begin” screen
  • Select the “Active Directory Enrollment Policy” and click “Next”
  • Select the “Web Server” template followed by clicking the “More Information…” link.
  • From the “Subject” tab, select the “Common name” drop down option located in the “Subject name:” section.
  • Type in the FQDN that was created in DNS previously and will be used to extend the web application in a later step.  In our example we are using “portalsearch.2008r2.local”.
  • Click the “Add >” button to list a CN entry in the right hand column

image

  • Select the “General” tab
  • Enter the optional friendly name and description for the certificate
  • Select the “Private Key” tab
  • Expand the “Key options” section
  • Click the “Make private key exportable” check box

image

  • Click “OK”
  • You should now be presented with an “Enroll” button.  Click it to begin the enrollment process.
  • Review the status message and click “Finish”

image

  • The certificate should now be displayed in the Personal –> Certificates folder of the Certificates console

image

Extend the Web Application

In this step we’ll extend the “Portal” web application and configure the “Intranet” zone to use Windows Authentication.  This will create another site in IIS that points to the same content in the “Portal” web application, but will allow us to configure another authentication provider and access the site through a different URL.

  • Open the “Central Administration” site
  • Click the “Manage web applications” link in the “Application Management” section
  • Highlight the targeted web application (Portal in our example) and click on the “Extend” button in the ribbon bar

image

  • Select the “Create a new IIS web site” radio button and fill in the fields identified below while leaving the defaults for for the others.  Please note that port 443 is already in use by the default zone so we’ll use port 444 for the extended web application.

image

  • Verify the “Public URL” section and that it’s configured to use the “Intranet” zone.  We can use any zone, but for the sake of the example we’ll stick with the “Intranet” zone.

image

  • Click OK to provision the extended web application.  ***Please note that after clicking the “OK” button in Central Administration that the dialogue display won’t immediately go away.  Do not continue to click “OK” and rather just wait a couple of minutes for the action to complete***
  • To verify the action completed successfully highlight the web application and click on the “Authentication Providers” button in the ribbon bar.

image

  • You should now see two entries in the “Authentication Providers” dialogue box.  If you click on the “Intranet” zone link you can verify that windows authentication is the only option selected in the “Claims Authentication Types” section

image

Bind Certificate to the Extended Web Application

At this point we won’t be able to browse to the extended web application and need to bind it to the correct IP address and SSL certificate.

  • On the WFE server, open the “Internet Information Services (IIS) Manager” console from the “Administrative Tools” folder in the Start Menu.
  • Highlight the IIS site created through the extended web application and click on the “Bindings” link in the right hand pane.

image

  • Highlight the portalsearch.2008r2.local entry and click the “Edit” button

image

  • In our example we have multiple IP addresses bound to the same network adapter.  The DNS A record we created for portalsearch.2008r2.local was assigned 192.168.153.16 so we need to specify that address.  In the “SSL certificate:” field click the drop down, select the certificate we created in the previous step, and click “OK” to apply it to the site.

image

  • Since we’ve modified IIS outside of SharePoint Central Administration you’ll need to perform these same steps on other servers in the farm.  The site should now render with the new URL using SSL

image

Configure Search Content Source

We are now ready to configure Search to crawl the extended web application using Windows authentication now that the extended web application has been provisioned.

By default SharePoint creates a content source called “Local SharePoint sites” and will place all the web applications in that entry.  To view the content sources you’ll need to open the “Search Administration” site in Central Admin:

  • Open the Central Administration site
  • Click on the “Manage service applications” link in the “Application Management” section
  • Click on the link for the “Search Service Application”
  • From the Search Administration page click on the “Content Sources” link located in the left hand pane under the “Crawling” section

image

  • The “Local SharePoint sites” entry should be present.  Click the drop down menu, and click “Edit”

image

  • In the “Start Addresses” section we’ll remove the entry for the main web application “https://portal.2008r2.local” and replace it with the extended web app entry “https://portalsearch.2008r2.local:444”.  By doing this search will no longer attempt to crawl the main web application and as a result we won’t get all those access denied errors in the crawl log.

image

  • Click OK to commit the change
Configure Server Name Mappings in Search

At this point if a crawl of the web application was performed we would be able to index the content via the extended web application, however the results would come back with https://portalsearch.2008r2.local:444 in the description instead of https://portal.2008r2.local.  We don’t want to inadvertently direct users to the extended web application so we need to configure a “Server Name Mappings” in Search to replace any address that starts with https://portalsearch.2008r2.local:444 with https://portal.2008r2.local.

image

  • In the Search Administration page click on the “Server Name Mappings” link in the “Crawling” section of the left hand pane

image

image

Execute a Full Crawl

In order for the “Server Name Mappings” entry to take effect a full crawl must be performed on the content source.

  • Navigate to the Central Administration site
  • Click on the “Manage service applications” link under the “Application Management” section
  • Click on the link to the “Search Service Application”
  • From the Search Administration page click on the “Content Sources” link under the “Crawling” section in the left hand pane
  • Click the drop down menu for the “Local SharePoint sites” content source and select the “Start Full Crawl” option

image

  • Click OK to proceed with the full crawl

Now if we execute a search query against the web application the results will modified to point to the main web applications URL of portal.2008r2.local.

image

Conclusion

Anyone who works with ADFS as a Trusted Identity Provider for SharePoint will more than likely come across little nuances that weren’t present when using only “Windows Authentication”.  Hopefully this article provided some insight on some of those nuances and potential ways to work around them.

Tagged with: , , ,
Posted in ADFS, Search, SharePoint 2013
26 comments on “SharePoint 2013 Configuring Search to Crawl Web Applications Using Claims and ADFS 2.0
  1. Excellent post. Thank you jasonth! The whole series of SharePoint 2013 and ADFS2 is great!

    Have you noticed issues with the search results when using server name mappings? In our environment the server name mappings are not applied to links that use a viewer application e.g. Office Web App Server’s WopiFrame.aspx. Excel Viewer and Visio viewer application pages’ URL are also not transformed as server name mappings should apply.

    The search result’s green URL, visible to the end user, is transformed based on the server name mappings, but the actual (a) href and (iframe) src values are not. In environments where Office Web Apps Server is used, this is a major issue.

    • jasonth says:

      Hello Pasi,

      Thanks for the positive comments! Yes we are currently struggling with this exact same issue. We have a case open with Microsoft Support and have also found several others experiencing the same issue in the MSDN forum post “How to setup SharePoint to allow both search crawling and ADFS authentication”.

      Unfortunately this issue has been present since the 2013 release of SharePoint and hasn’t been fixed. We are trying to get confirmation from Microsoft to determine whether or not this is a known issue/bug, expected behavior, and if there’s a work around. We’ll post an update on the blog and in the MSDN forum on what we find out.

  2. Hi Jason!

    I made some tests and this is what I found out…

    My initial setup for a web application:
    Default zone: https://sites.domain.com (ADFS/Trusted IP, zone used by end users)
    Intranet zone: http://server.domain.com:8765 (NTLM, zone used by search crawl)

    Server name mapping was added for transforming search result URLs
    http://server.domain.com:8765/* => https://sites.domain.com/*

    As specified in this thread, not all of the URLs were transformed => major issues.

    I reversed the zones:
    Default zone: http://server.domain.com:8765 (NTLM, zone used by search crawl)
    Intranet zone: https://sites.domain.com (ADFS/Trusted IP, zone used by end users)

    Search results do not have issues and server name mapping(s) is not required.

    => DEFAULT zone must use windows integrated authentication. 😦 Trusted identity provier(s) should use the other zones.

    • jasonth says:

      Hi Pasi,

      Thanks for looking into that. Just to clarify when you setup the default zone as NTLM and then the intranet zone using ADFS/Trusted IP, the server mappings for OWA are correct? So in your example if you return search results and click on a Word, Excel, PowerPoint, or OneNote file when it’s opened in the browser the URL is https://sites.domain.com?

  3. Hi,

    Thanks for this post. However after extending my website to Intranet zone, i got Cannot Display Webpage error on my default zone… Any idea?

    Thanks,
    Andreas

  4. I want to seriously caution against using Server Name Mappings, particularly in SharePoint 2013.

    Admittedly, with SharePoint 2010, Server Name Mappings did appear to provide a workaround. However, although they appear to work, Server Name Mappings were definitely not designed for this particular scenario.

    Second, In SharePoint 2013, I know for certain that some managed properties (e.g. SPSiteUrl and ParentUrl to name two) in the Index absolutely do not get *updated by Server Name Mappings, so adding them will only make the problem worse!!! In other words, you’ll have some URL-based properties that are relative to one URL and other MPs relative to the mapped URL…

    But because Server Name Mappings were not intended for this scenario, I would not have expectation that this should work in all cases.

  5. I would like to caution\advise against using Server Name Mappings, particularly in SharePoint 2013.

    While in 2007 and 2010, Server Name Mappings did appear to provide a workaround or a band-aid and although they appear to work, Server Name Mappings were not designed for this particular scenario.

    In SharePoint 2013, there are some managed properties, like SPSiteUrl and ParentUrl to name a couple, that will not get *updated by Server Name Mappings in the index, so putting in Server Name Mappings can make problems worse. In other words, you’ll have some URL-based properties that are relative to one URL and other MPs relative to the mapped URL..

  6. Jason.. can you elaborate on what issues you are seeing when you have a ADFS based Extended app and Claims\NTLM (only) as the default? I would think that if you are crawling a purely claims\ntlm based Web App as the default and its extended to ADFS in like the extranet or intranet zone, then you should not need a custom sign in page that needs to bypass the selector.. it seems as if you are using NTLM\ADFS both in your web app.. ?

    • jasonth says:

      Hi Anthony,

      You are correct. With NTLM/Claims on the default web app and ADFS authentication on the extended web app we do get past the hurdle of having a selector page presented because we direct all the users to the extended ADFS web application which only has one authentication mechanism setup. In this setup the search issue is partially resolved because the default zone is being crawled and we able to configure the servername mappings to reflect the ADFS extended zone.

      To recap here’s what we have setup

      Default Zone (NTLM\Claims) – Used primarily for Search
      https://spt.contoso.com:10000
      https://my.contoso.com:10000
      Extended (ADFS) – Users access this zone
      https://spt.contoso.com:443
      https://my.contoso.com:443

      So by going the above route here are the issues we ran into:

      -Search scoped at the site, list, or document library level doesn’t work when the results are directed to the osssearchresults.aspx page. You can create custom search result pages scoped at that level, but by default it doesn’t work.
      -MySites are continually being redirected back to the default zone
      -Alerts contain links that point back to the default zone instead of the extended web apps
      -some third party controls end up pointing back to the default zone

      We are testing the autosignin.aspx CodePlex project (http://spautomaticsignin.codeplex.com/) so we can do the following:

      -Have a single web app/Zone with both authentication providers configured
      -Automatically direct the SharePoint servers to the windows authentication provider and the users to the ADFS auth provider

      By doing this all the issues we previously encountered with splitting the auth providers into separate web apps appear to be resolved, but we are about half way through the testing phase so I can’t say it will resolve every issue.

      In short, when using ADFS there are huddles that you’ll need to jump that aren’t present with straight NTLM\Claims.

  7. Hey Jason,
    Thanks for the detailed response here – Anthony and I work together, so that’s why you both of us on this 🙂

    If you’re crawling the default zone, then you don’t need Server Name Mappings because the re-mapping to other zones happen auto-magically. In other words, if you crawl https://spt.contoso.com:10000 but then query from https://spt.contoso.com:443 then the query processor will kick back results (from this Web App) all relative to the https://spt.contoso.com:443 URL (even though you crawled https://spt.contoso.com:10000 ). However, if you have Server Name Mappings, not all URL-based properties adhere to the Server Name Mappings… so you end up with *broken properties (e.g. inconsistent URLs in the properties), which would be consistent with the behavior you described regarding the osssearchresults.aspx page.

    Regarding the MySites being redirected to default zone…
    Are the MySites and your content Web Apps extended consistently (e.g. if the MySite Web App is extended to the “Intranet” zone, is the content Web App also extended to its respective “Intranet” zone, too? …or is the MySite Web App extended to, say, the Extranet and the content Web App something different like “Custom”)?

    I noted some of this in my post (“Beware crawling the non-Default zone for a SharePoint 2013 Web Application” http://blogs.msdn.com/b/sharepoint_strategery/archive/2013/02/20/beware-crawling-the-non-default-zone-for-a-sharepoint-2013-web-application.aspx ), where I noted (these will make more sense in the context of the screen shots I provided in that post):
    ——————
    Results related to another Web App would also be relative to this zone (which to knowledge is new to SP2013). For example, if I query from the http://initech URL (in other words, from the Intranet zone), then all results related to this Web App would be relative to the http://initech URL (such as http://initech/result1.aspx, http://initech/result2.aspx, etc…)

    Further, the query, which was issued from the Intranet zone of the http://faceman Web App (In this case, http://initech as seen in the browser’s address navigation bar), the results related to the http://sp-foo:88 Web App would also be relative to that Web App’s Intranet zone

    If the zone you’re in doesn’t exist in the other Web App, the ***results will just defer to the Default zone for that applicable Web App***
    ——————

    As far as the Alerts are concerned…
    Agreed, this is a concern… I don’t have a good answer for you here at this time, but hopefully that changes (e.g. working on it).

    • whorn76 says:

      Hi Brian,

      Thanks for all your comments. I work with Jason and that’s why you get both of us as well ;).

      As far as MySites go they seem to work fairly well in the setup that we have, one of the main issues that we have is that the welcoming email that is received when a user’s MySite is automatically created is populated with URLs for the default web app. I’ve researched around and it appears there is no way to change that. If they user happens to visit that Default Zone URL then they don’t see their newly created MySite but get that basic ‘About Me’ page.

      However testing with the autosignin.aspx CodePlex project seems to be going well.

      Another issue I’ve recently come across and perhaps you can verify this is that using SharePoint-hosted Apps is not supported with ADFS 2.0 since it doesn’t support wildcard URL redirection. Is this something that you’re come across or perhaps have a work around for? I’ve read that provider-hosted web apps would be supported with some additional configuration.

      Cheers.

  8. Sammy says:

    Try with the script from http://blogs.msdn.com/b/josrod/archive/2012/01/10/add-a-user-or-group-to-the-spwebapplication-user-policy-for-all-web-applications-in-the-farm.aspx by explicitly adding domain\username via powershell when you have ADFS. If you add crawl account via UI, it simply doesn’t work.

    I added domain\username via powershell and everything worked fine. Took me hours to realized this issue. Ugh!

  9. Malek says:

    Thanks for this very detailed post.

    I have done the same thing, one zone for Windows auth (crawling) and the second for ADFS wich is the one accessible by users.

    My problem is that i get results from some web apps and not others, and some users get more results thant me as if they have more permissions but i am an admin of all site collections

    can you help me please

    • Malek, is the Windows Authentication zone your Default zone? If not, then what you describe can be expected (e.g. there is a baked in assumption that the Default zone is being crawled, which means your Default zone needs to use Windows Auth)

      • Malek says:

        Thanks Brian,

        Actually, the problem seems to be that the security trimmer is case sensitive.

        When i migrate the SharePoint users (AD groups) to ADFS, i have used a mix of lower and uper case for the login names. I see in the ULS that the security trimmer don’t get matches for the user groups send as the claims.

        It’s crazy, but using lower case for login names solved my problem

  10. One option I have found is to ad a 302 redirect on the the /_login/ folder to ../_trust/default.aspx and set to only this no sub folders. This may not work for mobile devices as they use a different login page but as the search crawler ignores the 302 and heads straight to the /_windows/ folder. This is doing the same as editing the login page.

    • I have done some more digging and have found that Office 365 gets around this problem by doing a 302 redirect to ../_forms/default.aspx on the /_login/ folder I am not sure if they do this in IIS or if they have a custom login page and dll to look at the user agent string. however the same redirect setup in IIS works or the one I mentioned in the previous folder to ../_trust/default.aspx also works.

  11. Prashant says:

    The steps below mentioned in the post are same for SP 2010 also? because I am facing same problem in SP 2010

    Create a DNS A-record for the extended web app
    Create a SSL Certificate
    Extend the Web Application
    Bind Certificate to Extended Web Application
    Configure Search Content Source
    Configure Server Name Mappings in Search
    Execute a Full Crawl

  12. If you declare a custom signin page of “/_trust/default.aspx” and use a custom CP for name resolution, you can have only one zone that services both WiA and ADFS. I vastly prefer that to having an extended app and two urls!

    look @ https://msdn.microsoft.com/en-us/library/office/hh237665(v=office.14).aspx

    ofc, i agree that if you don’t want custom code, extending is the only way 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: