De duplication does not work with body field

(2 posts) (2 voices)

Tags:

No tags yet.

senthilkumar.arumugam
Member

Hi,

We have few pages that are represented by muliple URLs, they are getting indexed more than once in LWE. When we search for specific content from that page(doc), same content is displaying multiple times with different URLS. We tried using de-duplication feature to remove these duplicated pages. We created a new field type and assigned <body> tag to the new field and selected the new field type for de-duplication as overwrite. But this did not help removing the content.

Is it possible to use the custom defined fields for de-duplication? Please suggest..

Thanks...

Posted 4 months ago #
alexey.serba
Moderator

Are you sure that your pages are exact duplicates? LWE de-duplication feature supports exact duplicate detection by default (although it is possible to switch over to fuzzy algorithm by changing related portion of Solr configuration). If you have some problems troubleshooting this feature I would recommend to 1) change signatureField to be stored 2) change de-duplication type to "Tag" option 3) re-index your content and 4) verify that signature hashes are the same for these documents.

Posted 4 months ago #

Reply

You must log in to post.