C＃ F23. Stringsimilarity Library: String repeatability, text similarity, anti plagiarism

        Recently in our Shop detective ( Buy a shop, rent a shop, just the upper one, detective ) In the project , In order to prevent brokers from copying other brokers' shop source introduction when publishing shop source , So we are going to make a preliminary anti plagiarism judgment according to the similarity of the content .

Don't talk much , Don't bother typing , The following is the dry goods ：

Directly in VS Of Nuget Search in manager ：F23.StringSimilarity install .GitHub Portal , There is a detailed introduction , And constantly updated .

At present, the library has implemented more than ten algorithms , Select the algorithm suitable for your business according to your own needs , Each algorithm has its own advantages and disadvantages , It is suggested to roughly understand each algorithm , It's convenient for you to choose which one to use , You can search and understand according to the name of each algorithm in the plug-in .

The following figure is a screenshot of more than ten algorithms automatically translated by Google , above GitHub The portal can go in and see , Each algorithm is introduced ：

  The use of each algorithm is simple , The following are examples of various algorithms ：

public static void Main(string[] args)
        {
            var str1 = Html2Text(@"<div><div style='padding: 10px 0 5px 0'><p>1： Property area ： On the second floor     1221㎡, Third floor ：518㎡.<br/>2： Whole floor selling , Independent aisle , You can get through and use it by yourself .<br/>3： Holiday Resort , The flow of people is very large .</p></div></div>");
            var str2 = Html2Text(@"<div><div style='padding: 10px 0 5px 0'><p>1：<br/>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Third floor ：518.76 Square meters <br/>&nbsp;2： Whole floor , Independent aisle , You can get through and use by yourself <br/>3： Holiday Resort , Large commercial Street Center , The traffic is huge .</p></div></div>");

            var jaroWinkler = new JaroWinkler();
            var a= jaroWinkler.Similarity(str1, str2);

            var normalizedLevenshtein = new NormalizedLevenshtein();
            var b= normalizedLevenshtein.Similarity(str1, str2);

            var cosine = new Cosine();
            var c= cosine.Similarity(str1, str2);

            var jaccard = new Jaccard();
            var d = jaccard.Similarity(str1, str2);

            var sorensenDice = new SorensenDice();
            var e = sorensenDice.Similarity(str1, str2);

            var ratcliffObershelp = new RatcliffObershelp();
            var f = ratcliffObershelp.Similarity(str1, str2);

            var longestCommonSubsequence = new LongestCommonSubsequence();
            var g= longestCommonSubsequence.Distance(str1, str2);
        }
public static string Html2Text(string htmlStr)
        {
            if (String.IsNullOrEmpty(htmlStr))
            {
                return "";
            }
            string regEx_style = "<style[^>]*?>[\\s\\S]*?<\\/style>"; // Definition style Regular expression of  
            string regEx_script = "<script[^>]*?>[\\s\\S]*?<\\/script>"; // Definition script Regular expression of  
            string regEx_html = "<[^>]+>"; // Definition HTML Regular expressions for tags  
            htmlStr = Regex.Replace(htmlStr, regEx_style, "");// Delete css
            htmlStr = Regex.Replace(htmlStr, regEx_script, "");// Delete js
            htmlStr = Regex.Replace(htmlStr, regEx_html, "");// Delete html Mark 
            htmlStr = Regex.Replace(htmlStr, "\\s*|\t|\r|\n", "");// Remove tab、 Space 、 Blank line 
            htmlStr = htmlStr.Replace(" ", "");
            htmlStr = htmlStr.Replace("\"", "");
            htmlStr = htmlStr.Replace("\"", "");
            htmlStr = htmlStr.Replace("&nbsp;", "");
            return htmlStr.Trim();
        }

By the way, give a filter html Functions of code , In the picture above Html2Text(string htmlStr)
ha-ha .

Then talk about the ideas I use , I use this ：NormalizedLevenshtein, The similarity value is 0~1 Between

Because I checked on the Internet , This algorithm is more suitable for plagiarism judgment of papers , More in line with my needs , Of course, you need to add some of your own logic , For example, put two strings of html Remove the code and compare , Whether numbers and letters need to be compared, etc , Then set a threshold , What I set up is 0.65, Greater than this value is considered plagiarism . Of course, plagiarism only refers to the direct plagiarism of the system , Because I only compared with the data of this system .