Log similarity - Revision history

4523550: added category

2020-11-21T13:08:14Z

added category

← Older revision		Revision as of 13:08, 21 November 2020
Line 39:		Line 39:

	In an ideal world we would rather use https://www.php.net/manual/en/function.levenshtein.php , which we actually used at first. But it's a multiple times slow. It also scales worse. With long logs it's ridiculous slow. The similar_text function scales is more linear in it's scaling.-->		In an ideal world we would rather use https://www.php.net/manual/en/function.levenshtein.php , which we actually used at first. But it's a multiple times slow. It also scales worse. With long logs it's ridiculous slow. The similar_text function scales is more linear in it's scaling.-->

			[[Category:Site Info]]

3305483: Mostly grammar

2020-11-11T14:22:39Z

Mostly grammar

4523550: Cleaned up this page a bit, earlier version is saved as a comment for easier copypasteing in case my rewriting doesn't make sense. :)

2020-11-10T23:32:41Z

Cleaned up this page a bit, earlier version is saved as a comment for easier copypasteing in case my rewriting doesn't make sense. :)

@@ Line 1: / Line 1: @@
 '''The following is a copy/paste of an answer on Facebook. It should be cleaned up.'''
 But it's pasted here for now. It contains a lot of related facts.
@@ Line 17: / Line 38: @@
 We rely on an open source function. The exact math can be learned by studying documentation and source code. https://www.php.net/manual/en/function.similar-text.php
-In an ideal world we would rather use https://www.php.net/manual/en/function.levenshtein.php , which we actually used at first. But it's a multiple times slow. It also scales worse. With long logs it's ridiculous slow. The similar_text function scales is more linear in it's scaling.
+In an ideal world we would rather use https://www.php.net/manual/en/function.levenshtein.php , which we actually used at first. But it's a multiple times slow. It also scales worse. With long logs it's ridiculous slow. The similar_text function scales is more linear in it's scaling.-->

3305483: Create page

2020-02-15T20:35:17Z

Create page

New page

'''The following is a copy/paste of an answer on Facebook. It should be cleaned up.'''
But it's pasted here for now. It contains a lot of related facts.

It's the content of the logs, not the logs themselves that are similar. 57% doesn't say that 57% of your logs are identical, but the content of the logs coming after the former one has 57% similarity in average.

Having code that was language aware would be close enough to impossible, the code isn't language aware. It's characters, nothing else. If you in one log write "ab" and "cd" in the next, those will have 0% similarity. Also, since it's not word aware, it doesn't matter if some words are more commonly used in some languages. "Rabbit" and "Tea" will also have a similarity. The similarity would be 22% actually.

Another example. Logging these three logs in a row would give 50% log similarity: "ab", "cd", "cd" ( (0+100)/2 ). Add another "ab" at the end and you would have 33% ( (0+100 + 0)/3 ).

The number itself doesn't really have to be understandable in our opinion. The worth of it is to keep it low. If it's lower than most others, then you are doing a better job of writing varying logs than the majority.

I don't think it has to be taken with a pinch of salt at all. It's a quite hard fact as a number, it's just hard to understand the number. But the lower it is, the more varying the logs are. High word count and low log similarity is definitely possible. In theory one could reach 0%, but that would definitely requiring aiming for it, and I can agree that it won't happen by chance.

You say that the numbers here says it can't, I don't agree. 213 words,17% and 308 word, 22% are quite long logs and a low similarity. Maybe you just have to high expectations of what "low similarity" is? I would personally say that everything below 50% is fairly low.

We rely on an open source function. The exact math can be learned by studying documentation and source code. https://www.php.net/manual/en/function.similar-text.php

In an ideal world we would rather use https://www.php.net/manual/en/function.levenshtein.php , which we actually used at first. But it's a multiple times slow. It also scales worse. With long logs it's ridiculous slow. The similar_text function scales is more linear in it's scaling.

← Older revision		Revision as of 14:22, 11 November 2020
Line 1:		Line 1:
	'''Log similarity''' is part of the [[Finds tab#Some number\|~~some~~ numbers-module]] of the [[Finds tab]] in the [[Profile Stats]]. It is an indication of how much the user's logs differ, a high score mean their logs have a lot of repeated content, a low score mean they are more diverse.		'''Log similarity''' is part of the [[Finds tab#Some number\|Some numbers-module]] of the [[Finds tab]] in the [[Profile Stats]]. It is an indication of how much the user's logs differ, a high score mean their logs have a lot of repeated content, a low score mean they are more diverse.

	== How does it work? ==		== How does it work? ==
	It is the content of the logs, not the actual logs that are similar. If the number is 57% that does not mean that 57% of the users logs are identical, but that the content of the logs coming after the former one has a average of 57% similarity.		It is the content of the logs, not the actual logs that are similar. If the number is 57% that does not mean that 57% of the users logs are identical, but that the content of the logs coming after the former one has an average of 57% similarity.

	The code is not aware of the language or words, it is based on characters only. So, if the user ~~write~~ "ab" in one log and "cd" in the next log, those logs will have 0% similarity. So logging the three logs "ab", "cd" and "cd" in a row would give 50% log similarity (0% similarity for "ab" and "cd", 100% similarity for "cd" and "cd". (0+100)/2=50 ), posting ~~an other~~ log of "ab" after that would ~~get~~ 33% ((0+100+0)/3=33). Since the code is not based on words it does not matter if some words are very common in the language of the user since all languages will have characters that are more common, as an example the words "Rabbit" and "Tea" have 22% similarity.		The code is not aware of the language or words, it is based on characters only. So, if the user writes "ab" in one log and "cd" in the next log, those logs will have 0% similarity. So logging the three logs "ab", "cd" and "cd" in a row would give 50% log similarity (0% similarity for "ab" and "cd", 100% similarity for "cd" and "cd". ''(0+100)/2=50'' ), posting another log of "ab" after that would result in 33% (''(0+100+0)/3=33''). Since the code is not based on words it does not matter if some words are very common in the language of the user since all languages will have characters that are more common, as an example the words "Rabbit" and "Tea" have a 22% similarity.

	== What is considered low log similarity? ==		== What is considered low log similarity? ==
	If the number is lower than most others, then the user is doing a better job of writing varying logs than the majority. That ~~beeing~~ said, everything below 50% is fairly low. It is definitely possible to ~~achive~~ a high word count and a low log similarity at the same ~~thime~~. In theory one could reach 0% but that would definitely requiring ~~aming~~ for it ~~and~~ it ~~will~~ not happen by chance.		If the number is lower than most others, then the user is doing a better job of writing varying logs than the majority. That being said, everything below 50% is fairly low. It is definitely possible to achieve a high word count and a low log similarity at the same time. In theory one could reach 0% but that would definitely requiring aiming for it, it's not likely to happen by chance.

	== Math and programming ==		== Math and programming ==
	The exact math can be learned by studying documentation and source code. We rely on an open source function ~~that is called~~ [https://www.php.net/manual/en/function.similar-text.php similar_text].		The exact math can be learned by studying documentation and source code. The algorithm rely on PHP's open source function [https://www.php.net/manual/en/function.similar-text.php similar_text].

	In an ideal world ~~we would rather use~~ [https://www.php.net/manual/en/function.levenshtein.php ~~levenshtein~~] and at first ~~we did!~~ But it is much slower and it scales worse so with long logs it is just too slow. The similar_text-function is more linear in its scaling.		In an ideal world the [https://www.php.net/manual/en/function.levenshtein.php Levenshtein algorithm] would be used, and at first it actually was. But it is much slower and it scales worse so with long logs it is just too slow. The similar_text-function is more linear in its scaling.


Line 18:		Line 18:


	<!-- I saved ~~the previous page~~ here ~~to make it easier to copypaste things if I messed any facts up in my re-writing~~. ~~If the article is fine this comment can~~ be removed. ~~/Pleu~~		<!-- The above is based on a copy/paste from Facebook. The original content is saved as a comment here. Can be removed in the future. -->
			<!--
	'''The following is a copy/paste of an answer on Facebook. It should be cleaned up.'''		'''The following is a copy/paste of an answer on Facebook. It should be cleaned up.'''
	But it's pasted here for now. It contains a lot of related facts.		But it's pasted here for now. It contains a lot of related facts.