Data synchronization - Revision history

21934609: Fix formatting.

2026-07-12T19:44:28Z

Fix formatting.

← Older revision		Revision as of 19:44, 12 July 2026
Line 66:		Line 66:
	When statistics are generated instead of served from cache, Project-GC will check whether the player's finds are up-to-date in the database. If they are not, the player's finds will be automatically refreshed as part of the rendering process. This automatic refresh can occur regardless of who is viewing the profile, including when viewing another player's statistics. However, it only occurs when new statistics are being generated (that is, not when cached statistics are served) and only if the stored finds are sufficiently out-of-date. Finds are considered out-of-date after one hour for paid members and after 30 hours for non-paid members.		When statistics are generated instead of served from cache, Project-GC will check whether the player's finds are up-to-date in the database. If they are not, the player's finds will be automatically refreshed as part of the rendering process. This automatic refresh can occur regardless of who is viewing the profile, including when viewing another player's statistics. However, it only occurs when new statistics are being generated (that is, not when cached statistics are served) and only if the stored finds are sufficiently out-of-date. Finds are considered out-of-date after one hour for paid members and after 30 hours for non-paid members.

	Paid members can also manually induce a refresh of their own finds by using the Regenerate button. This forces a refresh of their finds and regeneration of their statistics, without waiting for the automatic refresh or cache expiration. The button refreshes only the player's finds; it does not update the underlying geocache data. The button may only be used if the player's finds in the database are at least one hour out-of-date.		Paid members can also manually induce a refresh of their own finds by using the '''Regenerate''' button. This forces a refresh of their finds and regeneration of their statistics, without waiting for the automatic refresh or cache expiration. The button refreshes only the player's finds; it does not update the underlying geocache data. The button may only be used if the player's finds in the database are at least one hour out-of-date.

	== Challenge checkers ==		== Challenge checkers ==

21934609: Update the section on Profile stats to reflect recent changes.

2026-07-12T19:43:54Z

Update the section on Profile stats to reflect recent changes.

← Older revision		Revision as of 19:43, 12 July 2026
Line 60:		Line 60:

	== Profile stats ==		== Profile stats ==
	* Generated from ''current state'' at the date ~~written~~ in the header.		Profile statistics are generated using data available to Project-GC at the moment of generation up to the date shown in the header. All statistics are generated from the primary database cluster. Paid members can optionally choose to include Adventure Lab caches in generated statistics through their settings.
	* Labs included depending on setting and paying/freemium.
	* Cached for 7 days for ~~freemium~~, ~~24h for paying~~.		To reduce server load, generated profile statistics are cached. The cache duration is determined by the membership level of the person requesting the page, not the membership level of the profile being viewed. Statistics are cached for up to 24 hours for paid members and up to 7 days for non-paid members. In other words, a paid member viewing a non-paid member's profile will see statistics that are at most 24 hours old, while a non-paid member viewing a paid member's profile may see statistics that are up to 7 days old.
	* Caching is not ~~shared between~~ the ~~user~~ the ~~data~~ is ~~about~~ and ~~other viewers~~.
	* ~~Based on~~ the ~~primary~~ database ~~cluster~~.		When statistics are generated instead of served from cache, Project-GC will check whether the player's finds are up-to-date in the database. If they are not, the player's finds will be automatically refreshed as part of the rendering process. This automatic refresh can occur regardless of who is viewing the profile, including when viewing another player's statistics. However, it only occurs when new statistics are being generated (that is, not when cached statistics are served) and only if the stored finds are sufficiently out-of-date. Finds are considered out-of-date after one hour for paid members and after 30 hours for non-paid members.

			Paid members can also manually induce a refresh of their own finds by using the Regenerate button. This forces a refresh of their finds and regeneration of their statistics, without waiting for the automatic refresh or cache expiration. The button refreshes only the player's finds; it does not update the underlying geocache data. The button may only be used if the player's finds in the database are at least one hour out-of-date.

	== Challenge checkers ==		== Challenge checkers ==

15789140: /* Refreshing geocache data */

2021-03-15T16:36:36Z

Refreshing geocache data

← Older revision		Revision as of 16:36, 15 March 2021
Line 34:		Line 34:
	The frequency of when [[Geocache]]s are refreshed are based on a mix of several variables, like for example:		The frequency of when [[Geocache]]s are refreshed are based on a mix of several variables, like for example:
	* Last found		* Last found
	* Hidden date		* [[Hidden date]]
	* Disabled/Archived state		* Disabled/Archived state
	* Number of logs		* Number of logs

4523550: Edited to working link.

2020-11-12T13:53:33Z

Edited to working link.

← Older revision		Revision as of 13:53, 12 November 2020
Line 75:		Line 75:
	* calendar loops		* calendar loops

	This does not affect [[Profile stats]]. If numbers like these are needed, [[Profile stats]] calculates them itself instead of using pre-calculated data. This is also needed since it may have [[~~Labcache~~]] data merged.		This does not affect [[Profile stats]]. If numbers like these are needed, [[Profile stats]] calculates them itself instead of using pre-calculated data. This is also needed since it may have [[Lab cache]] data merged.

6041005: some minor edits for British English style

2020-11-12T12:53:49Z

some minor edits for British English style

← Older revision		Revision as of 12:53, 12 November 2020
Line 9:		Line 9:
	Geocaching data is continuously being updated in [[Project-GC]]'s databases via the mentioned API. This is done using several parallel methods:		Geocaching data is continuously being updated in [[Project-GC]]'s databases via the mentioned API. This is done using several parallel methods:
	* Continuously fetching newly published [[Geocache]]s. Normal latency 0-35 seconds.		* Continuously fetching newly published [[Geocache]]s. Normal latency 0-35 seconds.
	* Continuously updating of finds for [[Geocacher]]s. See [[Data synchronization#Refreshing_profiles\|Refreshing profiles]] for more details.		* Continuously updating finds for [[Geocacher]]s. See [[Data synchronization#Refreshing_profiles\|Refreshing profiles]] for more details.
	* Regular updates of [[Geocache]] information. See [[Data synchronization#Refreshing geocache data\|Refreshing geocache data]] for more details.		* Regular updates of [[Geocache]] information. See [[Data synchronization#Refreshing geocache data\|Refreshing geocache data]] for more details.

Line 46:		Line 46:
	Now most [[Geocache]] data and logs exists in the primary database cluster and are fairly up-to-date. Most of the data will only be hours old in the database, but a fair share is expected to be 24-36 hours old.		Now most [[Geocache]] data and logs exists in the primary database cluster and are fairly up-to-date. Most of the data will only be hours old in the database, but a fair share is expected to be 24-36 hours old.

	However [[Project-GC]] has more than one database cluster. Most statistics on the web are created based on data from another database cluster. The primary database cluster replicates its data every ~~4th hour into~~ the second cluster, adding an additional ~4 hours of latency. As an example most top lists are based on this secondary cluster, while [[Profile stats]] are not.		However [[Project-GC]] has more than one database cluster. Most statistics on the web are created based on data from another database cluster. The primary database cluster replicates its data every 4 hours o the second cluster, adding an additional ~4 hours of latency. As an example most top lists are based on this secondary cluster, while [[Profile stats]] are not.

	As a technical note, the primary database cluster is a row-based relational database. The secondary cluster is created and meant for database harvesting and is a column-oriented DBMS.		As a technical note, the primary database cluster is a row-based relational database. The secondary cluster is created and meant for database harvesting and is a column-oriented DBMS.

	== Statistics ==		== Statistics ==
	As mentioned in [[Data synchronization#Databases\|Databases]], most statistics are based on a secondary database cluster. Even if [[Project-GC]] itself has up-to-date profile data the secondary database cluster might still have the old data. This is due to the fact that a full replica of the data-set is copied every ~~fourth hour~~.		As mentioned in [[Data synchronization#Databases\|Databases]], most statistics are based on a secondary database cluster. Even if [[Project-GC]] itself has up-to-date profile data the secondary database cluster might still have the old data. This is due to the fact that a full replica of the data-set is copied every four hours.

	Most top lists are using this secondary cluster. Basically everything that gets heavily computed (data harvesting) uses the secondary cluster, while more raw fetches use the primary one. [[Profile stats]] is an exception.		Most top lists are using this secondary cluster. Basically everything that gets heavily computed (data harvesting) uses the secondary cluster, while more raw fetches use the primary one. [[Profile stats]] is an exception.
Line 57:		Line 57:
	It's also worth mentioning here as well that [[Lab caches]] generally aren't included in statistics. They aren't technically compatible and it would be very complex to do this. Again, [[Profile stats]] can be an exception.		It's also worth mentioning here as well that [[Lab caches]] generally aren't included in statistics. They aren't technically compatible and it would be very complex to do this. Again, [[Profile stats]] can be an exception.

	Finally all generated statistics are [[Data caching\|cached]]. The period it ~~gets~~ cached ~~may vary~~ but ~~everything~~ between 5 minutes and 1 hour is very common. So if two ~~persons~~ ask for the same statistic, it will only be computed the first time, the second person will receive a cached version. This has the downside that it potentially also adds more latency and shows older data in some cases.		Finally, all generated statistics are [[Data caching\|cached]]. The period for which it is cached varies but anything between 5 minutes and 1 hour is very common. So if two people ask for the same statistic, it will only be computed the first time, the second person will receive a cached version. This has the downside that it potentially also adds more latency and shows older data in some cases.

	== Profile stats ==		== Profile stats ==
Line 70:		Line 70:

	== Special numbers ==		== Special numbers ==
	Some statistics are very special since they are based on pre-calculated values. This is usually because it would be extremely hard to calculate this in real-time, therefore it's based on pre-calculated values. [[Project-GC]] calculates the following data for every [[Geocacher]] in the background on a regular basis. Normally daily, but there are some exceptions to it~~. Usually~~ that some users gets calculated more often.		Some statistics are very special since they are based on pre-calculated values. This is usually because it would be extremely hard to calculate this in real-time, therefore it's based on pre-calculated values. [[Project-GC]] calculates the following data for every [[Geocacher]] in the background on a regular basis. Normally daily, but there are some exceptions to it (usually that some users gets calculated more often).
	* dt loops		* D/T loops
	* streaks		* streaks
	* calendar loops		* calendar loops

	This is not ~~affecting~~ [[Profile stats]]. If numbers like these are needed, [[Profile stats]] calculates them itself instead of using pre-calculated data. This is also needed since it may have [[Labcache]] data merged.		This does not affect [[Profile stats]]. If numbers like these are needed, [[Profile stats]] calculates them itself instead of using pre-calculated data. This is also needed since it may have [[Labcache]] data merged.

3305483: grammar

2020-11-12T12:44:44Z

grammar

← Older revision		Revision as of 12:44, 12 November 2020
Line 7:		Line 7:

	== Fetching of data ==		== Fetching of data ==
	Geocaching data is continuously being updated in [[Project-GC]]s databases via the mentioned API. This is done using several parallel methods:		Geocaching data is continuously being updated in [[Project-GC]]'s databases via the mentioned API. This is done using several parallel methods:
	* Continuously fetching newly published [[Geocache]]s. Normal latency 0-35 seconds.		* Continuously fetching newly published [[Geocache]]s. Normal latency 0-35 seconds.
	* Continuously updating of finds for [[Geocacher]]s. See [[Data synchronization#Refreshing_profiles\|Refreshing profiles]] for more details.		* Continuously updating of finds for [[Geocacher]]s. See [[Data synchronization#Refreshing_profiles\|Refreshing profiles]] for more details.

3305483: Fixed fact about fetching newly published geocaches

2020-11-12T12:19:57Z

Fixed fact about fetching newly published geocaches

← Older revision		Revision as of 12:19, 12 November 2020
Line 8:		Line 8:
	== Fetching of data ==		== Fetching of data ==
	Geocaching data is continuously being updated in [[Project-GC]]s databases via the mentioned API. This is done using several parallel methods:		Geocaching data is continuously being updated in [[Project-GC]]s databases via the mentioned API. This is done using several parallel methods:
	* ~~Daily fetch of all~~ newly published [[Geocache]]s.		* Continuously fetching newly published [[Geocache]]s. Normal latency 0-35 seconds.
	* Continuously updating of finds for [[Geocacher]]s. See [[Data synchronization#Refreshing_profiles\|Refreshing profiles]] for more details.		* Continuously updating of finds for [[Geocacher]]s. See [[Data synchronization#Refreshing_profiles\|Refreshing profiles]] for more details.
	* Regular updates of [[Geocache]] information. See [[Data synchronization#Refreshing geocache data\|Refreshing geocache data]] for more details.		* Regular updates of [[Geocache]] information. See [[Data synchronization#Refreshing geocache data\|Refreshing geocache data]] for more details.

3305483: grammar

2020-11-12T12:12:46Z

grammar

← Older revision		Revision as of 12:12, 12 November 2020
Line 23:		Line 23:
	* ... and so on, there are ~10 rules like this in place.		* ... and so on, there are ~10 rules like this in place.

	Also whenever a user visits the website of [[Project-GC]]s the system checks when that user was updated. If it's a paying member there will be yet another update if the data is more than 1 hour old. However, this is a job that gets queued in the background and it might take a few minutes until it kicks in.		Also whenever a user visits the website of [[Project-GC]]s the system checks when that user was last updated. If it's a paying member there will be yet another update if the data is more than 1 hour old. However, this is a job that gets queued in the background and it might take a few minutes until it kicks in.

	A [[Geocache]] with new logs will also get scheduled to have its metadata (difficulty, terrain, size, country ...) updated.		A [[Geocache]] with new logs will also get scheduled to have its metadata (difficulty, terrain, size, country ...) updated.

254729: /* Profile stats */

2020-11-12T12:06:31Z

Profile stats

← Older revision		Revision as of 12:06, 12 November 2020
Line 63:		Line 63:
	* Labs included depending on setting and paying/freemium.		* Labs included depending on setting and paying/freemium.
	* Cached for 7 days for freemium, 24h for paying.		* Cached for 7 days for freemium, 24h for paying.
	* Caching is not shared between the user the data is about and other ~~viewer~~.		* Caching is not shared between the user the data is about and other viewers.
	* Based on the primary database cluster.		* Based on the primary database cluster.

254729: /* Profile stats */

2020-11-12T12:05:37Z

Profile stats

← Older revision		Revision as of 12:05, 12 November 2020
Line 63:		Line 63:
	* Labs included depending on setting and paying/freemium.		* Labs included depending on setting and paying/freemium.
	* Cached for 7 days for freemium, 24h for paying.		* Cached for 7 days for freemium, 24h for paying.
	* Caching not shared between ~~foreign~~ viewer ~~and domestic~~.		* Caching is not shared between the user the data is about and other viewer.
	* Based on the primary database cluster.		* Based on the primary database cluster.