Advanced

IMPORTANT new polygon data

IMPORTANT new polygon data
April 21, 2021 01:55PM
As you already know we are updating polygon data from time to time. Next time, will be a huge update, causing a lot of issues.

All the code behind polygons is being updated, it has been an ongoing project for a year! I am finally starting to see the end of things which leads me to this post. All polygon data will be updated at the same time next time we update the data. We have already created the data for all countries but United Kingdom. The source is OpenStreetMap for all countries but Canada (where it's a mix of OSM and government census data). The plan is to use OSM data for UK as well, hopefully UK will be finished within a week.

Why a huge upgrade? There are three major reasons.
* Easier update path in the future.
* Improved code which wouldn't be very compatible with the old data.
* Maps in Profile stats can be made a lot better looking. I will attach examples in a replay.

What's the issue? The issue is that there will be a lot of new names again. Not the least for some countries. I have written a tool that will list all the changes (linked below). Again, UK isn't ready, so ignore that for now. I will update the list once it is.

List of changes

Note that the changes aren't live yet, so don't use the new names yet. It will probably take at least a month more. UK needs to be fixed, I need to write Map-Regions and Map-Counties tools. Then there is quite a lot of old GIS code that needs to be replaced as well.

Any suggestions for how we best make sure that we update every (active) script and tag so that it matches the new names? I am thinking I could fix United States and Australia by a script that does search and replace. Both of those have very unique names on what's currently live, I can't imagine that should give any false positive hits.

Any suggestions of how we can prepare new scripts and tags without touching the live data yet? One way I have been thinking about is to export all the current data to GitLab.com where they can be edited. Then I can import from there when making the release, if they haven't been updated at PGC yet. I could also potentiallty just export it as "text" files and share a Google Drive folder. Easier to edit, no history(?), but harder to add users (I need to add them). I also don't know what restricts there are, like max number of users. Any better suggestions?



Edited 2 time(s). Last edit at 05/07/2021 12:30PM by magma1447. (view changes)
Re: IMPORTANT new polygon data
April 21, 2021 02:01PM
Map examples available here.



Edited 1 time(s). Last edit at 04/21/2021 02:02PM by magma1447. (view changes)
Re: IMPORTANT new polygon data
April 21, 2021 02:05PM
How many relevant scripts are there?
Re: IMPORTANT new polygon data
May 18, 2021 01:02PM
I am preparing for a release for tomorrow (Wednesday). If something unexpected happens or shows up it might be delayed with one more day.

The above link with updated names (difference between now and future) has been updated (several times). It should now be final.

30 Challenge checker tags will be disabled by the system because we couldn't solve them ourselves. It's these:
21724, 21899, 22742, 24021, 24384, 25018, 29037, 32729, 32781, 33473, 34855, 34938, 35784, 36957, 40927, 41172, 46078, 46269, 46281, 46863, 48367, 53540, 53668, 53961, 56288, 56739, 59764, 59802, 48799, 37024

A lot of tags and quite a few scripts will be automatically updated based on a translation table that has been manually built. We have tried to be careful and only add names that never should have any false positive hits. The former versions will be stored offline in case we need to investigate.

39 tags will have their config updated based on manual work.

There will be a news entry published tomorrow together with the update. If you have any questions before or after that, feel free to ask them here, or elsewhere.
Re: IMPORTANT new polygon data
May 19, 2021 11:24AM
Would have been cool to list the tagger names against those 30, so we know which ones needs attention. :)
3 of mine got disabled, but luckily I also got a message about those. Updates on those:

36957: contacted the CO
37024: contacted the CO
53961: fixed the config (minor name changes)
Re: IMPORTANT new polygon data
May 19, 2021 12:24PM
Yes, it would have been smart of me to mention the tag-owner's name. Here is the same list, with the name of the creator.

+-------+-----------+
| id    | username  |
+-------+-----------+
| 21724 | vogelbird |
| 21899 | vogelbird |
| 22742 | vogelbird |
| 24021 | vogelbird |
| 24384 | vogelbird |
| 25018 | Turako    |
| 29037 | jpavlik   |
| 32729 | vogelbird |
| 32781 | vogelbird |
| 33473 | Pzi       |
| 34855 | vogelbird |
| 34938 | msei      |
| 35784 | msei      |
| 36957 | pieterix  |
| 37024 | pieterix  |
| 40927 | vogelbird |
| 41172 | Rikitan   |
| 46078 | al1pb     |
| 46269 | vogelbird |
| 46281 | vogelbird |
| 46863 | vogelbird |
| 48367 | PattuX    |
| 48799 | vogelbird |
| 53540 | vogelbird |
| 53668 | vogelbird |
| 53961 | pieterix  |
| 56288 | vogelbird |
| 56739 | GreyHams  |
| 59764 | vogelbird |
| 59802 | Rikitan   |
+-------+-----------+
Re: IMPORTANT new polygon data
May 19, 2021 12:40PM
Please note that none of the disabled checkers will have "minor name changes" as the reason for disable, minor name changes should already have been taken care of manually and if I've missed them they will not be on the disable list.

@pieterix For checker 53961 the changes are: Chur (18) and Haldenstein (18) was merged and is now Chur, Neuchâtel (24) was merged with Peseux (24) and is now Neuchâtel. CO should be contacted about how they want to solve this for their challenge.
Re: IMPORTANT new polygon data
May 19, 2021 01:04PM
@Pleu: thanks for clarifying. Assumptions on my side were obviously wrong. I've contacted the CO.
Re: IMPORTANT new polygon data
May 19, 2021 02:15PM
No worries pieterix, I've spent some hours with this data so I have a head start. :)

Here's a list of the merges/divides that I found, if the country/area of the disabled checker is not on this list the reason for disable is some version of border changes. Most likely the reason can be found in the list of changes from magma1447's first post, some countries will for example have a different number of regions that I have not labeled as a merge for different reasons.

Canada
Division No.  1 Avalon Peninsula-St. John's	| Labrador and Newfoundland
Francheville					| Les Chenaux and Trois-Rivières
Minganie--Le Golfe-du-Saint-Laurent		| Minganie and Le Golfe-du-Saint-Laurent
Sept-Rivičres--Caniapiscau			| Caniapiscau and Sept-Rivières

Luxembourg
Boevange-sur-Attert				| Merge: Helperknapp
Hobscheid					| Merge: Habscht
Mompach						| Merge: Rosport-Mompach
Rosport						| Merge: Rosport-Mompach
Septfontaines					| Merge: Habscht
Tuntange					| Merge: Helperknapp

Netherlands
Appingedam					| Merge: Eemsdelta
Delfzijl					| Merge: Eemsdelta
Haaren						| Merge with Oisterwijk as Oisterwijk
Loppersum					| Merge: Eemsdelta
Oisterwijk					| Merge with Haaren as Oisterwijk

Italy
Carbonia-Iglesias				| Merge: Sud Sardegna
Medio Campidano					| Merge: Sud Sardegna
Nuoro						| Merge with Ogliastra as Nuoro.
Ogliastra					| Merge with Nuoro as Nuoro.
Olbia-Tempio					| Merge with Sassari as Sassari.
Sassari						| Merge with Olbia-Tempio as Sassari.

Finland
Honkajoki					| Merged as Kankaanpää
Valtimo						| Merged as Nurmes
Nurmes						| Merged as Nurmes.
Kankaanpää					| Merged as Kankaanpää.

Switzerland
Alterswil (10)					| Merge: Tafers
Altwis (3)					| Merge: Hitzkirch
Arconciel (10)					| Merge: Bois-d’Amont
Aubonne (22)					| Merge: Aubonne
Bagnes (23)					| Merge: Val de Bagnes
Bauen (4)					| Merge: Seedorf (UR)
Brione (Verzasca) (21)				| Merge: Verzasca
Casti-Wergenstein (18)				| Merge: Muntogna da Schons
Charrat (23)					| Merge: Martigny
Cheiry (10)					| Merge: Surpierre
Chur (18)					| Merge: Chur
Corcelles-Cormondrèche (24)			| Merge: Neuchâtel
Corippo (21)					| Merge: Verzasca
Donat (18)					| Merge: Muntogna da Schons
Ependes (FR) (10)				| Merge: Bois-d’Amont
Frasco (21)					| Merge: Verzasca
Gettnau (3)					| Merge: Willisau
Gänsbrunnen (11)				| Merge: Welschenrohr-Gänsbrunnen
Haldenstein (18)				| Merge: Chur
Hindelbank (2)					| Merge: Hindelbank
Hitzkirch (3)					| Merge: Hitzkirch
Langenthal (2)					| Merge: Langenthal
Le Locle (24)					| Merge: Le Locle
Les Brenets (24)				| Merge: Le Locle
Lohn (GR) (18)					| Merge: Muntogna da Schons
Martigny (23)					| Merge: Martigny
Mathon (18)					| Merge: Muntogna da Schons
Miège (23)					| Merge: Noble-Contrée
Montherod (22)					| Merge: Aubonne
Mötschwil (2)					| Merge: Hindelbank
Neuchâtel (24)					| Merge: Neuchâtel
Obersteckholz (2)				| Merge: Langenthal
Peseux (24)					| Merge: Neuchâtel
Riggisberg (2)					| Merge: Riggisberg
Rohr (SO) (11)					| Merge: Stüsslingen
Rümligen (2)					| Merge: Riggisberg
Seedorf (UR) (4)				| Merge: Seedorf (UR)
Senèdes (10)					| Merge: Bois-d’Amont
Sonogno (21)					| Merge: Verzasca
St. Antoni (10)					| Merge: Tafers
Stüsslingen (11)				| Merge: Stüsslingen
Surpierre (10)					| Merge: Surpierre
Tafers (10)					| Merge: Tafers
Valangin (24)					| Merge: Neuchâtel
Venthône (23)					| Merge: Noble-Contrée
Veyras (23)					| Merge: Noble-Contrée
Vogorno (21)					| Merge: Verzasca
Vollèges (23)					| Merge: Val de Bagnes
Willisau (3)					| Merge: Willisau
Welschenrohr (11)				| Merge: Welschenrohr-Gänsbrunnen

Belgium
Aalter						| Merge with Knesselare as Aalter.
Deinze 						| Merge with Nevele as Deinze
Knesselare					| Merge: Aalter.
Kruishoutem					| Merge: Kruisem
Lovendegem					| Merge: Lievegem
Meeuwen-Gruitrode				| Merge: Oudsbergen
Neerpelt					| Merge: Pelt
Nevele						| Merge: Deinze
Opglabbeek					| Merge: Oudsbergen
Overpelt					| Merge: Pelt
Puurs						| Merge: Puurs-Sint-Amands
Sint-Amands					| Merge: Puurs-Sint-Amands
Waarschoot					| Merge: Lievegem
Zingem						| Merge: Kruisem
Zomergem					| Merge: Lievegem



Edited 1 time(s). Last edit at 05/22/2021 03:51PM by Pleu. (view changes)
Re: IMPORTANT new polygon data
May 19, 2021 05:27PM
Situation on my disabled tags
adjusted and re-enabled
21724, 21899, 32729, 32781, 40927, 46269, 46281, 46863, 48799, 53540, 53668, 56288
34855 re-enabled with the new county name after consultation with the CO

adjusted but were not disabled
13827, 46287, 56997 (Netherlands only)

awaiting info from CO
22742, 34855, 59764

to be checked with CO
24021, 24384



Edited 2 time(s). Last edit at 05/19/2021 09:33PM by vogelbird. (view changes)
Re: IMPORTANT new polygon data
May 20, 2021 06:17AM
I have fixed and re-enabled the checker listed against me: 56739



Edited 1 time(s). Last edit at 05/20/2021 06:19AM by GreyHams. (view changes)
Re: IMPORTANT new polygon data
May 20, 2021 08:06PM
Just realized there are a few checkers that's broken due to outdated JSON files used with PGC.GetJsonData(). Got a report about https://project-gc.com/Challenges/GC43JTY/29431 (by sloth96) for example. There are probably more of them.

I could try to backup all the json files tomorrow and run a script that automatically replaces those names we believe are safe to automatic update. However, there may very well be names that I can't update automatically, and there might very well be other issues with some of the files. I am not sure it's worth it that I start touching them.

Pleu has compiled this list with names that most likely are safe to use with search and replace.
http://1447.se/tmp/PGC-OSMB-Translate-v2.csv

If there are ways I can assist, please just ask.
Re: IMPORTANT new polygon data
May 21, 2021 03:48PM
From the thread https://project-gc.com/forum/read?8,51464,51464#msg-51464
I believe that the function:
PGC.GetSimplifiedRegionPolygons('United States')
is broken, at least the scripts
https://project-gc.com/Tools/Challenges?edit&scriptId=7284
and
https://project-gc.com/Tools/Challenges?edit&scriptId=2635
for at least the scripts that have the country set to 'United States'. I'm guessing this is due to the polygon changes, since they now return error.



Edited 1 time(s). Last edit at 05/21/2021 03:52PM by DrAcorn. (view changes)
Re: IMPORTANT new polygon data
May 21, 2021 03:52PM
@DrAcorn That is correct, that data doesn't exist anymore. I will investigate how it's used and what to do.
Re: IMPORTANT new polygon data
May 21, 2021 03:57PM
Seems like at least those two scripts are using GetSimplifiedRegionPolygons() for something completely different than its meant for. I knew about this method when I made this release, and I knew it would stop working, but I didn't think it was used.

I will implement a new method that actually does what these scripts wants. If anyone else knows something GetSimplifiedRegionPolygons() is used for, please tell me for what so we can find a solution.

@DrAcorn Do you think you can patch your version of the script if I supply a new method? Maybe fix pieterix's script as well (should be the same fix), and I can update his for him. Hoping to have a new method ready within the hour.
Re: IMPORTANT new polygon data
May 21, 2021 04:00PM
Happy to fix my script if I know how. That was the only method I could use to get a list of regions/counties. If life gives you lemons... *grin*
Re: IMPORTANT new polygon data
May 21, 2021 04:06PM
Adding these 3 new methods, should be available in 5-10 minutes:
/// @brief Returns a list of region names given a country
///
/// @param[in] string country
public static function GetRegionNamesInCountry($country) {

/// @brief Returns a list of county names given a country
///
/// @param[in] string country
public static function GetCountyNamesInCountry($country) {

/// @brief Returns a list of county names given a country
///
/// @param[in] string country
/// @param[in] string region
public static function GetCountyNamesInRegion($country, $region) {

Use them like this: PGC.GetRegionNamesInCountry('United States')

EDIT:
Tested it with this script, https://project-gc.com/Tools/Challenges?edit&tagId=60925
Release in two minutes.



Edited 1 time(s). Last edit at 05/21/2021 04:09PM by magma1447. (view changes)
Re: IMPORTANT new polygon data
May 21, 2021 05:51PM
I've adapted my script.
Re: IMPORTANT new polygon data
May 21, 2021 06:48PM
Mine’s been updated too now. Thanks for the quick fix. Btw, the new maps look great!
Re: IMPORTANT new polygon data
May 21, 2021 07:32PM
Broken scripts

Extended Contiguous Region Checker(by sloth96)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1623
Not working
Error log
[string ""]:352: attempt to call field 'GetSimplifiedRegionPolygons' (a nil value)


Extended Contiguous Region Checker(by arisoft)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=6408
Not working
Error log
[string ""]:363: attempt to call field 'GetSimplifiedRegionPolygons' (a nil value)


Extend Contiguous Nation Checker(by sloth96)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1648
Map Output not working


Extended Connected Counties and Regions(by sloth96)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1609
Not working
Error log
[string ""]:330: attempt to call field 'GetSimplifiedCountyPolygons' (a nil value)


Connected counties and regions(by Target.)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1391
Not working
Error log
[string ""]:290: attempt to call field 'GetSimplifiedCountyPolygons' (a nil value)


The oldest cache from N groups(by arisoft)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=3533
Map Output not working
Re: IMPORTANT new polygon data
May 21, 2021 10:00PM
I will implement a new version of GetSimplifiedRegionPolygons and GetSimplifiedCountyPolygons. The rest I haven't looked at yet.
Re: IMPORTANT new polygon data
May 21, 2021 11:54PM
I bet all those script depends on the now defunct GetSimplifiedRegionPolygons() and GetSimplifiedCountyPolygons().

Since the scripts don't need the polygons themselves (I think and hope) there is no sense to passing the whole polygon via the scripts. Not the least now when there are higher precision in the polygons. Project-GC don't have any "simplified" anymore either. Therefore I have a new idea of approach.

First, three new methods implemented:
  • GetRegionPolygonInfoFromCountry(country)
  • GetCountyPolygonInfoFromCountry(country)
  • GetCountyPolygonInfoFromRegion(country, region)

All three functions returns an object looking like this:
{ id: "text-numeric", "name": "text-name", "minLat": float, "maxLat": float, "minLon": float, "maxLon": float }

Use the returned map variable exactly as before, but instead of providing polygon data, just add the id that these functions returns. Proof of concept is written here:
https://project-gc.com/Tools/Challenges?edit&tagId=60934

This should be much more lightweight and efficient. The downside is that some scripts needs to be adjusted. I honestly don't feel to comfortable with LUA since I touch it like once per year. But if there is a script owned by someone who isn't active, I can update it if someone provides me new source code. Alternatively I can also move tags from one script to another (updated version of the original).

Consider this beta in the way that I might update the functions in different ways depending on feedback here. Besides that they should work, it's a fairly simple approach. I will release these callbacks in a few minutes. Any feedback and thoughts is welcome. If something is being done today that isn't possible anymore, please tell me about it.



Edited 1 time(s). Last edit at 05/22/2021 12:49AM by magma1447. (view changes)
Re: IMPORTANT new polygon data
May 22, 2021 12:42AM
I assume you meant GetRegionPolygonInfoFromCountry() instead, since the GetRegionNamesInCountry() call you added earlier is still operating as before and return a list and not an object.
Re: IMPORTANT new polygon data
May 22, 2021 12:49AM
@pieterix: Yes, I will edit my post. A bad copy/paste from my side.

The new calls are slower, mostly due to the area calculations.
Re: IMPORTANT new polygon data
May 22, 2021 12:56PM
I believe Target. is inactive, so I have updated the script
https://project-gc.com/Tools/Challenges?edit&scriptId=1391
in my dev script
https://project-gc.com/Tools/Challenges?edit&scriptId=6927
I assume you could copy it over from that?
I can do the same for the other scripts if need be.
Re: IMPORTANT new polygon data
May 22, 2021 01:05PM
Thanks. I have copied your code into his. I also added his old code into the history table just in case.

And yes, Target is inactive, sadly. I knew him in real life. Something got too much in his life and he seems to have cut of most everything and everyone. A real loss from many aspects.
Re: IMPORTANT new polygon data
May 22, 2021 04:33PM
Couple more updates, both
https://project-gc.com/Tools/Challenges?edit&scriptId=1623 (lines 1-487)
and
https://project-gc.com/Tools/Challenges?edit&scriptId=1609 (lines 488-937)
in my dev script
https://project-gc.com/Tools/Challenges?edit&scriptId=6927

Additionally, I spotted another problem - when the second of those scripts is used in
https://project-gc.com/Challenges/GC8JKC5/48489
the json data it retrieves uses the old county names rather than the updated ones, and as such the checker returns fail for everyone currently. I don't know if this is also the case for other json data sets.
Re: IMPORTANT new polygon data
May 22, 2021 04:53PM
@DrAcorn
Most of the connections_sets need updating which is causing me a lot of work because as far as I know all the sets where finally made by me. But first the scripts need to be operational for testing.
I notified sloth96 this morning but have not heard back from him.
Re: IMPORTANT new polygon data
May 22, 2021 04:55PM
okay, at least the problem is known then :)
Re: IMPORTANT new polygon data
May 22, 2021 05:00PM
I am not at home anymore today, I needed a break. I have an email thread with sloth, but he got some outdated info from me, I will update him later today, and link this thread.

@vogelbird: I can probably write a script that tells which polygons connects with which. I can play around with it later today or tomorrow. Might be great to have in the future as well. Do you think you could supply me with a really small example so I know how it should look? Is hard to get the overview of files with hundreds of items.
Re: IMPORTANT new polygon data
May 22, 2021 05:23PM
Is there a proof-of-concept script that shows mapping Countries, Regions, and Counties.? I cannot see how to map countries with the new API calls now.
Re: IMPORTANT new polygon data
May 22, 2021 05:27PM
Was countries possible before? I missed that. I will have to add an api method for that as well then. For regions and counties I add an example script here somewhere. Search for GetRegionPolygonInfoFromCountry. I am just on my phone for most of the day today, so a bit handicapped.
Re: IMPORTANT new polygon data
May 22, 2021 05:42PM
No countries were not possible before but someone was complaining that a connected country checker was showing a bad map.

magma1447 Wrote:
-------------------------------------------------------
> Was countries possible before? I missed that. I
> will have to add an api method for that as well
> then. For regions and counties I add an example
> script here somewhere. Search for
> GetRegionPolygonInfoFromCountry. I am just on my
> phone for most of the day today, so a bit
> handicapped.
Re: IMPORTANT new polygon data
May 22, 2021 05:29PM
Throw me an email to implement GetCountryPolygonInfo() so I don't forget. Shouldn't take me more than 15 minutes when I am at my battle station.
Re: IMPORTANT new polygon data
May 22, 2021 06:34PM
I think my scripts are all patched. Let me know if otherwise.
Re: IMPORTANT new polygon data
May 22, 2021 06:35PM
That was fast. :)
Re: IMPORTANT new polygon data
May 22, 2021 09:49PM
Folks are commenting to me that the maps are not showing up as they used to. Some my checkers are giving a "Unknown error fetching map output" message as they try to render the map.

For now I am telling folks growing pains.

This checker https://project-gc.com/Challenges/GC43JTY/29431 gives the error with user "RW and VJ".



Edited 1 time(s). Last edit at 05/22/2021 09:53PM by sloth96. (view changes)
Re: IMPORTANT new polygon data
May 22, 2021 10:46PM
It's a memory issue, the new polygons are much more detailed. There are pretty much three ways to solve this:
* Increase amount of memory allowed. Will still be very heavy for the client.
* Create simpler polygons again. Will look uglier (like before), and also takes some time to produce, only needs to be run once though, not for every checker result obviously.
* Re-implement the map thing so that it loads one polygon per ajax call. The map implementation right now sucks a lot. It was just meant as a proof of concept, then I have never gotten to actually rewrite it. Rewrite it correctly would be a fairly big job, but something I wish to do. Project-GC uses a very old Leaflet library for most maps, and that should be upgraded, which doesn't work out of the box, so things needs to be rewritten.

The third solution is the one I want, but for now I went with the first. I will have to get back to this. I haven't released the changes yet but I will try to do it within 1-2 hours.
Sorry, you do not have permission to post/reply in this forum.