IMPORTANT new polygon data

Re: IMPORTANT new polygon data

magma1447 — Sat, 22 May 2021 20:46:13 +0000

It's a memory issue, the new polygons are much more detailed. There are pretty much three ways to solve this:
* Increase amount of memory allowed. Will still be very heavy for the client.
* Create simpler polygons again. Will look uglier (like before), and also takes some time to produce, only needs to be run once though, not for every checker result obviously.
* Re-implement the map thing so that it loads one polygon per ajax call. The map implementation right now sucks a lot. It was just meant as a proof of concept, then I have never gotten to actually rewrite it. Rewrite it correctly would be a fairly big job, but something I wish to do. Project-GC uses a very old Leaflet library for most maps, and that should be upgraded, which doesn't work out of the box, so things needs to be rewritten.

The third solution is the one I want, but for now I went with the first. I will have to get back to this. I haven't released the changes yet but I will try to do it within 1-2 hours.

Re: IMPORTANT new polygon data

sloth96 — Sat, 22 May 2021 19:49:36 +0000

Folks are commenting to me that the maps are not showing up as they used to. Some my checkers are giving a "Unknown error fetching map output" message as they try to render the map.

For now I am telling folks growing pains.

This checker https://project-gc.com/Challenges/GC43JTY/29431 gives the error with user "RW and VJ".

Re: IMPORTANT new polygon data

magma1447 — Sat, 22 May 2021 16:35:04 +0000

That was fast. :)

Re: IMPORTANT new polygon data

sloth96 — Sat, 22 May 2021 16:34:15 +0000

I think my scripts are all patched. Let me know if otherwise.

Re: IMPORTANT new polygon data

sloth96 — Sat, 22 May 2021 15:42:09 +0000

No countries were not possible before but someone was complaining that a connected country checker was showing a bad map.

magma1447 Wrote:
-------------------------------------------------------
> Was countries possible before? I missed that. I
> will have to add an api method for that as well
> then. For regions and counties I add an example
> script here somewhere. Search for
> GetRegionPolygonInfoFromCountry. I am just on my
> phone for most of the day today, so a bit
> handicapped.

Re: IMPORTANT new polygon data

magma1447 — Sat, 22 May 2021 15:29:26 +0000

Throw me an email to implement GetCountryPolygonInfo() so I don't forget. Shouldn't take me more than 15 minutes when I am at my battle station.

Re: IMPORTANT new polygon data

magma1447 — Sat, 22 May 2021 15:27:34 +0000

Was countries possible before? I missed that. I will have to add an api method for that as well then. For regions and counties I add an example script here somewhere. Search for GetRegionPolygonInfoFromCountry. I am just on my phone for most of the day today, so a bit handicapped.

Re: IMPORTANT new polygon data

sloth96 — Sat, 22 May 2021 15:23:42 +0000

Is there a proof-of-concept script that shows mapping Countries, Regions, and Counties.? I cannot see how to map countries with the new API calls now.

Re: IMPORTANT new polygon data

magma1447 — Sat, 22 May 2021 15:00:50 +0000

I am not at home anymore today, I needed a break. I have an email thread with sloth, but he got some outdated info from me, I will update him later today, and link this thread.

@vogelbird: I can probably write a script that tells which polygons connects with which. I can play around with it later today or tomorrow. Might be great to have in the future as well. Do you think you could supply me with a really small example so I know how it should look? Is hard to get the overview of files with hundreds of items.

Re: IMPORTANT new polygon data

DrAcorn — Sat, 22 May 2021 14:55:37 +0000

okay, at least the problem is known then :)

Re: IMPORTANT new polygon data

vogelbird — Sat, 22 May 2021 14:53:35 +0000

@DrAcorn
Most of the connections_sets need updating which is causing me a lot of work because as far as I know all the sets where finally made by me. But first the scripts need to be operational for testing.
I notified sloth96 this morning but have not heard back from him.

Re: IMPORTANT new polygon data

DrAcorn — Sat, 22 May 2021 14:33:04 +0000

Couple more updates, both
https://project-gc.com/Tools/Challenges?edit&scriptId=1623 (lines 1-487)
and
https://project-gc.com/Tools/Challenges?edit&scriptId=1609 (lines 488-937)
in my dev script
https://project-gc.com/Tools/Challenges?edit&scriptId=6927

Additionally, I spotted another problem - when the second of those scripts is used in
https://project-gc.com/Challenges/GC8JKC5/48489
the json data it retrieves uses the old county names rather than the updated ones, and as such the checker returns fail for everyone currently. I don't know if this is also the case for other json data sets.

Re: IMPORTANT new polygon data

magma1447 — Sat, 22 May 2021 11:05:26 +0000

Thanks. I have copied your code into his. I also added his old code into the history table just in case.

And yes, Target is inactive, sadly. I knew him in real life. Something got too much in his life and he seems to have cut of most everything and everyone. A real loss from many aspects.

Re: IMPORTANT new polygon data

DrAcorn — Sat, 22 May 2021 10:56:41 +0000

I believe Target. is inactive, so I have updated the script
https://project-gc.com/Tools/Challenges?edit&scriptId=1391
in my dev script
https://project-gc.com/Tools/Challenges?edit&scriptId=6927
I assume you could copy it over from that?
I can do the same for the other scripts if need be.

Re: IMPORTANT new polygon data

magma1447 — Fri, 21 May 2021 22:49:26 +0000

@pieterix: Yes, I will edit my post. A bad copy/paste from my side.

The new calls are slower, mostly due to the area calculations.

Re: IMPORTANT new polygon data

pieterix — Fri, 21 May 2021 22:42:05 +0000

I assume you meant GetRegionPolygonInfoFromCountry() instead, since the GetRegionNamesInCountry() call you added earlier is still operating as before and return a list and not an object.

Re: IMPORTANT new polygon data

magma1447 — Fri, 21 May 2021 21:54:49 +0000

I bet all those script depends on the now defunct GetSimplifiedRegionPolygons() and GetSimplifiedCountyPolygons().

Since the scripts don't need the polygons themselves (I think and hope) there is no sense to passing the whole polygon via the scripts. Not the least now when there are higher precision in the polygons. Project-GC don't have any "simplified" anymore either. Therefore I have a new idea of approach.

First, three new methods implemented:

GetRegionPolygonInfoFromCountry(country)
GetCountyPolygonInfoFromCountry(country)
GetCountyPolygonInfoFromRegion(country, region)

All three functions returns an object looking like this:
{ id: "text-numeric", "name": "text-name", "minLat": float, "maxLat": float, "minLon": float, "maxLon": float }

Use the returned map variable exactly as before, but instead of providing polygon data, just add the id that these functions returns. Proof of concept is written here:
https://project-gc.com/Tools/Challenges?edit&tagId=60934

This should be much more lightweight and efficient. The downside is that some scripts needs to be adjusted. I honestly don't feel to comfortable with LUA since I touch it like once per year. But if there is a script owned by someone who isn't active, I can update it if someone provides me new source code. Alternatively I can also move tags from one script to another (updated version of the original).

Consider this beta in the way that I might update the functions in different ways depending on feedback here. Besides that they should work, it's a fairly simple approach. I will release these callbacks in a few minutes. Any feedback and thoughts is welcome. If something is being done today that isn't possible anymore, please tell me about it.

Re: IMPORTANT new polygon data

magma1447 — Fri, 21 May 2021 20:00:18 +0000

I will implement a new version of GetSimplifiedRegionPolygons and GetSimplifiedCountyPolygons. The rest I haven't looked at yet.

Re: IMPORTANT new polygon data

vogelbird — Fri, 21 May 2021 17:32:33 +0000

Broken scripts

Extended Contiguous Region Checker(by sloth96)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1623
Not working
Error log
[string ""]:352: attempt to call field 'GetSimplifiedRegionPolygons' (a nil value)

Extended Contiguous Region Checker(by arisoft)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=6408
Not working
Error log
[string ""]:363: attempt to call field 'GetSimplifiedRegionPolygons' (a nil value)

Extend Contiguous Nation Checker(by sloth96)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1648
Map Output not working

Extended Connected Counties and Regions(by sloth96)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1609
Not working
Error log
[string ""]:330: attempt to call field 'GetSimplifiedCountyPolygons' (a nil value)

Connected counties and regions(by Target.)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=1391
Not working
Error log
[string ""]:290: attempt to call field 'GetSimplifiedCountyPolygons' (a nil value)

The oldest cache from N groups(by arisoft)
https://project-gc.com/Tools/Challenges?edit&addTag&scriptId=3533
Map Output not working

Re: IMPORTANT new polygon data

DrAcorn — Fri, 21 May 2021 16:48:26 +0000

Mine’s been updated too now. Thanks for the quick fix. Btw, the new maps look great!

Re: IMPORTANT new polygon data

pieterix — Fri, 21 May 2021 15:51:48 +0000

I've adapted my script.

Re: IMPORTANT new polygon data

magma1447 — Fri, 21 May 2021 14:06:33 +0000

Adding these 3 new methods, should be available in 5-10 minutes:
/// @brief Returns a list of region names given a country
///
/// @param[in] string country
public static function GetRegionNamesInCountry($country) {

/// @brief Returns a list of county names given a country
///
/// @param[in] string country
public static function GetCountyNamesInCountry($country) {

/// @brief Returns a list of county names given a country
///
/// @param[in] string country
/// @param[in] string region
public static function GetCountyNamesInRegion($country, $region) {

Use them like this: PGC.GetRegionNamesInCountry('United States')

EDIT:
Tested it with this script, https://project-gc.com/Tools/Challenges?edit&tagId=60925
Release in two minutes.

Re: IMPORTANT new polygon data

pieterix — Fri, 21 May 2021 14:00:19 +0000

Happy to fix my script if I know how. That was the only method I could use to get a list of regions/counties. If life gives you lemons... *grin*

Re: IMPORTANT new polygon data

magma1447 — Fri, 21 May 2021 13:57:55 +0000

Seems like at least those two scripts are using GetSimplifiedRegionPolygons() for something completely different than its meant for. I knew about this method when I made this release, and I knew it would stop working, but I didn't think it was used.

I will implement a new method that actually does what these scripts wants. If anyone else knows something GetSimplifiedRegionPolygons() is used for, please tell me for what so we can find a solution.

@DrAcorn Do you think you can patch your version of the script if I supply a new method? Maybe fix pieterix's script as well (should be the same fix), and I can update his for him. Hoping to have a new method ready within the hour.

Re: IMPORTANT new polygon data

magma1447 — Fri, 21 May 2021 13:52:07 +0000

@DrAcorn That is correct, that data doesn't exist anymore. I will investigate how it's used and what to do.

Re: IMPORTANT new polygon data

DrAcorn — Fri, 21 May 2021 13:48:31 +0000

From the thread https://project-gc.com/forum/read?8,51464,51464#msg-51464
I believe that the function:
PGC.GetSimplifiedRegionPolygons('United States')
is broken, at least the scripts
https://project-gc.com/Tools/Challenges?edit&scriptId=7284
and
https://project-gc.com/Tools/Challenges?edit&scriptId=2635
for at least the scripts that have the country set to 'United States'. I'm guessing this is due to the polygon changes, since they now return error.

Re: IMPORTANT new polygon data

magma1447 — Thu, 20 May 2021 18:06:23 +0000

Just realized there are a few checkers that's broken due to outdated JSON files used with PGC.GetJsonData(). Got a report about https://project-gc.com/Challenges/GC43JTY/29431 (by sloth96) for example. There are probably more of them.

I could try to backup all the json files tomorrow and run a script that automatically replaces those names we believe are safe to automatic update. However, there may very well be names that I can't update automatically, and there might very well be other issues with some of the files. I am not sure it's worth it that I start touching them.

Pleu has compiled this list with names that most likely are safe to use with search and replace.
http://1447.se/tmp/PGC-OSMB-Translate-v2.csv

If there are ways I can assist, please just ask.

Re: IMPORTANT new polygon data

GreyHams — Thu, 20 May 2021 04:17:46 +0000

I have fixed and re-enabled the checker listed against me: 56739

Re: IMPORTANT new polygon data

vogelbird — Wed, 19 May 2021 15:27:51 +0000

Situation on my disabled tags
adjusted and re-enabled
21724, 21899, 32729, 32781, 40927, 46269, 46281, 46863, 48799, 53540, 53668, 56288
34855 re-enabled with the new county name after consultation with the CO

adjusted but were not disabled
13827, 46287, 56997 (Netherlands only)

awaiting info from CO
22742, ~~34855~~, 59764

to be checked with CO
24021, 24384

Re: IMPORTANT new polygon data

Pleu — Wed, 19 May 2021 12:15:37 +0000

No worries pieterix, I've spent some hours with this data so I have a head start. :)

Here's a list of the merges/divides that I found, if the country/area of the disabled checker is not on this list the reason for disable is some version of border changes. Most likely the reason can be found in the list of changes from magma1447's first post, some countries will for example have a different number of regions that I have not labeled as a merge for different reasons.

Canada
Division No.  1 Avalon Peninsula-St. John's	| Labrador and Newfoundland
Francheville					| Les Chenaux and Trois-Rivières
Minganie--Le Golfe-du-Saint-Laurent		| Minganie and Le Golfe-du-Saint-Laurent
Sept-Rivičres--Caniapiscau			| Caniapiscau and Sept-Rivières

Luxembourg
Boevange-sur-Attert				| Merge: Helperknapp
Hobscheid					| Merge: Habscht
Mompach						| Merge: Rosport-Mompach
Rosport						| Merge: Rosport-Mompach
Septfontaines					| Merge: Habscht
Tuntange					| Merge: Helperknapp

Netherlands
Appingedam					| Merge: Eemsdelta
Delfzijl					| Merge: Eemsdelta
Haaren						| Merge with Oisterwijk as Oisterwijk
Loppersum					| Merge: Eemsdelta
Oisterwijk					| Merge with Haaren as Oisterwijk

Italy
Carbonia-Iglesias				| Merge: Sud Sardegna
Medio Campidano					| Merge: Sud Sardegna
Nuoro						| Merge with Ogliastra as Nuoro.
Ogliastra					| Merge with Nuoro as Nuoro.
Olbia-Tempio					| Merge with Sassari as Sassari.
Sassari						| Merge with Olbia-Tempio as Sassari.

Finland
Honkajoki					| Merged as Kankaanpää
Valtimo						| Merged as Nurmes
Nurmes						| Merged as Nurmes.
Kankaanpää					| Merged as Kankaanpää.

Switzerland
Alterswil (10)					| Merge: Tafers
Altwis (3)					| Merge: Hitzkirch
Arconciel (10)					| Merge: Bois-d’Amont
Aubonne (22)					| Merge: Aubonne
Bagnes (23)					| Merge: Val de Bagnes
Bauen (4)					| Merge: Seedorf (UR)
Brione (Verzasca) (21)				| Merge: Verzasca
Casti-Wergenstein (18)				| Merge: Muntogna da Schons
Charrat (23)					| Merge: Martigny
Cheiry (10)					| Merge: Surpierre
Chur (18)					| Merge: Chur
Corcelles-Cormondrèche (24)			| Merge: Neuchâtel
Corippo (21)					| Merge: Verzasca
Donat (18)					| Merge: Muntogna da Schons
Ependes (FR) (10)				| Merge: Bois-d’Amont
Frasco (21)					| Merge: Verzasca
Gettnau (3)					| Merge: Willisau
Gänsbrunnen (11)				| Merge: Welschenrohr-Gänsbrunnen
Haldenstein (18)				| Merge: Chur
Hindelbank (2)					| Merge: Hindelbank
Hitzkirch (3)					| Merge: Hitzkirch
Langenthal (2)					| Merge: Langenthal
Le Locle (24)					| Merge: Le Locle
Les Brenets (24)				| Merge: Le Locle
Lohn (GR) (18)					| Merge: Muntogna da Schons
Martigny (23)					| Merge: Martigny
Mathon (18)					| Merge: Muntogna da Schons
Miège (23)					| Merge: Noble-Contrée
Montherod (22)					| Merge: Aubonne
Mötschwil (2)					| Merge: Hindelbank
Neuchâtel (24)					| Merge: Neuchâtel
Obersteckholz (2)				| Merge: Langenthal
Peseux (24)					| Merge: Neuchâtel
Riggisberg (2)					| Merge: Riggisberg
Rohr (SO) (11)					| Merge: Stüsslingen
Rümligen (2)					| Merge: Riggisberg
Seedorf (UR) (4)				| Merge: Seedorf (UR)
Senèdes (10)					| Merge: Bois-d’Amont
Sonogno (21)					| Merge: Verzasca
St. Antoni (10)					| Merge: Tafers
Stüsslingen (11)				| Merge: Stüsslingen
Surpierre (10)					| Merge: Surpierre
Tafers (10)					| Merge: Tafers
Valangin (24)					| Merge: Neuchâtel
Venthône (23)					| Merge: Noble-Contrée
Veyras (23)					| Merge: Noble-Contrée
Vogorno (21)					| Merge: Verzasca
Vollèges (23)					| Merge: Val de Bagnes
Willisau (3)					| Merge: Willisau
Welschenrohr (11)				| Merge: Welschenrohr-Gänsbrunnen

Belgium
Aalter						| Merge with Knesselare as Aalter.
Deinze 						| Merge with Nevele as Deinze
Knesselare					| Merge: Aalter.
Kruishoutem					| Merge: Kruisem
Lovendegem					| Merge: Lievegem
Meeuwen-Gruitrode				| Merge: Oudsbergen
Neerpelt					| Merge: Pelt
Nevele						| Merge: Deinze
Opglabbeek					| Merge: Oudsbergen
Overpelt					| Merge: Pelt
Puurs						| Merge: Puurs-Sint-Amands
Sint-Amands					| Merge: Puurs-Sint-Amands
Waarschoot					| Merge: Lievegem
Zingem						| Merge: Kruisem
Zomergem					| Merge: Lievegem