Polygon definitions

From Project-GC
Jump to: navigation, search

Page description

This is an admin/moderator page to which only specific users have access. The page is used to create rules used when importing polygon data from OpenStreetMap into Project-GC. More precisely, Project-GC imports from OSM-Boundaries.com, which in turn has imported its data from OpenStreetMap. The data is then used for calculations regarding placing geocaches in their correct region/county. The data is also used for rendering maps, as for example the maps in Profile stats.

Country selector

The country selector lists every country available in the Geocaching LIVE api. By selecting a new entry the country is automatically loaded, allow it 1-2 seconds to retrieve its data.

Data shown

In the first section there is basic metadata about the country shown, like the name of the country, and the ID in the API. Then it shows who has created the last definitions of rules and when. On the right side there is a list of region names used by Geocaching.com.

The region names in Project-GC must match those of Geocaching.com. Use relevant rule options to make this happen.

Source site

Project-GC uses OSM-Boundaries.com as a source when importing its polygon data. This site is also a great source to use when building the import rules. OSM-Boundaries.com in turn has imported its data from data dumps of OpenStreetMap.

Rules

For Project-GC to know where a country (or region/county) exists in the world it needs the borders. In the most simple cases we can just match a country with an ID in the OpenStreetMap database and it's done. However it's not always that simple, especially for countries. For example the polygon for Finland includes Aland Islands while Aland Islands is its own country according to Geocaching.com. In this case we need to make more advanced rules defining and calculating new polygons in Project-GC.

The rule-set for countries differs a bit from regions/counties, the reason is that a country should always end with a single (multi)polygon, while regions and counties needs one (multi)polygon per region/county.

All import rules are created as JSON-strings. Generally it consists of a list of rules for the polygons itself, and the a list of rules for post-processing names. When the import tool runs it will first run all rules under polygons, and then the rules under postprocessing. Each rule within these two is executed in the order it's listed as.

You can look at already defined definitions to get an overall understanding how the definitions should look.

Base OSMB Database

This is the OSM-Boundaries database version that will be used as a basis for this country's polygon import. It can then be overridden for specific rules later on. It's not always using the latest since all data must be reviewed, it's not uncommon that OpenStreetMap has brooken or incompatible data. A typical use-case for overriding is when Openstreetmap has broken data.

Country rules

All countries must be defined in Project-GC. This is primarily used when rendering maps for Profile stats, and the maps would look a bit funny if a country was missing.

The country rules are the simplest. There is no post-processing of names and there are only two types of rules available.

  • includeOsmIds - List of integer IDs from OSM-Boundaries.com. These are IDs of polygons that will be included in the country.
  • subtractOsmIds - List of integer IDs from OSM-Boundaries.com. The IDs listed here will be subtracted from the includeOsmIds. For example in the case of Finland, the ID of Finland can be chosen as the country and then Aland Island can be subtracted.

Each rule can also use the db attribute to specify another database than the base one.

Rather use one big polygon as include and a few subtract than a lot of includes. For example, USA either requires including the whole USA and then subtracting 3-5 polygons, or making a list of ~50 polygons for inclusion. There are two good reasons to rather include the whole country and then make a few subtracts.

  • It's faster during the import phase.
  • It makes a more readable definition.

The country polygons ignores the names in OpenStreetMap, names from the API will be used instead. This is also why the post-processing rule-set hasn't been implemented.

Example of how to define Norway: 

{ "polygons": [ "includeOsmIds":[-2978650],"subtractOsmIds":[-2425963,-1337126,-1337397] ] }

Region rules

The best way to add rules is by using the Add rule button on the right side of the input form. This will help you keeping the syntax correct. All values that are defined as either 0 (zero) or empty strings ("") are required to fill in. Keys that has empty lists as values are optional.

There are several types of rules that can be used to import polygons, and they should be listed under the key polygons. The rules are used by setting type to the ID of the rule, and then defining at least the required keys.

  • adminLevel - This is the easiest and most common way to. In a perfect world this would be the only rule needed. Basically it includes everything with a specified adminLevel under a parentOsmId.
    • Required keys
      • parentOsmId
      • adminLevel
    • Optional keys
      • excludeOsmIds - Exclude specific osmIds even though they match the parentOsmId and the adminLevel.
      • excludeParentOsmIds - Exclude anything that has these osmIds as a parent in OSM-Boundaries tree-view.
      • subtractOsmIds - Subtract polygons listed here. This is a lot slower than excludeParentOsmIds and should only be used when we actually need to punch a whole into something.
      • allowNonAdministrative - By default the system requires that OSM's tag boundary is set to administrative. If this is not wished for, set this to true (do not quote it).
      • db - Override the base database.
  • immediateChildrenOf - Add all immediate children of this osmb-id, regardless of the admin-level.
    • Required keys
    • Optional keys
      • boundary - Require the OSM tag boundary to be set to this string value, for example administrative.
      • db - Override the base OSMB database.
      • excludeOsmIds - Exclude these IDs. A list with IDs as shown on OSM-Boundaries.com.
  • osmIds - This rule allows adding a list of OSM-IDs as regions.
    • Required keys
      • osmIds - List of osmIds
    • Optional keys
      • subtractOsmIds - Subtract polygons from the ones that are being imported. If possible, try to split the rule into two rules, where only those needing the subtract are listed together with the subtract. And those not needing subtraction for themself. This will increase the import performance a lot.
      • fillHoles - Boolean, if true all holes inside the polygon will be filled as if there were no holes. If it was used on Sweden there would be no Vättern och Vänern for example.
      • db - Override the base database.
  • union - If there aren't polygons available that works, but there are other polygons which could be joined together to create what's needed this rule-type can be used.** ** Required keys
      • osmIds - List of osmIds
    • Optional keys
      • subtractOsmIds
      • fillHoles - Boolean, if true all holes inside the polygon will be filled as if there were no holes. If it was used on Sweden there would be no Vättern och Vänern for example.
      • db - Override the base database.
  • file - This is a a special case option that shouldn't be used in general. But for Canada it has been used. Basically it uses stored geojson files instead of OSM-Boundaries.com as a source. Canada actually is a mix of Openstreetmap and Census data.
    • Required keys
      • filename
      • propertyKey
    • Optional keys
      • propertyValue
      • name
      • fillHoles - Boolean, if true all holes inside the polygon will be filled as if there were no holes. If it was used on Sweden there would be no Vättern och Vänern for example.

Then there are a number of rules for post-processing names as well. Since they are executed in the order they are listed the order might be important.

  • renameOsmId
    • Required keys
      • osmId
      • name
  • removePrefixes
    • Required keys
      • prefixes - List of strings.
  • removeSuffixes
    • Required keys
      • suffixes - List of strings.
  • prefixToSuffix
    • Required keys
      • prefix
      • suffix - Don't forget to start with a space character if wished for.
  • suffixBasedOnRegionName
    • Required keys
      • regionName
      • suffix
  • removeMultiByteAbove - Will remove all characters in the string that has an utf8 value higher than maxValue. Be aware that there might be some left-overs like dash after this rule has been run. Running a removeSuffix afterwards can help with that.
    • Required keys
      • maxValue
  • convertCase - Convert the casing using PHPs mb_convert_case() function, using the mode specified in the require mode option.
    • Required keys
      • mode - Valid modes are: MB_CASE_UPPER, MB_CASE_LOWER, MB_CASE_TITLE, MB_CASE_FOLD, MB_CASE_UPPER_SIMPLE, MB_CASE_LOWER_SIMPLE, MB_CASE_TITLE_SIMPLE, MB_CASE_FOLD_SIMPLE.

Each rule can be repeated as many times as needed. Every rule can also include the optional key comment. It can be used to document why that specific rule exists, or why a specific osmId is subtracted for example.

Finally there is an option that can be set on the same level as polygons and postprocessing

  • internationalNames - The import scripts defaults to using local region and county names. But we also want to avoid using names that are impossible to write for users who are most familiar with the latin alphabet. So for example when the names are in Asian or Arabic variants we can use this parameter to override the default. But again, in cases where Geocaching.com has regions, it's most important that we match those. It's a boolean option, which defaults to false. Example:
{ "internationalNames": true }

When regions exists at Geocaching.com the region names in Project-GC must match these when possible. If they don't, it should clearly be stated in the #Notes -field, and why.

Note that subtractOsmIds is an expensive operation. When reasonable (not too much work), try to only use subtractOsmIds together with those osmIds that actually requires it. Example: 

 { "type": "osmIds", "osmIds": [1,2,3,4,5,6,7,8,9], "subtractOsmIds": [1000] }

might be a lot faster as: 

 { "type": "osmIds", "osmIds": [1,2,3,4,5,6,7,8] }
 { "type": "osmIds", "osmIds": [9], "subtractOsmIds": [1000] }

This of course requires that osmId 9 is the only polygon that needs to subtract. Similar solution could be used with adminLevel by using excludeOsmIds if there is only a handful of polygons needing a subtract. That handful number of polygons can then be added with an osmIds-rule instead. Example: 

 { "type": "adminLevel", "parentOsmId": 123, "adminLevel": 4, "subtractOsmIds": [1000] }

might be faster with this method: 

 { "type": "adminLevel", "parentOsmId": 123, "adminLevel": 4, "excludeOsmIds": [100, 200] }
 { "type": "osmIds", "osmIds": [100, 200], "subtractOsmIds": [1000] }


Full example: 

 {
   "internationalNames": true,
   "polygons": [
     { "type": "osmIds", "osmIds": [123], "subtractOsmIds": [] },
     { "type": "osmIds", "osmIds": [456], "subtractOsmIds": [] },
     { "type": "adminLevel", "parentOsmId": 123, "adminLevel": 4, "excludeOsmIds": [], "subtractOsmIds": [] },
     { "type": "adminLevel", "parentOsmId": 123, "adminLevel": 6, "excludeOsmIds": [], "subtractOsmIds": [] },
     { "type": "union", "name": "Malmö", "osmIds": [123, 456], "subtractOsmIds": [] },
     { "type": "union", "name": "Lund", "osmIds": [321, 654], "subtractOsmIds": [] }
   ],
   "postprocessing": [
     { "type": "renameOsmId", "osmId": -54409, "name": "Scania" },
     { "type": "removePrefixes", "prefixes": ["foo"] },
     { "type": "removeSuffixes", "suffixes": ["bar"] },
     { "type": "prefixToSuffix", "prefix": "Landkreis", "suffix": " (Lkr)" },
     { "type": "suffixBasedOnRegionName", "regionName": "California", "suffix": " (Ca)" }
   ]
 }


County rules

The county rules works exactly like the region rules.

Notes

This is a free-text form where you have the option to leave some notes. If there is nothing to say, just leave it empty. But for example when making a subtract it can be worth mentioning what's being subtracted.

Saving errors

There isn't much feedback to the web-ui yet. If the input form turns red the JSON is invalid, and therefore couldn't be saved. Upon save the JSON is also verified in several ways. For example it looks for keys that shouldn't exist (where they exist), and that the values are of the correct type (list, integers, strings, ...).

Preview data

There is not yet any way to preview the data. Depending on how complex the definitions are it takes hours to calculate new polygons which makes it troublesome to show the moderator some form of preview. Ultimately we should also calculate the number of geocaches included in the polygons, both with old and new definitions - This would be even more expensive.

Historical data

Historical definitions exists in the database but can not yet be viewed on the web. If they are needed due to mistakenly destroying complex definitions we can retrieve them fairly easily.

Notes

Retrieved from "https://project-gc.com/wiki/index.php?title=Polygon_definitions&oldid=1904"