Re: GetOldestCaches allow array for regions/countries filter?

Hello,

I have been using GetOldestCaches to fetch the oldest cache in each of the 50 US states.
http://project-gc.com/Challenges/GC6K9N3/23184

Several complained about the script not completing for them. After several attempts to speed it up, I bootstrapped with just getting the oldest caches in the country. The API call takes a string and not an array. This meant I had to cycle through one region at the time to finish it out. Each GetOldestCaches took about ~10 seconds so this took a while. Runtimes for me are about 130 seconds which seems undesirably long for a single checker.

My initial hope was to use an array of regions to speed it up and minimize api calls/server loading/ and the like.

I have a similar script with countries that has the same problems but the times don't seem so bad.

Existing code uses this api call so perhaps a seconds is needed.

Right now, I would rate this nice to have not necessary.

I am sorry for the late response. I actually haven't seen the request, which is quite bad. I should watch this subforum, but that isn't possible in the forum engine. I know there are a few patches and/or addons for that. I will look into if we can fix something.

I will prepare a patch for this.

A patch has been created. It's currently included with the http://project-gc.com/forum/read?6,7273 commit so I will hold off the release. I assume it's not urgent since I am quite late on it anyway.

It is not urgent. It is more of a performance enhancement than functuonality.

Fewer timeouts are better. If this helps great.

Given the recent speedup in getoldest this may no longer be required. It would still allow some processing to make fewer calls but my scripts are much farther away from from timing out and resources may be best spent elsewhere.

This is included with the patch that was released today. It's up to you if you want to rewrite or not.

But to be clear. It will fetch the X oldest for region A and B. If X is 100, that is a 100 in total. Not 100+100.

I assume this was what you wanted. To me it's the only thing that makes sense, otherwise it's just as easy and fast to make two calls (since it's not async anyway).

Excellent. I am fine with the limit not increasing.

Just for clarity, I thought the new approach was around 1000 depending on date distribution.

The max limit has changed. It's not max 1000. The default is 100 though. But a GetOldest({limit: 1000}) will get you 1000.

This reminds me that I have forgotten to update the autogenerated documentation of the API. I will do that today.

I tried getting this working with using region as an array. It did not work for me. Using just a single string worked fine.

Maybe I am missing something.

No it's not you, it's me. As I mentioned I didn't test the patch. When I reviewed the code now it was obvious what was wrong.

I have released a new version. I still haven't tested it though. If it doesn't work now I will make a proper test.

http://project-gc.com/Challenges/GC6K9P6/23464 does not even run. It is the dreaded silent death.

I created http://project-gc.com/Challenges//24954 which is the same tag and only does GetOldest queries with either empty or arrays and that dies silently as well.

These both work on regions. I have not looked at countries.

Oops, I lost a line of code when making the fix earlier.

Sorry about all the issues, working on some other things and feel a bit stressed and distracted today. I tested both checkers that you linked, they execute now.

Don't get stressed on my account.

Glad for the fix.

I have good news and I have bad news.

The Good news is has many faces. First, this request has has been sitting here for a while and things have been ok. It can wait a little longer. Second, looking at the oldest caches with a limit of 1000 and the other changes made in the other thread has made most of the speedup hoped for by this not needed for existing checkers. We are down from 180 seconds to run the oldest in N regions on my account to about 1.2 seconds. This is spectacular by most any benchmarking standard.

The bad news is that the array input is not treated correctly and does not seem to be working properly for me. On my debugging version, I cranked down the limit so the initial shotgun would not catch all the oldest in the 50 ish US states and the array version started reporting incorrect results.

The two possibilities are I am still learning lua nuances. The other is the API call is broken.

The active checkers are working fine and faster than ever. This can go somewhere down on the priority list. I would prefer not 4 months down. I will take a look at my Lua syntax over the weekend. There is no need to make this near the top priority.

You are right. I should have tested it from the start.

I have now created a test script and corrected the last bug. It's working now, at least as I intended it to. Example, the oldest in country=Sweden, region=[Skåne, Stockholm, Västra Götaland] are these:

Quote

| GC4C | 2000-06-14 | Stockholm |
| GC4D | 2000-08-11 | Stockholm |
| GC16B | 2001-01-14 | Västra Götaland |
| GC602 | 2001-04-03 | Västra Götaland |
| GC721 | 2001-04-15 | Dalarna |
| GC743 | 2001-04-19 | Halland |
| GC7FF | 2001-04-28 | Västra Götaland |
| GC8A6 | 2001-05-06 | Stockholm |
| GC8A7 | 2001-05-06 | Stockholm |
| GC8D4 | 2001-05-07 | Stockholm |

When running GetOldest with the above locations as filter and a limit of 3, the following three are returned:
GC4C, GC4D, GC16B.

I also added a minHiddenDate filter. It will default to 2000-05-03. I will make another post about that.

Thanks for highlighting the min date filter.

Just to make sure we are on the same page, your example is great. There also is the use case country=["united states","canada","mexico"] that I would like to work.

Would I be asking too much for country=[united states], region=massachusetts, county=[middlesex ma, Essex ma]?

The last one I know is scope creep but it rounds out the set.

["united states","canada","mexico"] works fine, might require proper casing. But I assume you were just lazy, just as my example wasn't the correct syntax.

country=[united states], region=massachusetts, county=[middlesex ma, Essex ma] works great as well, assuming the counties both exists in massachusetts, which exists in usa.

You can mix string and array locations as you wish. For example country = [USA, Mexico], region = 'Texas'. This works technically, though it would be the same as leaving out Mexico from the start.

country=[USA, Mexico], county = [A-county-usa, A-county-in-mexico] would return oldest for those two matching counties.

For performance reasons and duplicate name reasons it's recommended to always include the country names. For example. If you wish to GetOldest() for Texas, you want to use the country = USA. That will make it faster (database indexes will work properly). It will also eliminate the risk that Germany has a region called Texas as well. Incorrect example since Germany doesn't have such region. But there are many real cases, I just didn't know one in my head.

Awesome. Allowing for capitalizaton and proper quoting and the existence of the counties, you understood.

I will kick the tires this weekend.

The oldest N in {Country,Region,County} checkers now all run faster thanks to this and the 1000 cache fetch limit. They also work fine with the new api calls.

Actually the oldest in N counties did not exist before this.

Thank you very much.