Big boring changes incoming (PHP7)

Re: Big boring changes incoming (PHP7)

magma1447 — Tue, 13 Mar 2018 10:53:19 +0000

I am not sure how familiar you are with Git and other version control systems. The tags, as you say, are for a specific version. They are also only a name to make some certain versions easier to recognize. Typically used to mark release versions when developing software. v1.0.0, v1.0.1 and so on.

The current idea and plan with GitLab is only to provide an alternative online editor of source code, where certain users also can edit other persons scripts. By having it in Git we get historical versions and diffs automatically. We can also keep track of who did what and much easier revert changes if needed.

If one doesn't want to work with an online editor, one can also clone the repository and edit the file locally, with ones favorite editor, commit the changes and just push it to origin.

Next step could be that we move all scripts into a scripts directory and add another directory to version control tags. I don't know if that's feasible. As it is now, we validate the JSON on save, I am unsure if we can add such hook to GitLab, especially when content gets pushed towards it.

We are also considering adding user contributed modules for Profile stats. If/when that happens I can imagine that going via GitLab will be the only alternative. In other words, we might not keep an online source code editor at Project-GC.

I don't see why the name of the script writer is needed in the repository at all. Or to be more exact, that name will exists in the commits only.

What we could (and should) do, is link the GitLab source-code file directly from the Challenge checker page. As it is now, it's not too easy to find the script number from a script. There could be a link going straight to (for example) https://gitlab.com/magma1447/PGC-ChallengeCheckers/blob/master/1005.lua

I am also looking into two other ideas for the future, where Git will help us.
1) We could work with branches, so that a script editor can edit the script in a development branch, and also execute the script from there. When done, he just merges it into the master branch and all scripts suddenly uses the new version.
2) If one makes breaking changes to a script and notices it afterwards, and it's not fixable. A tag could point to a certain version of a script, instead of always pointing to the latest. I am not sure this is a good idea, or worth it. But it's something that's easier to do with Git than how we store it now.

Re: Big boring changes incoming (PHP7)

travelingGeek — Mon, 12 Mar 2018 20:08:59 +0000

Hi,
I know there is a ton on the plate, so feel free delaying your response.

I was looking at GitLab today. Looks to me like it's just by Script number file name. I tracked down one of mine and tagged it with my project-gc username.... but i think it only tags the current version, so if i update it in the future, a tag search will return the old one.

Any suggestion on how best to work with this repository? Will you somehow migrate the script writer, tags, etc to GitLab?

-TG

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 19:24:37 +0000

Seems like a success.

FYI, there is also a PGC.GetCPUUsage() in Dev. Not available on Live servers, it comes with the new Sandbox.

It should return number of seconds the Sandbox believes it has been running. When that reaches 60, the Sandbox will timeout.

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 09 Mar 2018 18:51:26 +0000

Live:
Date: 0.00851655 ms
VincentyList: 0.02342321 ms 111420.728 m
Vincenty: 0.02293138 ms 111420.728 m
Haversine: 0.01790795 ms 111195.08 m
Pythagoras: 0.00240524 ms 111420.69881874 m
OsClock: 0.00030678000000002 ms

Dev:
Date: 0.0125416 ms = slower
VincentyList: 0.0097734 ms 111420.728 m = faster
Vincenty: 0.0099198 ms 111420.728 m = faster
Haversine: 0.0069936 ms 111195.08 m = faster
Pythagoras: 0.0018786 ms 111420.69881874 m = faster
os.clock: 0.0015934 ms = slower

Only os.clock is slower but not too slow.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 18:45:17 +0000

I will remove the DistanceVincentyList() and DistanceHaversineList() callbacks.

I'll leave them for now, but I won't commit them tomorrow/on Monday when I clean up my repository.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 18:41:35 +0000

I found something super interesting. ~~Not sure I will finish this today, seems like a mess.~~

I currently have 4 PHP-servers running for PGC. All of them are virtual machines.
Dev: 10s (PHP7)
Live: 6.76s (LUA isn't executed here in reality)
Dedicated LUA server 1: 17.5s (for auto-checkers in the background, likely to be loaded) 7.1s after removing all other running LUA scripts
Dedicated LUA server 2: 1.7s (this is where live runs its lua scripts)

Virtualization server : instance
2 : dev
3 : Live
1: lua-01
3: lua-02

Virtualization server 3 has the best hardware, 1 and 2 are identical in CPU.

~~What I am considering right now is to move DEV to virtualization server 3 to see if it becomes faster. But in that case I would expect Live to already be faster.~~

Found the issue! Now I have a billion tabs open and lost track of what I am doing. But I found a tag that seems fast on DEV now.

There were debug-processing installed on more servers than there should be. I even had it on a TODO-list related to this whole PHP7 thing to remove that! If I only had gotten that far...

Please confirm that it's faster now. :)

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 09 Mar 2018 18:12:51 +0000

Penlight library is only sligthly slightly slower. It may be implemented in LUA.
Live Date: 0.00953597 ms
Dev Date: 0.0123386 ms

Quote

Since that didn't help I was curious if it was the phpgeo library or not. I then made a small PHP script that calculates Haversine 100,000 times in a loop. 12.141s on DEV and 11.315s on Live. I can't say if the small difference is due to hardware or software, but I don't think we care about that small difference. I also only ran it once.

There is something special. 100,000 Haversines should take only 1.745 s in Live - not 11,315 as you get - if you are using compatible method.

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 09 Mar 2018 17:49:15 +0000

I picked the OsClock() for testing because it may be used many times. It is some kind of library function in LUA. Don't know anything about the implementation but for some reason it is also many times slower without reason.

What do we know about different type of library functions? Are they all slower? At least Math library does not seem to be slower.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 17:44:59 +0000

Seems like you got the same conclusion as I did in https://project-gc.com/forum/read?6,17264,17531#msg-17531

Requesting a list doesn't help.

So, in pure PHP, there is no speed difference between the PHP5.6 version and PHP7 version of Haversine().
There is a huge difference when requesting many times.
There is a huge difference when requesting a list of coordinates.

Seems quite weird right?

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 17:40:43 +0000

Wrote a test myself.

-- 11 seconds
for var=1,100000 do
PGC.DistanceHaversine(10, 12, 11, 13)
end

--13 seconds
list = {}
for var=1,100000 do
TableInsert(list, {lat1 = 10, lon1 = 12, lat2 = 11, lon2 = 13})
end
PGC.DistanceHaversineList(list)

I am currently out of ideas.

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 09 Mar 2018 17:40:39 +0000

I added test for / function DistanceVincentyList(list)

You can now replace the script version with PGC version to compare results.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 17:17:41 +0000

The DEV server now has a more recent version of that library. I ran your test checker to see the results. Still quite slow.
Vincenty: 0.1489622 ms 111420.728 m
Haversine: 0.1146362 ms 111195.08 m
Pythagoras: 0.0019958 ms 111420.69881874 m
os.clock: 0.0015268 ms

I had to enable your script and tag meanwhile,hope I didn't interrupt any editing.

Since that didn't help I was curious if it was the phpgeo library or not. I then made a small PHP script that calculates Haversine 100,000 times in a loop. 12.141s on DEV and 11.315s on Live. I can't say if the small difference is due to hardware or software, but I don't think we care about that small difference. I also only ran it once.

So, it doesn't seem to be the library. I am thinking maybe it's the cost of using the callbacks added to the Sandbox. Do you think you can implement a test that asks for a list of distances? So instead of calling DistanceVincenty()/DistanceHaversine() X times in a loop, we build a list of data and call a test function like DistanceVincentyList()/DistanceHaversineList()?
The list should then be something like
{ {lat1 = 10, lon1 = 10, lat2 = 20, lon2 = 20}, {lat1 = 30, lon1 = 30, lat2 = 31, lon2 = 31} ... }

I prefer to not implement that on Live, but at least we can see if it becomes much faster on DEV.

EDIT: Such callback exists now (not tested). It will return the results. As it is now, it just returns an array with the distances in the same order as it got the coordinate pairs. Probably not a useful design, but good enough for testing.

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 09 Mar 2018 17:02:55 +0000

For a curiosity I also added OsClock() to the test and it is also about 5 times slower in Dev,

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 09 Mar 2018 16:55:24 +0000

The PHP library used for those algorithms has a much newer version that requires PHP7. I assume it's a good sign it requires PHP7. Since it does, it's patched with that in mind, and hopefully at least as fast as well.

I'll see of I can upgrade it straight away, then we will run your checker in DEV again to see the results.

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 09 Mar 2018 16:48:44 +0000

ganja1447 Wrote:
-------------------------------------------------------
> To the above, I did not count your script that's
> slow arisoft
> (https://dev-01.project-gc.com/Challenges//30511).
> Since it works, I didn't see it as priority 1.
> It's still slow though.

Now I have studied this speed problem. My original guess was right. It is a server side problem.
I made a proof of concept checker https://project-gc.com/Challenges//32341

Live server gives this kind of (debug) results:
Vincenty: 0.02340241 ms 111420.728 m
Haversine: 0.01744844 ms 111195.08 m
Pythagoras: 0.00234117 ms 111420.69881874 m

Dev server gives this kind of results:
Vincenty: 0.155944 ms 111420.728 m
Haversine: 0.127434 ms 111195.08 m
Pythagoras: 0.0019912 ms 111420.69881874 m

Two notes:

- Both Vincenty and Haversine are about 7 times slower in Dev but my own algorithm, which is used in Cache chains, is slightly faster! That explains why Cache chains is not affected by this problem.

- My Pythagoras algorithm seems to be faster and more precise than Haversine ;) but it is usable only up to 100km range.

Re: Big boring changes incoming (PHP7)

magma1447 — Thu, 08 Mar 2018 21:31:51 +0000

With those server side changes most scripts with issues passed the tests now. I went through my big post and added strikethrough to those that now works.

If I didn't miss any script, there are now five scripts with issues. All with a known fix noted close to them. If no one else fixes them before Monday, I will probably fix them then.

To the above, I did not count your script that's slow arisoft (https://dev-01.project-gc.com/Challenges//30511). Since it works, I didn't see it as priority 1. It's still slow though.

Re: Big boring changes incoming (PHP7)

magma1447 — Thu, 08 Mar 2018 21:14:52 +0000

I believe I have patched every method now to not return anything with index 0. Just hoping I didn't fix one thing and break something else.

Re: Big boring changes incoming (PHP7)

magma1447 — Thu, 08 Mar 2018 21:09:14 +0000

I think you are onto something. I thought I had checked that GetFinds() actually returned the same amount of caches on both Live and Dev. Either I failed my test, or something else screwed me over.

I just did a patch on Dev that makes this checker work. Ie, I insert a NULL item first in the list, which will then get index 0. That index 0 item seems to disappear.

Now I wonder if this is a good solution that I should apply to all the callbacks, or there is some drawback...

Re: Big boring changes incoming (PHP7)

arisoft — Thu, 08 Mar 2018 20:59:21 +0000

Quote

https://project-gc.com/Challenges//984 scriptId: 2486, creator: arisoft
https://project-gc.com/Challenges//9533 scriptId: 826, creator: vogelbird
local cache = list[word.codes[word.select]]
word.select becomes 1 and word.codes[word.select] becomes GC1T3TV. However, that gccode does not exist in the variable list.

I checked this and there may be someting wrong with PGC_GetFinds

I added debug output to show what caches are requested and what caches are received.

When I tried this with my own id the missing item was cache GCVXQE. Live server returns 115 caches as requested but Dev server returns 114 caches for some reason. The debug output contains "Requested codes:" and "Received codes:" which should be the same but they are not. From the debug output I can not find any rational reason why PGC_GetFinds leaves one cache out. Does it return array with [0] index?

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 21:58:07 +0000

Updated information, if you haven't seen it already: https://project-gc.com/forum/read?6,17332

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 21:39:11 +0000

That's on purpose. There should be an "alert" added to the web that more information is coming soon. I am currently writing that post. Then there will probably be a few hours until scripts can be edited at all.

Currently pushing all scripts to GitLab.com.

Re: Big boring changes incoming (PHP7)

arisoft — Fri, 02 Mar 2018 21:37:31 +0000

I tried to fix "table" variables from 32368 but I can not save the edited script. Error saving...

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 19:26:23 +0000

I'll summarize all scripts that needs to be fixed here:

https://project-gc.com/Challenges//20319 scriptId: 1307, creator: famv17
Typecasting issue in fixed_length_right
Fix: string = tostring(string)

~~https://project-gc.com/Challenges//984 scriptId: 2486, creator: arisoft~~
~~https://project-gc.com/Challenges//9533 scriptId: 826, creator: vogelbird~~
local cache = list[word.codes[word.select]]
word.select becomes 1 and word.codes[word.select] becomes GC1T3TV. However, that gccode does not exist in the variable list.
Fixed with server side patch.

https://project-gc.com/Challenges//2580 scriptId: 528, creator: the Seagnoid
< if TGrid[Letter][CacheType] ~= nil and TResultsTypes[CacheType] == nil then
> if TGrid[Letter] ~= nil and TGrid[Letter][CacheType] ~= nil and TResultsTypes[CacheType] == nil then

https://project-gc.com/Challenges//7295 scriptId: 767, creator: hampf
https://project-gc.com/Challenges//24234 scriptId: 1550, creator: jpavlik
https://project-gc.com/Challenges//25085 scriptId: 1679, creator: sloth96
Prints a table, need to remove that print for now

~~https://project-gc.com/Challenges//11986 scriptId: 919, creator: Target.~~
[string ""]:115: attempt to index field '?' (a nil value)
Fixed with server side patch.

~~https://project-gc.com/Challenges//14305 scriptId: 971, creator: jpavlik~~
~~https://project-gc.com/Challenges//18169 scriptId: 1226, creator: vogelbird (fork of the above)~~
[string ""]:26: attempt to compare nil with number
Fixed with server side patch.

~~https://project-gc.com/Challenges//21705 scriptId: 1287, creator: Target.~~
Typecast issue
[string ""]:207: attempt to perform arithmetic on field 'radius' (a string value)
local latadd=point.radius/1000*latconv
local lonadd=point.radius/1000*lonconv
Fixed with server side patch. (wasn't a typecast issue)

~~https://project-gc.com/Challenges//25004 scriptId: 1623, creator: sloth96~~
[string ""]:284: bad argument #2 to 'min' (number expected, got nil)
map.outboundBox[2]=math.min(map.outboundBox[2],list[2])
Fixed with server side patch.

~~https://project-gc.com/Challenges//25151 scriptId: 1690, creator: sloth96~~
~~https://project-gc.com/Challenges//27426 scriptId: 1690, creator: sloth96~~
~~Uses a json file (US_State_Capitals) that works in PHP 5.6 but not in 7.0. Most likely invalid in some way.~~
There was a tab character in a key in the JSON (FIXED).
The scripts output is faulty on my name though, because of 0-based indexes and this code:
local north = poly.rect[1]
local south = poly.rect[2]
local east = poly.rect[3]
local west = poly.rect[4]
Fixed with server side patch.

~~https://project-gc.com/Challenges//14410 scriptId: 1061, creator: SeekerSupreme~~
0-based index issue?
[string ""]:26: attempt to index field '?' (a nil value)
hides[1] can be nil at line 26
FIX: if hides[0] ~= nil then .. else .. end
Fixed with server side patch.

~~https://project-gc.com/Challenges//26424 scriptId: 1914, creator: AnnaMoritz~~
~~https://project-gc.com/Challenges//26933 scriptId: 1996, creator: AnnaMoritz~~
0-based index issue?
[string ""]:1423: attempt to index global 'lastFind' (a nil value)
Fixed with server side patch.

~~https://project-gc.com/Challenges//26726 scriptId: 1609, creator: sloth96~~
0-based index issue?
map.outboundBox[1]=math.min(map.outboundBox[1],list[1])
map.outboundBox[3]=math.max(map.outboundBox[3],list[1])
map.outboundBox[2]=math.min(map.outboundBox[2],list[2]) <----- 296
map.outboundBox[4]=math.max(map.outboundBox[4],list[2])
I assume this is also a 0-based index issue. In that case, it should be 0, 0, 1, 1
Fixed with server side patch.

Scripts destroying the class table (scriptId:tagId):
603:2079, 735:5985, 743:6151, 837:9830, 845:10031, 857:10800, 940:12285, 937:12987, 1007:13527, 1021:13839, 1065:14494, 1098:15162, 1109:15561, 1131:15958, 1165:16268, 1315:20367, 1363:20877, 1397:21440, 1503:23224, 1508:23316, 1518:23541, 1524:23672, 1525:23684, 1553:24296, 1466:24856, 1685:25130, 1989:26860, 1538:27817, 2245:28296, 2292:28663, 2385:29146, 2467:29749, 2513:30397, 2514:30511, 2535:30568, 2558:30693, 2581:30849, 2582:30857, 2624:31203, 2704:32366, 2602:32368

Scripts destroying the class pairs:
~~https://project-gc.com/Challenges//24247 scriptId: 1552, creator: jpavlik~~

When I am more ready, there will be a way for certain users to edit ANY script to help us through this. I'll get back with more details later, might not be today. Quite tired now, been working double time for a few days.
Scripts can now be edited via GitLab.com.

All 0-based index issues was a server side issue, it was fixed (thanks arisoft for making me realize). That also fixed most of the broken scripts actually.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 16:38:58 +0000

https://project-gc.com/Challenges//26726
[string ""]:296: bad argument #2 to 'min' (number expected, got nil)

map.outboundBox[1]=math.min(map.outboundBox[1],list[1])
map.outboundBox[3]=math.max(map.outboundBox[3],list[1])
map.outboundBox[2]=math.min(map.outboundBox[2],list[2]) <----- 296
map.outboundBox[4]=math.max(map.outboundBox[4],list[2])

I assume this is also a 0-based index issue. In that case, it should be 0, 0, 1, 1

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 16:32:04 +0000

Another 0-based index issue
https://project-gc.com/Challenges//26424
[string ""]:1423: attempt to index global 'lastFind' (a nil value)

The issue comes from this function:

function getLastFind (profileId)
	lastFinds =  PGC.GetFinds( profileId, { fields = { 'gccode', 'visitdate', 'cache_name'}, 
				order = 'NEWESTFIRST', 
				limit = 1} )
	return lastFinds[1]
end

https://project-gc.com/Challenges//26933 seems to have the same code.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 16:06:51 +0000

https://project-gc.com/Challenges//25151 uses a json file (US_State_Capitals) that works in PHP 5.6 but not in 7.0. Most likely invalid in some way.

https://project-gc.com/Challenges//27426 uses the same json.

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 15:54:41 +0000

https://project-gc.com/Challenges//25085
Cannot pass circular reference to PHP

Need to remove
PGC.print(extreme)

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 15:50:50 +0000

https://project-gc.com/Challenges//25004
[string ""]:284: bad argument #2 to 'min' (number expected, got nil)
map.outboundBox[2]=math.min(map.outboundBox[2],list[2])

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 15:35:16 +0000

https://project-gc.com/Challenges//24234
Cannot pass circular reference to PHP
Need to remove Print(years)

Re: Big boring changes incoming (PHP7)

magma1447 — Fri, 02 Mar 2018 14:41:32 +0000

Typecast issue
https://project-gc.com/Challenges//21705
[string ""]:207: attempt to perform arithmetic on field 'radius' (a string value)
local latadd=point.radius/1000*latconv
local lonadd=point.radius/1000*lonconv