×

To be able to write in the forum you need to authenticate. Meanwhile it's read-only.

Big boring changes incoming (PHP7)

Big boring changes incoming (PHP7)
March 01, 2018 12:07PM
(LUA sandbox changes after some intro story)

We are working on finalizing something we started 1-2 years ago. Upgrading from PHP 5.6 to 7.x.

The first thing we noticed was that we needed to change our database connection implementations. We did that a long time ago, but left a lot of legacy code everywhere. That was fixed last year. 2-3 weeks alone with rewriting code.

Yesterday I upgraded our development environment to PHP 7.0 and things stopped working, as expected. We are using some not too common PHP modules which started causing issues.

We had already moved away from our old Sphinxsearch implementation and (hopefully) use SphinxQL everywhere. This only needs a mysql compatible client library and therefore works as automatically as it can.

The two biggest tasks was the map generation and the LUA sandbox. The PHP module for map generation (static images, not JS) that we use doesn't officially support PHP 7.x yet. Though there is a fork. After many hours we have finally made it work, the only downside is that we can't get GIF-output to work for some reason. We used to store our maps as PNG but noticed they became smaller at no visual loss with GIF. Now we will have to go back to PNG again.

The LUA sandbox was worse. The implementation we have used to connect PHP with LUA has caused us issues before. We have been using a several year old version because newer versions kept segfaulting. Now with PHP 7.x we were forced to upgrade. We still got segfaults with the newer versions. After spending quite a lot of time trying to figure out why, we started looking for other solutions. The fact is, our current LUA sandbox also crashes sometimes, we haven't figured out if it's our fault or something else. But maybe there are some memory leaks in it. Maybe it's good to leave that implementation behind.

Instead we will now use Mediawiki's implementation if a LUA sandbox. There wasn't much documentation on the old implementation we used, but the new one has even less. It's up and running in the development environment though, and it seems stable.

The problem is that it will be a complete new implementation from our side. This will affect you as well. As it's right now, there is no code taking care of errors and trying to show those to the users. That's what our next step. The up-side is that we can clean up the current code which is a mess.

With the new Sandbox we don't control which LUA functions exists. It's what Mediawiki has considered to be safe. Anything else will need to be implemented in PHP as callbacks. This far we have only tested 2 of over 2000 scripts and therefore don't know yet how big of a problem this is.

All current functions that begins with PGC_ will in the future be PGC. instead. (PGC_GetFinds will become PGC.GetFinds). This is currently solved with a search and replace in the LUA source code, which isn't the best solution.

Also all built in LUA functions will change names. With our old Sandbox we couldn't expose them with the same name, therefore tonumber was called ToNumber for example. Or string.byte was called StringByte. With the new Sandbox they will have their correct names. Existing checkers are search-and-replaced for this as well. Most unreliable search-and-replace is that 'Type will become 'type'. Other terms we search for probably won't match anything unintentional.


So why PHP 7.x then? Well, it's supposed to be more than twice as fast. It's supposed to consume less memory, though my single test I have made increased memory usage with 20%. We of course don't want to be stuck with old dependencies either. Long-term it's wise to try to catch up with recent releases.

Follow this topic if you want to make sure you don't miss anything. You can follow it by replying, or clicking the Follow topic button. When we are done with what we can thing of, we will probably open up the development environment and use it as a staging environment for a few days, to get some feedback, especially around LUA development.



Edited 1 time(s). Last edit at 03/01/2018 12:15PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 01, 2018 12:18PM
Sounds like the results will be good but a lot of work to get there.

Presumably for the search and replace of Lua scripts some automated testing could be done by executing the 'validate last 10 logs' for each tag with the old and new data and see whether the results stay consistent.
Re: Big boring changes incoming (PHP7)
March 01, 2018 12:22PM
I will try my best to find most of the issues before anyone else sees them. Right now I am trying to catch some errors. Created a LUA script which doesn't work (on purpose). This far I have not been able to catch the "compilation" errors.
Re: Big boring changes incoming (PHP7)
March 01, 2018 04:13PM
For some unexplained reason one of your scripts failed Mole125.
The script behind https://project-gc.com/Challenges//179

It fails in fixed_length_right because you are treating the variable string as a string, though it sometimes is an integer. Easy to fix by adding a
string = tostring(string)

I can't explain why it works on the Live servers but not with the new LUA Sandbox though. I guess it's minor variances in LUA itself. Both are 5.1, but not sure about minor versions.

I will exclude that script from further testing.
Re: Big boring changes incoming (PHP7)
March 01, 2018 04:43PM
I've updated the script with the old syntax ToString so hopefully it should work in the new environment.

My guess is it is something to do with the switch of using StringLen to stringlen, did your wrapper do some implicit conversion of number parameters to strings?

Have you thought about a standard 'preamble' script that gets prepended to all the lua scripts before they are executed, these could define the existing 'custom' lua functions (StringLen, PGC_GetFinds etc) inline and provide the mapping to the new versions (with extra manipulation if necessary). Eg
function PGC_GetFinds(x,y,z)
PGC.GetFinds(x,y,z)
end
function StringLen(str)
stringlen(tostring(str))
end
etc.

That said, if you are finding the finds and replace is working sufficiently then long term just doing the conversion is going to be most performant.
Re: Big boring changes incoming (PHP7)
March 01, 2018 04:53PM
Your script works now.

There was nothing made by me on the way before. It was just an associative array of function names between the sandbox and the LUA engine.

Good idea about pre-prending a script. I'll see if I need it. The advantage of search and replace is that I can apply it for real to the scripts (update them) and then get rid of that code. But that depends on how well my regex works out. My first and second attempt failed for various reasons. This is the current regex:
$source = preg_replace("/(^|\W)({$old})(\W)/", "$1{$new}$3", $source);
\W is a PHP shorthand for "not a word character"

I am actually already pre-pending scripts with the new implementation. I needed to get parts of Penlight in there. It's a bit of a mess. It has a lot of "require" which isn't available in the Sandbox. I have to cut and paste from several files and build them together. Also need to declare a few things as globals. Long-term it's probably wise to drop Penlight and replace it with functions in PHP. I believe it's mostly the class Date that's used.
Re: Big boring changes incoming (PHP7)
March 02, 2018 01:59PM
https://project-gc.com/Challenges//20319 seems to have borrowed the fixed_length_right function, needs the same fix. Author: famv17
Re: Big boring changes incoming (PHP7)
March 01, 2018 05:11PM
Testing in progress.
Re: Big boring changes incoming (PHP7)
March 01, 2018 07:46PM
In this script made by Arisoft there is some issue as well.

This line works on live servers, but not on the development environment.
local cache = list[word.codes[word.select]]

I ran it on my name, and word.select becomes 1 and word.codes[word.select] becomes GC1T3TV. However, that gccode does not exist in the variable list. I have not yet looked into how list is built.



Edited 1 time(s). Last edit at 03/01/2018 09:09PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 09:13AM
https://project-gc.com/Challenges//9533 seems to have the exact same issue.
Re: Big boring changes incoming (PHP7)
March 01, 2018 08:48PM
This script needs the following patch:
< if TGrid[Letter][CacheType] ~= nil and TResultsTypes[CacheType] == nil then
> if TGrid[Letter] ~= nil and TGrid[Letter][CacheType] ~= nil and TResultsTypes[CacheType] == nil then



Edited 1 time(s). Last edit at 03/01/2018 09:09PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 01, 2018 10:29PM
https://project-gc.com/Challenges//7295 has a new issue. It prints a table, which doesn't work anymore. Not sure how to solve that right now.
Re: Big boring changes incoming (PHP7)
March 02, 2018 09:26AM
I'll gather all script links that uses the variable name table here, and delete my old posts about it. The issue is that they are destroying the table "class" by doing so.
https://project-gc.com/Challenges//10031
https://project-gc.com/Challenges//9830
https://project-gc.com/Challenges//6151
https://project-gc.com/Challenges//5985
https://project-gc.com/Challenges//2079
https://project-gc.com/Challenges//32368
https://project-gc.com/Challenges//10800
https://project-gc.com/Challenges//12285
https://project-gc.com/Challenges//12987
https://project-gc.com/Challenges//13527
https://project-gc.com/Challenges//13839
https://project-gc.com/Challenges//14494
https://project-gc.com/Challenges//15162
https://project-gc.com/Challenges//15561
https://project-gc.com/Challenges//15958
https://project-gc.com/Challenges//16268
https://project-gc.com/Challenges//20367
https://project-gc.com/Challenges//20877
https://project-gc.com/Challenges//21440
https://project-gc.com/Challenges//23224
https://project-gc.com/Challenges//23316
https://project-gc.com/Challenges//23541
https://project-gc.com/Challenges//23672
https://project-gc.com/Challenges//23684
https://project-gc.com/Challenges//24296
https://project-gc.com/Challenges//24856
https://project-gc.com/Challenges//25130
https://project-gc.com/Challenges//26860
https://project-gc.com/Challenges//27817
https://project-gc.com/Challenges//28296
https://project-gc.com/Challenges//28663
https://project-gc.com/Challenges//29146
https://project-gc.com/Challenges//29749
https://project-gc.com/Challenges//30397
https://project-gc.com/Challenges//30511
https://project-gc.com/Challenges//30568
https://project-gc.com/Challenges//30693
https://project-gc.com/Challenges//30849
https://project-gc.com/Challenges//30857
https://project-gc.com/Challenges//31203
https://project-gc.com/Challenges//32366

Same issue, but using the variable pairs and not table:
https://project-gc.com/Challenges//24247



Edited 37 time(s). Last edit at 03/02/2018 06:30PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 10:22AM
https://project-gc.com/Challenges//11986
[string ""]:115: attempt to index field '?' (a nil value)

I didn't spend the time to understand how the script works and how to fix it.
Re: Big boring changes incoming (PHP7)
March 02, 2018 11:42AM
https://project-gc.com/Challenges//14305 by jpavlik doesn't work.
[string ""]:26: attempt to compare nil with number

Most likely possible to solve. From what I can see it's not the regex upgrade of the script that breaks it.

UPDATE:
Seems like vogelbird has a fork of the script with the same issue.
https://project-gc.com/Challenges//18169



Edited 1 time(s). Last edit at 03/02/2018 01:24PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 02:41PM
Typecast issue
https://project-gc.com/Challenges//21705
[string ""]:207: attempt to perform arithmetic on field 'radius' (a string value)
local latadd=point.radius/1000*latconv
local lonadd=point.radius/1000*lonconv
Re: Big boring changes incoming (PHP7)
March 02, 2018 03:35PM
https://project-gc.com/Challenges//24234
Cannot pass circular reference to PHP
Need to remove Print(years)
Re: Big boring changes incoming (PHP7)
March 02, 2018 03:50PM
https://project-gc.com/Challenges//25004
[string ""]:284: bad argument #2 to 'min' (number expected, got nil)
map.outboundBox[2]=math.min(map.outboundBox[2],list[2])
Re: Big boring changes incoming (PHP7)
March 02, 2018 04:06PM
https://project-gc.com/Challenges//25151 uses a json file (US_State_Capitals) that works in PHP 5.6 but not in 7.0. Most likely invalid in some way.

https://project-gc.com/Challenges//27426 uses the same json.



Edited 1 time(s). Last edit at 03/02/2018 04:55PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 12:09PM
https://project-gc.com/Challenges//14410

This one is scary!

[string ""]:26: attempt to index field '?' (a nil value)
hides[1] can be nil at line 26

If I replace it with hides[0] the script works, and gives the same result as live. From what I remember, LUA doesn't base indexes on 0? But obviousle table.sort in this LUA version suddenly does?

A solution is of course something like if hides[0] ~= nil then .. else .. end

Any thoughts on how many scripts this might affect?



Edited 1 time(s). Last edit at 03/02/2018 12:23PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 03:54PM
https://project-gc.com/Challenges//25085
Cannot pass circular reference to PHP

Need to remove
PGC.print(extreme)
Re: Big boring changes incoming (PHP7)
March 02, 2018 04:32PM
Another 0-based index issue
https://project-gc.com/Challenges//26424
[string ""]:1423: attempt to index global 'lastFind' (a nil value)

The issue comes from this function:
function getLastFind (profileId)
	lastFinds =  PGC.GetFinds( profileId, { fields = { 'gccode', 'visitdate', 'cache_name'}, 
				order = 'NEWESTFIRST', 
				limit = 1} )
	return lastFinds[1]
end


https://project-gc.com/Challenges//26933 seems to have the same code.



Edited 1 time(s). Last edit at 03/02/2018 04:45PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 04:38PM
https://project-gc.com/Challenges//26726
[string ""]:296: bad argument #2 to 'min' (number expected, got nil)

map.outboundBox[1]=math.min(map.outboundBox[1],list[1])
map.outboundBox[3]=math.max(map.outboundBox[3],list[1])
map.outboundBox[2]=math.min(map.outboundBox[2],list[2]) <----- 296
map.outboundBox[4]=math.max(map.outboundBox[4],list[2])

I assume this is also a 0-based index issue. In that case, it should be 0, 0, 1, 1
Re: Big boring changes incoming (PHP7)
March 02, 2018 07:26PM
I'll summarize all scripts that needs to be fixed here:

https://project-gc.com/Challenges//20319 scriptId: 1307, creator: famv17
Typecasting issue in fixed_length_right
Fix: string = tostring(string)


https://project-gc.com/Challenges//984 scriptId: 2486, creator: arisoft
https://project-gc.com/Challenges//9533 scriptId: 826, creator: vogelbird
local cache = list[word.codes[word.select]]
word.select becomes 1 and word.codes[word.select] becomes GC1T3TV. However, that gccode does not exist in the variable list.
Fixed with server side patch.


https://project-gc.com/Challenges//2580 scriptId: 528, creator: the Seagnoid
< if TGrid[Letter][CacheType] ~= nil and TResultsTypes[CacheType] == nil then
> if TGrid[Letter] ~= nil and TGrid[Letter][CacheType] ~= nil and TResultsTypes[CacheType] == nil then


https://project-gc.com/Challenges//7295 scriptId: 767, creator: hampf
https://project-gc.com/Challenges//24234 scriptId: 1550, creator: jpavlik
https://project-gc.com/Challenges//25085 scriptId: 1679, creator: sloth96
Prints a table, need to remove that print for now


https://project-gc.com/Challenges//11986 scriptId: 919, creator: Target.
[string ""]:115: attempt to index field '?' (a nil value)
Fixed with server side patch.


https://project-gc.com/Challenges//14305 scriptId: 971, creator: jpavlik
https://project-gc.com/Challenges//18169 scriptId: 1226, creator: vogelbird (fork of the above)
[string ""]:26: attempt to compare nil with number
Fixed with server side patch.


https://project-gc.com/Challenges//21705 scriptId: 1287, creator: Target.
Typecast issue
[string ""]:207: attempt to perform arithmetic on field 'radius' (a string value)
local latadd=point.radius/1000*latconv
local lonadd=point.radius/1000*lonconv
Fixed with server side patch. (wasn't a typecast issue)


https://project-gc.com/Challenges//25004 scriptId: 1623, creator: sloth96
[string ""]:284: bad argument #2 to 'min' (number expected, got nil)
map.outboundBox[2]=math.min(map.outboundBox[2],list[2])
Fixed with server side patch.


https://project-gc.com/Challenges//25151 scriptId: 1690, creator: sloth96
https://project-gc.com/Challenges//27426 scriptId: 1690, creator: sloth96
Uses a json file (US_State_Capitals) that works in PHP 5.6 but not in 7.0. Most likely invalid in some way.
There was a tab character in a key in the JSON (FIXED).
The scripts output is faulty on my name though, because of 0-based indexes and this code:
local north = poly.rect[1]
local south = poly.rect[2]
local east = poly.rect[3]
local west = poly.rect[4]
Fixed with server side patch.



https://project-gc.com/Challenges//14410 scriptId: 1061, creator: SeekerSupreme
0-based index issue?
[string ""]:26: attempt to index field '?' (a nil value)
hides[1] can be nil at line 26
FIX: if hides[0] ~= nil then .. else .. end
Fixed with server side patch.


https://project-gc.com/Challenges//26424 scriptId: 1914, creator: AnnaMoritz
https://project-gc.com/Challenges//26933 scriptId: 1996, creator: AnnaMoritz
0-based index issue?
[string ""]:1423: attempt to index global 'lastFind' (a nil value)
Fixed with server side patch.


https://project-gc.com/Challenges//26726 scriptId: 1609, creator: sloth96
0-based index issue?
map.outboundBox[1]=math.min(map.outboundBox[1],list[1])
map.outboundBox[3]=math.max(map.outboundBox[3],list[1])
map.outboundBox[2]=math.min(map.outboundBox[2],list[2]) <----- 296
map.outboundBox[4]=math.max(map.outboundBox[4],list[2])
I assume this is also a 0-based index issue. In that case, it should be 0, 0, 1, 1
Fixed with server side patch.


Scripts destroying the class table (scriptId:tagId):
603:2079, 735:5985, 743:6151, 837:9830, 845:10031, 857:10800, 940:12285, 937:12987, 1007:13527, 1021:13839, 1065:14494, 1098:15162, 1109:15561, 1131:15958, 1165:16268, 1315:20367, 1363:20877, 1397:21440, 1503:23224, 1508:23316, 1518:23541, 1524:23672, 1525:23684, 1553:24296, 1466:24856, 1685:25130, 1989:26860, 1538:27817, 2245:28296, 2292:28663, 2385:29146, 2467:29749, 2513:30397, 2514:30511, 2535:30568, 2558:30693, 2581:30849, 2582:30857, 2624:31203, 2704:32366, 2602:32368


Scripts destroying the class pairs:
https://project-gc.com/Challenges//24247 scriptId: 1552, creator: jpavlik


When I am more ready, there will be a way for certain users to edit ANY script to help us through this. I'll get back with more details later, might not be today. Quite tired now, been working double time for a few days.
Scripts can now be edited via GitLab.com.

All 0-based index issues was a server side issue, it was fixed (thanks arisoft for making me realize). That also fixed most of the broken scripts actually.



Edited 10 time(s). Last edit at 03/08/2018 09:27PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 02, 2018 09:37PM
I tried to fix "table" variables from 32368 but I can not save the edited script. Error saving...
Re: Big boring changes incoming (PHP7)
March 02, 2018 09:39PM
That's on purpose. There should be an "alert" added to the web that more information is coming soon. I am currently writing that post. Then there will probably be a few hours until scripts can be edited at all.

Currently pushing all scripts to GitLab.com.
Re: Big boring changes incoming (PHP7)
March 02, 2018 09:58PM
Updated information, if you haven't seen it already: https://project-gc.com/forum/read?6,17332
Re: Big boring changes incoming (PHP7)
March 08, 2018 08:59PM
Quote

https://project-gc.com/Challenges//984 scriptId: 2486, creator: arisoft
https://project-gc.com/Challenges//9533 scriptId: 826, creator: vogelbird
local cache = list[word.codes[word.select]]
word.select becomes 1 and word.codes[word.select] becomes GC1T3TV. However, that gccode does not exist in the variable list.

I checked this and there may be someting wrong with PGC_GetFinds

I added debug output to show what caches are requested and what caches are received.

When I tried this with my own id the missing item was cache GCVXQE. Live server returns 115 caches as requested but Dev server returns 114 caches for some reason. The debug output contains "Requested codes:" and "Received codes:" which should be the same but they are not. From the debug output I can not find any rational reason why PGC_GetFinds leaves one cache out. Does it return array with [0] index?



Edited 1 time(s). Last edit at 03/08/2018 09:01PM by arisoft. (view changes)
Re: Big boring changes incoming (PHP7)
March 08, 2018 09:09PM
I think you are onto something. I thought I had checked that GetFinds() actually returned the same amount of caches on both Live and Dev. Either I failed my test, or something else screwed me over.

I just did a patch on Dev that makes this checker work. Ie, I insert a NULL item first in the list, which will then get index 0. That index 0 item seems to disappear.

Now I wonder if this is a good solution that I should apply to all the callbacks, or there is some drawback...
Re: Big boring changes incoming (PHP7)
March 08, 2018 09:14PM
I believe I have patched every method now to not return anything with index 0. Just hoping I didn't fix one thing and break something else.
Re: Big boring changes incoming (PHP7)
March 08, 2018 09:31PM
With those server side changes most scripts with issues passed the tests now. I went through my big post and added strikethrough to those that now works.

If I didn't miss any script, there are now five scripts with issues. All with a known fix noted close to them. If no one else fixes them before Monday, I will probably fix them then.

To the above, I did not count your script that's slow arisoft (https://dev-01.project-gc.com/Challenges//30511). Since it works, I didn't see it as priority 1. It's still slow though.
Re: Big boring changes incoming (PHP7)
March 09, 2018 04:48PM
ganja1447 Wrote:
-------------------------------------------------------
> To the above, I did not count your script that's
> slow arisoft
> (https://dev-01.project-gc.com/Challenges//30511).
> Since it works, I didn't see it as priority 1.
> It's still slow though.

Now I have studied this speed problem. My original guess was right. It is a server side problem.
I made a proof of concept checker https://project-gc.com/Challenges//32341

Live server gives this kind of (debug) results:
Vincenty: 0.02340241 ms 111420.728 m
Haversine: 0.01744844 ms 111195.08 m
Pythagoras: 0.00234117 ms 111420.69881874 m

Dev server gives this kind of results:
Vincenty: 0.155944 ms 111420.728 m
Haversine: 0.127434 ms 111195.08 m
Pythagoras: 0.0019912 ms 111420.69881874 m

Two notes:

- Both Vincenty and Haversine are about 7 times slower in Dev but my own algorithm, which is used in Cache chains, is slightly faster! That explains why Cache chains is not affected by this problem.

- My Pythagoras algorithm seems to be faster and more precise than Haversine ;) but it is usable only up to 100km range.



Edited 1 time(s). Last edit at 03/09/2018 04:49PM by arisoft. (view changes)
Re: Big boring changes incoming (PHP7)
March 09, 2018 04:55PM
The PHP library used for those algorithms has a much newer version that requires PHP7. I assume it's a good sign it requires PHP7. Since it does, it's patched with that in mind, and hopefully at least as fast as well.

I'll see of I can upgrade it straight away, then we will run your checker in DEV again to see the results.
Re: Big boring changes incoming (PHP7)
March 09, 2018 05:02PM
For a curiosity I also added OsClock() to the test and it is also about 5 times slower in Dev,
Re: Big boring changes incoming (PHP7)
March 09, 2018 05:17PM
The DEV server now has a more recent version of that library. I ran your test checker to see the results. Still quite slow.
Vincenty: 0.1489622 ms 111420.728 m
Haversine: 0.1146362 ms 111195.08 m
Pythagoras: 0.0019958 ms 111420.69881874 m
os.clock: 0.0015268 ms

I had to enable your script and tag meanwhile,hope I didn't interrupt any editing.


Since that didn't help I was curious if it was the phpgeo library or not. I then made a small PHP script that calculates Haversine 100,000 times in a loop. 12.141s on DEV and 11.315s on Live. I can't say if the small difference is due to hardware or software, but I don't think we care about that small difference. I also only ran it once.

So, it doesn't seem to be the library. I am thinking maybe it's the cost of using the callbacks added to the Sandbox. Do you think you can implement a test that asks for a list of distances? So instead of calling DistanceVincenty()/DistanceHaversine() X times in a loop, we build a list of data and call a test function like DistanceVincentyList()/DistanceHaversineList()?
The list should then be something like
{ {lat1 = 10, lon1 = 10, lat2 = 20, lon2 = 20}, {lat1 = 30, lon1 = 30, lat2 = 31, lon2 = 31} ... }

I prefer to not implement that on Live, but at least we can see if it becomes much faster on DEV.

EDIT: Such callback exists now (not tested). It will return the results. As it is now, it just returns an array with the distances in the same order as it got the coordinate pairs. Probably not a useful design, but good enough for testing.



Edited 3 time(s). Last edit at 03/09/2018 05:26PM by ganja1447. (view changes)
Re: Big boring changes incoming (PHP7)
March 09, 2018 05:40PM
I added test for / function DistanceVincentyList(list)

You can now replace the script version with PGC version to compare results.
Re: Big boring changes incoming (PHP7)
March 09, 2018 05:44PM
Seems like you got the same conclusion as I did in https://project-gc.com/forum/read?6,17264,17531#msg-17531

Requesting a list doesn't help.

So, in pure PHP, there is no speed difference between the PHP5.6 version and PHP7 version of Haversine().
There is a huge difference when requesting many times.
There is a huge difference when requesting a list of coordinates.

Seems quite weird right?
Re: Big boring changes incoming (PHP7)
March 09, 2018 05:40PM
Wrote a test myself.

-- 11 seconds
for var=1,100000 do
PGC.DistanceHaversine(10, 12, 11, 13)
end

--13 seconds
list = {}
for var=1,100000 do
TableInsert(list, {lat1 = 10, lon1 = 12, lat2 = 11, lon2 = 13})
end
PGC.DistanceHaversineList(list)


I am currently out of ideas.
Re: Big boring changes incoming (PHP7)
March 09, 2018 05:49PM
I picked the OsClock() for testing because it may be used many times. It is some kind of library function in LUA. Don't know anything about the implementation but for some reason it is also many times slower without reason.

What do we know about different type of library functions? Are they all slower? At least Math library does not seem to be slower.
Re: Big boring changes incoming (PHP7)
March 09, 2018 06:12PM
Penlight library is only sligthly slightly slower. It may be implemented in LUA.
Live Date: 0.00953597 ms
Dev Date: 0.0123386 ms

Quote

Since that didn't help I was curious if it was the phpgeo library or not. I then made a small PHP script that calculates Haversine 100,000 times in a loop. 12.141s on DEV and 11.315s on Live. I can't say if the small difference is due to hardware or software, but I don't think we care about that small difference. I also only ran it once.

There is something special. 100,000 Haversines should take only 1.745 s in Live - not 11,315 as you get - if you are using compatible method.
Re: Big boring changes incoming (PHP7)
March 09, 2018 06:41PM
I found something super interesting. Not sure I will finish this today, seems like a mess.

I currently have 4 PHP-servers running for PGC. All of them are virtual machines.
Dev: 10s (PHP7)
Live: 6.76s (LUA isn't executed here in reality)
Dedicated LUA server 1: 17.5s (for auto-checkers in the background, likely to be loaded) 7.1s after removing all other running LUA scripts
Dedicated LUA server 2: 1.7s (this is where live runs its lua scripts)

Virtualization server : instance
2 : dev
3 : Live
1: lua-01
3: lua-02

Virtualization server 3 has the best hardware, 1 and 2 are identical in CPU.

What I am considering right now is to move DEV to virtualization server 3 to see if it becomes faster. But in that case I would expect Live to already be faster.

Found the issue! Now I have a billion tabs open and lost track of what I am doing. But I found a tag that seems fast on DEV now.

There were debug-processing installed on more servers than there should be. I even had it on a TODO-list related to this whole PHP7 thing to remove that! If I only had gotten that far...

Please confirm that it's faster now. :)
Re: Big boring changes incoming (PHP7)
March 09, 2018 06:51PM
Live:
Date: 0.00851655 ms
VincentyList: 0.02342321 ms 111420.728 m
Vincenty: 0.02293138 ms 111420.728 m
Haversine: 0.01790795 ms 111195.08 m
Pythagoras: 0.00240524 ms 111420.69881874 m
OsClock: 0.00030678000000002 ms

Dev:
Date: 0.0125416 ms = slower
VincentyList: 0.0097734 ms 111420.728 m = faster
Vincenty: 0.0099198 ms 111420.728 m = faster
Haversine: 0.0069936 ms 111195.08 m = faster
Pythagoras: 0.0018786 ms 111420.69881874 m = faster
os.clock: 0.0015934 ms = slower

Only os.clock is slower but not too slow.
Re: Big boring changes incoming (PHP7)
March 09, 2018 07:24PM
Seems like a success.

FYI, there is also a PGC.GetCPUUsage() in Dev. Not available on Live servers, it comes with the new Sandbox.

It should return number of seconds the Sandbox believes it has been running. When that reaches 60, the Sandbox will timeout.
Re: Big boring changes incoming (PHP7)
March 09, 2018 06:45PM
I will remove the DistanceVincentyList() and DistanceHaversineList() callbacks.

I'll leave them for now, but I won't commit them tomorrow/on Monday when I clean up my repository.
Re: Big boring changes incoming (PHP7)
March 12, 2018 08:08PM
Hi,
I know there is a ton on the plate, so feel free delaying your response.

I was looking at GitLab today. Looks to me like it's just by Script number file name. I tracked down one of mine and tagged it with my project-gc username.... but i think it only tags the current version, so if i update it in the future, a tag search will return the old one.

Any suggestion on how best to work with this repository? Will you somehow migrate the script writer, tags, etc to GitLab?

-TG
Re: Big boring changes incoming (PHP7)
March 13, 2018 10:53AM
I am not sure how familiar you are with Git and other version control systems. The tags, as you say, are for a specific version. They are also only a name to make some certain versions easier to recognize. Typically used to mark release versions when developing software. v1.0.0, v1.0.1 and so on.

The current idea and plan with GitLab is only to provide an alternative online editor of source code, where certain users also can edit other persons scripts. By having it in Git we get historical versions and diffs automatically. We can also keep track of who did what and much easier revert changes if needed.

If one doesn't want to work with an online editor, one can also clone the repository and edit the file locally, with ones favorite editor, commit the changes and just push it to origin.

Next step could be that we move all scripts into a scripts directory and add another directory to version control tags. I don't know if that's feasible. As it is now, we validate the JSON on save, I am unsure if we can add such hook to GitLab, especially when content gets pushed towards it.

We are also considering adding user contributed modules for Profile stats. If/when that happens I can imagine that going via GitLab will be the only alternative. In other words, we might not keep an online source code editor at Project-GC.

I don't see why the name of the script writer is needed in the repository at all. Or to be more exact, that name will exists in the commits only.

What we could (and should) do, is link the GitLab source-code file directly from the Challenge checker page. As it is now, it's not too easy to find the script number from a script. There could be a link going straight to (for example) https://gitlab.com/magma1447/PGC-ChallengeCheckers/blob/master/1005.lua

I am also looking into two other ideas for the future, where Git will help us.
1) We could work with branches, so that a script editor can edit the script in a development branch, and also execute the script from there. When done, he just merges it into the master branch and all scripts suddenly uses the new version.
2) If one makes breaking changes to a script and notices it afterwards, and it's not fixable. A tag could point to a certain version of a script, instead of always pointing to the latest. I am not sure this is a good idea, or worth it. But it's something that's easier to do with Git than how we store it now.
Sorry, you do not have permission to post/reply in this forum.