×

To be able to write in the forum you need to authenticate. Meanwhile it's read-only.

Change History

New api methods will be announced here. Changes in our data that is relevant to checkers scripts/tags will also be announced here.

Message: Big boring changes incoming (PHP7)

Changed By: magma1447
Change Date: March 01, 2018 12:15PM

Big boring changes incoming (PHP7)
[i](LUA sandbox changes after some intro story)[/i]

We are working on finalizing something we started 1-2 years ago. Upgrading from PHP 5.6 to 7.x.

The first thing we noticed was that we needed to change our database connection implementations. We did that a long time ago, but left a lot of legacy code everywhere. That was fixed last year. 2-3 weeks alone with rewriting code.

Yesterday I upgraded our development environment to PHP 7.0 and things stopped working, as expected. We are using some not too common PHP modules which started causing issues.

We had already moved away from our old Sphinxsearch implementation and (hopefully) use SphinxQL everywhere. This only needs a mysql compatible client library and therefore works as automatically as it can.

The two biggest tasks was the map generation and the LUA sandbox. The PHP module for map generation (static images, not JS) that we use doesn't officially support PHP 7.x yet. Though there is a fork. After many hours we have finally made it work, the only downside is that we can't get GIF-output to work for some reason. We used to store our maps as PNG but noticed they became smaller at no visual loss with GIF. Now we will have to go back to PNG again.

The LUA sandbox was worse. The implementation we have used to connect PHP with LUA has caused us issues before. We have been using a several year old version because newer versions kept segfaulting. Now with PHP 7.x we were forced to upgrade. We still got segfaults with the newer versions. After spending quite a lot of time trying to figure out why, we started looking for other solutions. The fact is, our current LUA sandbox also crashes sometimes, we haven't figured out if it's our fault or something else. But maybe there are some memory leaks in it. Maybe it's good to leave that implementation behind.

Instead we will now use Mediawiki's implementation if a LUA sandbox. There wasn't much documentation on the old implementation we used, but the new one has even less. It's up and running in the development environment though, and it seems stable.

The problem is that it will be a complete new implementation from our side. This will affect you as well. As it's right now, there is no code taking care of errors and trying to show those to the users. That's what our next step. The up-side is that we can clean up the current code which is a mess.

With the new Sandbox we don't control which LUA functions exists. It's what Mediawiki has considered to be safe. Anything else will need to be implemented in PHP as callbacks. This far we have only tested 2 of over 2000 scripts and therefore don't know yet how big of a problem this is.

All current functions that begins with PGC_ will in the future be PGC. instead. (PGC_GetFinds will become PGC.GetFinds). This is currently solved with a [i]search and replace[/i] in the LUA source code, which isn't the best solution.

Also all built in LUA functions will change names. With our old Sandbox we couldn't expose them with the same name, therefore tonumber was called ToNumber for example. Or string.byte was called StringByte. With the new Sandbox they will have their correct names. Existing checkers are search-and-replaced for this as well. Most unreliable search-and-replace is that 'Type will become 'type'. Other terms we search for probably won't match anything unintentional.


So why PHP 7.x then? Well, it's supposed to be more than twice as fast. It's supposed to consume less memory, though my single test I have made increased memory usage with 20%. We of course don't want to be stuck with old dependencies either. Long-term it's wise to try to catch up with recent releases.

Follow this topic if you want to make sure you don't miss anything. You can follow it by replying for example. , or clicking the [i]Follow topic[/i] button. When we are done with what we can thing of, we will probably open up the development environment and use it as a staging environment for a few days, to get some feedback, especially around LUA development.

Original Message

Author: magma1447
Date: March 01, 2018 12:07PM

Big boring changes incoming (PHP7)
[i](LUA sandbox changes after some intro story)[/i]

We are working on finalizing something we started 1-2 years ago. Upgrading from PHP 5.6 to 7.x.

The first thing we noticed was that we needed to change our database connection implementations. We did that a long time ago, but left a lot of legacy code everywhere. That was fixed last year. 2-3 weeks alone with rewriting code.

Yesterday I upgraded our development environment to PHP 7.0 and things stopped working, as expected. We are using some not too common PHP modules which started causing issues.

We had already moved away from our old Sphinxsearch implementation and (hopefully) use SphinxQL everywhere. This only needs a mysql compatible client library and therefore works as automatically as it can.

The two biggest tasks was the map generation and the LUA sandbox. The PHP module for map generation (static images, not JS) that we use doesn't officially support PHP 7.x yet. Though there is a fork. After many hours we have finally made it work, the only downside is that we can't get GIF-output to work for some reason. We used to store our maps as PNG but noticed they became smaller at no visual loss with GIF. Now we will have to go back to PNG again.

The LUA sandbox was worse. The implementation we have used to connect PHP with LUA has caused us issues before. We have been using a several year old version because newer versions kept segfaulting. Now with PHP 7.x we were forced to upgrade. We still got segfaults with the newer versions. After spending quite a lot of time trying to figure out why, we started looking for other solutions. The fact is, our current LUA sandbox also crashes sometimes, we haven't figured out if it's our fault or something else. But maybe there are some memory leaks in it. Maybe it's good to leave that implementation behind.

Instead we will now use Mediawiki's implementation if a LUA sandbox. There wasn't much documentation on the old implementation we used, but the new one has even less. It's up and running in the development environment though, and it seems stable.

The problem is that it will be a complete new implementation from our side. This will affect you as well. As it's right now, there is no code taking care of errors and trying to show those to the users. That's what our next step. The up-side is that we can clean up the current code which is a mess.

With the new Sandbox we don't control which LUA functions exists. It's what Mediawiki has considered to be safe. Anything else will need to be implemented in PHP as callbacks. This far we have only tested 2 of over 2000 scripts and therefore don't know yet how big of a problem this is.

All current functions that begins with PGC_ will in the future be PGC. instead. (PGC_GetFinds will become PGC.GetFinds). This is currently solved with a [i]search and replace[/i] in the LUA source code, which isn't the best solution.

Also all built in LUA functions will change names. With our old Sandbox we couldn't expose them with the same name, therefore tonumber was called ToNumber for example. Or string.byte was called StringByte. With the new Sandbox they will have their correct names. Existing checkers are search-and-replaced for this as well. Most unreliable search-and-replace is that 'Type will become 'type'. Other terms we search for probably won't match anything unintentional.


So why PHP 7.x then? Well, it's supposed to be more than twice as fast. It's supposed to consume less memory, though my single test I have made increased memory usage with 20%. We of course don't want to be stuck with old dependencies either. Long-term it's wise to try to catch up with recent releases.

Follow this topic if you want to make sure you don't miss anything. You can follow it by replying for example. When we are done with what we can thing of, we will probably open up the development environment and use it as a staging environment for a few days, to get some feedback, especially around LUA development.