Re: New LUA Sandbox

If you have read the topic Big boring changes incoming (PHP7) you know that we are up for some changes. I'll try to make it short to not repeat myself too much. Read the linked topic for more details.

We are upgrading to PHP 7 (from 5.6). The old LUA Sandbox implementation doesn't work anymore, we have added a new one. It affects quite a few scripts. It also affects the syntax that will be used in the future. Also, we are adding all of the LUA scripts to GitLab.

Due to current incompatibility issues between our Development environment and the Live servers we have disabled all editing of LUA source code and creating of LUA scripts on the Live servers. Until further notice, they have to be edited via a development server that will be used as a staging server. More information later in this post.

All scripts in THIS THREAD are broken with the new LUA sandbox. For most of them, there are notes about why. I will try to fix a few of them myself as well, as soon as I get the time. Most of them are very easy to fix.

As mentioned, to edit LUA scripts you now need to access a development server. Every user who has access to create LUA scripts now has will get a new menu item at Project-GC. Admin. In this menu you will find one item only, use it to grant yourself access to the development server. While on that, you can edit your own scripts, just as usual.

We have also pushed are also pushing all LUA scripts to GitLab.com, including all historical versions (with the wrong time-stamps though). When a script is edited at Project-GC.com it will automatically be stored both in our database and at GitLab.com. If edited at GitLab.com, it will be pushed to Project-GC.com as well. This push normally takes 1-2 seconds.

With these changes there are now three ways to edit LUA scripts.

1. At Project-GC.com, just as before.
2. Online at GitLab.com, via their online editors.
3. Cloning the repository, editing, pushing back to origin.

The repositories at GitLab.com are private. They are not open for the public (though we could very well make them readable in the future). If you want access there, register an account and reply to this post with your GitLab username, we will then grant you access to the repository, if you have access to writing LUA scripts at Project-GC.com. When editing scripts at GitLab.com you can edit ANY script, not just your own. This will hopefully help us emptying the list of broken scripts.

While fixing scripts it's important that we keep them compatible with the Live servers as well, which is using the old Sandbox. We therefore added a new parameter with a sandbox version that the scripts can access.
args[1].environmentSettings.sandboxVersion will equal 2 when in the new sandbox. In the old sandbox args[1].environmentSettings.sandboxVersion does not exist.
With this you could potentially have different code paths if needed.

All LUA functions have been renamed with the new Sandbox. For example, TableSort will become table.sort. Until later on you should not care about that. To keep it compatible with the Live servers we need to stick to the old syntax until the Live servers has been updated. We will then update the documentation as well.


There are 66000 versions of the scripts to push to GitLab.com. We are currently pushing one version per second (due to rate limits). Therefore it will take almost a day until they are all pushed. Once they are all pushed, we will add the menu item to access the development environment, and we will also start granting access to GitLab.com. Feel free to register there and give us your usernames beforehand.


PS! It's not impossible that we will use different branches on GitLab.com for keeping development versions of scripts in the future. This is step one towards endless of possibilities.



Edited 2 time(s). Last edit at 03/02/2018 09:54PM by ganja1447. (view changes)

Oh right. Forgot to make it extra clear. Most of the code around the LUA Sandbox has been rewritten. That includes error handling/feedback, timeouts, memory allowance and so forth.

Once you have access to the development server, feel free to test such things as well, including trying to produce errors and see if you get the error texts back.

Don't try any denial of service though. The development server does have less resources than the live system.

Gitlab account: vogelbird

ganja1447 Wrote:
-------------------------------------------------------
> Once you have access to the development server,
> feel free to test such things as well, including
> trying to produce errors and see if you get the
> error texts back.

I see error messages but they are pretty useless to locate the error. For example "Error 8, challenges.php:1590 Undefined offset: 0" means that there is no return statement in the end of the script and "Error 4096, challenges.php:1522 Object of class LuaSandboxFunction could not be converted to string" means that I forgot empty parentheses from a function call.

Is this just temporary or something which should be addressed?

arisoft Wrote:
-------------------------------------------------------
> ganja1447 Wrote:
> --------------------------------------------------
> -----
> > Once you have access to the development server,
> > feel free to test such things as well,
> including
> > trying to produce errors and see if you get the
> > error texts back.
>
> I see error messages but they are pretty useless
> to locate the error. For example "Error 8,
> challenges.php:1590 Undefined offset: 0" means
> that there is no return statement in the end of
> the script and "Error 4096, challenges.php:1522
> Object of class LuaSandboxFunction could not be
> converted to string" means that I forgot empty
> parentheses from a function call.
>
> Is this just temporary or something which should
> be addressed?

I am not sure how I could generally give better errors when you are using callback functions in an unexpected way.

The first error sure, a better error message could be given after checking if the return value is valid or not. But for the second error, that would mean a lot of error handling for every callback there is. And this specific error probably wouldn't be covered even then. I might be able to catch the exception and rephrase it though.

My point when asking if you got decent error messages actually was, "where you got it before". It's supposed to be at least as good as before. My focus was that the LUA errors are still getting back to the developer. But I do hear you. Some decent error handling could very well be implemented, like checking that the parameters to functions exists and are of the correct type.

I will definitely keep this in mind. I am playing with the idea to have user contributed modules for Profile stats as well, and then things like this should be kept in mind.

Gitlab account: pinkunicorn

Great idea syncing with GitLab.

Gitlab account: mole125

Gitlab account: josefpavlik
(hopelessly jpavlik has been already taken :-( )

Gitlab account: taxoft

Gitlab account: JuRoot

Gitlab account: PetziAt

gitlab account: sloth96

Gitlab account: AnnaMoritz

An update!

I was expecting the upload to GitLab to be done late Saturday, or at least Sunday morning. It hasn't.

Sometimes the script was aborted due to errors, and it had to be manually restarted with some adjustments, it wasn't built for knowing where it was. It's supposed to be a one time thing and I didn't want to spend too much time on the script.

At first I thought it was errors due to rate limiting. I knew there was rate limiting, but couldn't find any exact numbers. I then added a sleep, and it seemed to work better, but not good enough. The problem was that it was interrupted during night time when I didn't restart it.

I then added code to read the rate limit ratio. I found out that it was impossible to hit the rate limit with a single thread. So the errors I got seem to be random HTTP 500. Since it has been weekend I have had some other stuff to do as well and have just restarted the script, a lot. This evening I did get the time to rewrite some code though. It now does retries on HTTP 500 and seems to work much better.

The retry on HTTP 500 has only been implemented in this one-time script now, but I will change that for the "real" code tomorrow. It should be a very easy change.

The upload itself could have been much faster by cloning the repository and adding/committing files locally, and then push to origin. But I wanted to test the new code, to find issues like this. I actually found another unrelated big issue with PHP 7.x and the database servers as well while at it. It has been a great extra test of the development new environment.

The current status is that ~1100 of ~2700 scripts has been uploaded. But from now on, I am not expecting any interrupts, and all sleeps has been removed. It seems to be ~28500 commits left to do. I expect them to take 1-2 seconds each, so 8-16 hours. With some luck, it's all ready tomorrow before lunch!

And 5-6 hours ago the process was interrupted, and now the repository at GitLab.com doesn't even work.

Whenever I try to update a file via the API I get a HTTP 400 and "Could not update branch master. Please refresh and try again.".

Similar via their web interface. For example, when creating a new file, I get a HTTP 500 and "Could not update branch master. Please refresh and try again.".

I have contacted their support. The upside is that now we will find out how their support is.

Gitlab account: Iluvatar0

My GitLab account is sumbloke

NOW, the development environment and the new code is available!

I gave up on having GitLab answer today. I through away the broken project/repository and created a new one. I clone the repository, filled it locally and pushed the data. It was 10-20 times larger than I expected. Lots of LUA-code (with historical versions).

Everyone with LUA-access should now have an Admin menu item, from where they can access the development environment. From here you should be able to edit your scripts (or at GitLab). I have invited everyone who wrote their GitLab account (if I didn't miss anyone).

I'll take a short break, then I will start working on adjusting the scripts that uses the variable name table. Remember, if you fix any scripts, test run them afterwards, especially if it's not your own scripts!

Wow, great work! Can't wait to help with correction of codes. GitLab access works great, just a small question, where should we look for admin menu - maybe a small glitch, but I've looked around and can't find it. Thanks.

JuRoot Wrote:
-------------------------------------------------------
> Wow, great work! Can't wait to help with
> correction of codes. GitLab access works great,
> just a small question, where should we look for
> admin menu - maybe a small glitch, but I've looked
> around and can't find it. Thanks.

The Admin menu should show at https://project-gc.com/ , it should look like this, https://1447.se/shutter/Screenshot_20180305_16%3A22%3A05_c93.png

If it doesn't, I have failed in some way. I looked at the code, and it seems like I needed to add permissions both to the menu item and the menu itself. I had only added to the menu item. Try now and see if it works better.

It seems like permission for menu itself worked, now I can see that item and works great! Many thanks!

My GitLab account is akhegr

@arisoft

I patched https://dev-01.project-gc.com/Challenges//30511 (script 2514) so that it works in the new Sandbox. What surprised me though is that it takes 8 seconds on live servers and 48 seconds on dev servers. I can not explain why.

GitLab username: king-ton

king-ton Wrote:
-------------------------------------------------------
> GitLab username: king-ton

Added your name to the Project as well. Missed your post at first, found an email about it a bit deeper in my inbox.

GitLab account: thomfre

I see I'm a bit late to the party, hope I'm not too late :)

I will probably go ahead and update the live servers next week, regardless of the state of the challenge checkers. There are still a few scripts in https://project-gc.com/forum/read?6,17264,17329#msg-17329 with issues.

I will try to fix a few more before that though. But I will probably skip those that requires me to really understand the code. Those where I believe I know how to fix it already, I will try to fix. It's also easier to fix them when they work on the Live servers since the result can then be compared.

I am rushing getting the latest changes onto the Live servers. It's such big changes that I prefer to do it while I still remember most of what I went through upgrading DEV. Also, it's a bit cumbersome having the Dev-environment completely different from the Live servers.

My GitLab Account is hampf

hampf Wrote:
-------------------------------------------------------
> My GitLab Account is hampf

It wasn't too easy to add you as a member to the Project. I get four matches on hampf and none of them are clear to me.

• CJ Diffee@champfish
• Champ@ChampFTW
• Hampf!@Hampf
• Hannes@hampfes

Which one is correct?

GitLab account: TravelingGeek

GitLab account: Rikitan

Added developer access to Rikitan and TravelingGeek just now.