Lua Interpreter

Message boards : Code and Servers : Lua Interpreter
Message board moderation

To post messages, you must log in.

AuthorMessage
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3604 - Posted: 16 Sep 2019, 22:48:26 UTC

I am working on boinc app that interprets Lua. It is not aimed to run the computation in lua, that would be quite wasteful. But the lua script could for example set up the enviroment, compile the main algorithm (in C) and then exec() it or drive the computation form high level.
Without CI/CD infrastructure, deploying or changing boinc application is hard and time consuming process. Having scriping environment available that could change the app from one workunit to another will i think be great help. There have been effort, with sztaki GenWrapper, but it only allowed posix-shell scripts, that are arcane to code anything but simple. Lua is also smaller and faster. With access to compiler, it should open great possibilities.

There will be an app in this very project to test that.
ID: 3604 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Sergey Kovalchuk

Send message
Joined: 18 Feb 19
Posts: 6
Credit: 41,603
RAC: 0
Message 3606 - Posted: 18 Sep 2019, 8:21:46 UTC - in response to Message 3604.  

Do I need any additional packages installed for the new application? For example, boinc-dev
Or will you provide everything you need along with the task (like boinc2docker projects)?
ID: 3606 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3607 - Posted: 18 Sep 2019, 12:11:47 UTC - in response to Message 3606.  

Do I need any additional packages installed for the new application? For example, boinc-dev
Or will you provide everything you need along with the task (like boinc2docker projects)?

There are two options. If a library is found installed, it will be used, otherwise:
1) The script will detect the missing libraries and ask user to install required packages via a boinc notice and a message in boinc log.
2) The script will detect the missing libraries and download them locally. The package will be installed for use by this project only.

The eventual solution will likely be a mix of the two, but we will try to always give user the explanation of why the task failed and how to fix it.

To answer your question, No, you do not have to install anything yet. The last lua8_t2 and lua8_t3 workunits are just poking around the system and generally checking what works and what does not.
ID: 3607 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Michael Goetz
Avatar

Send message
Joined: 18 Feb 19
Posts: 22
Credit: 147,831
RAC: 861
Message 3608 - Posted: 18 Sep 2019, 14:46:29 UTC

A lot of the people I know who run BOINC for fun -- but treat it as a serious hobby -- besides doing the traditional "run BOINC in your spare computer cycles" thing, actively buy, build, and or rent computers just for running BOINC projects.

Many times Linux machines are set up just for running BOINC. This includes VMs running on Windows machines as well as cloud servers from AWS, or less expensive options like Digital Ocean. In most cases, these are minimal Linux installations, sometimes with a boinc-client package installed and nothing else.

Of the three linux VMs running in my house, only one has compiler tools. The one that ran your Lua tasks does not. And, of course, every cloud server I've spun up for running BOINC did not.

I would be curious to know how many of your _t3 test results found a compiler installed vs. not installed. Please share your results!
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

ID: 3608 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3611 - Posted: 18 Sep 2019, 18:22:43 UTC - in response to Message 3608.  

I would be curious to know how many of your _t3 test results found a compiler installed vs. not installed. Please share your results!

I was not smart about the output and did not make the result of compiler detection easily parse-able out of the stderr. I can try my luck at regex or submit new batch.
ID: 3611 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Michael Goetz
Avatar

Send message
Joined: 18 Feb 19
Posts: 22
Credit: 147,831
RAC: 861
Message 3612 - Posted: 18 Sep 2019, 20:01:01 UTC - in response to Message 3611.  
Last modified: 18 Sep 2019, 20:03:16 UTC

I would be curious to know how many of your _t3 test results found a compiler installed vs. not installed. Please share your results!

I was not smart about the output and did not make the result of compiler detection easily parse-able out of the stderr. I can try my luck at regex or submit new batch.


No need to run new tasks. The information is sitting there in your database.

SELECT 
  COUNT(*) "Total Tasks", 
  SUM(stderr_out LIKE '%gcc: not found%") "No gcc",
  SUM(stderr_out LIKE '%g++: not found%") "No g++"
  SUM(stderr_out LIKE '%gcc: not found%" AND stderr_out LIKE '%g++: not found%") "Neither",
FROM 
  result
WHERE
  server_state=5 
  AND outcome=1
  AND stderr_out LIKE "%lua8_t3.lua%
  AND appid=2 ;


Change appid to whatever is used for your lua app.

It's easier to detect the error message from the compiler being missing than it is to detect all possible valid responses from multiple compiler versions, but that's possible if you really want to do it. For example, you can parse the output to determine which compiler version is present.
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

ID: 3612 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3615 - Posted: 19 Sep 2019, 20:05:00 UTC
Last modified: 19 Sep 2019, 20:10:22 UTC

Thank you very much Mr. Goetz for your SQL. Let me show you the results as of now:
+-------------+--------+--------+---------+
| Total Tasks | No gcc | No g++ | Neither |
+-------------+--------+--------+---------+
|          36 |     13 |     17 |      13 |
+-------------+--------+--------+---------+

More results may come in later.
However, the script does not set locale to C, so the file not found messages might be localized and therefore not caught by the regex.
ID: 3615 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3616 - Posted: 19 Sep 2019, 20:12:16 UTC
Last modified: 6 Oct 2019, 21:40:34 UTC

Another version (updated 06.10.):
+-------------+---------+---------+------+
| Total Tasks | Has gcc | Has g++ | Both |
+-------------+---------+---------+------+
|          56 |      42 |      25 |   25 |
+-------------+---------+---------+------+

SELECT 
  COUNT(*) "Total Tasks", 
  SUM(stderr_out LIKE "%gcc (%") "Has gcc",
  SUM(stderr_out LIKE "%g++ (%") "Has g++",
  SUM(stderr_out LIKE "%gcc (%" AND stderr_out LIKE "%g++ (%") "Both"
FROM 
  result
WHERE
  server_state=5 
  AND outcome=1
  AND name LIKE "lua8_t3_%"
  AND appid=8
ID: 3616 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3636 - Posted: 6 Oct 2019, 22:28:38 UTC

About dependency management.

There will be a script file, that the application loads, where user will be able to override automatic dependency management.
If a workunit depends on something (eg compiler), it will first consult the user override script, and if no such option was set by user it will proceed with automatic dependency management. First the dependency will be searched in some standard location. Eg: compiler on linux in PATH. If it is not found, and we have means of installing it, it will be downloaded and installed locally. Because downloading executables is what trojans do, we will ONLY download dependencies in source form and with checksums. This rules out downloading a compiler and any big frameworks.
If the dependency can not be found nor installed, the user will be notified via boinc notice and log message. Inside that message, a link to project page with instructions will be provided. After that, the workunit will either fail, or deffer itself for some time.
I am not fan of automatically downloading a dependency if there exists easily-installable package. Even windows users will be expected to use something like MinGW/MSYS2, which features a package manager.
ID: 3636 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
VADemon

Send message
Joined: 30 Oct 19
Posts: 3
Credit: 25,796
RAC: 0
Message 3758 - Posted: 4 Nov 2019, 21:31:11 UTC
Last modified: 4 Nov 2019, 21:31:49 UTC

It is not aimed to run the computation in lua, that would be quite wasteful. But the lua script could for example set up the enviroment, compile the main algorithm (in C) and then exec() it or drive the computation form high level.
[...] sztaki GenWrapper, but it only allowed posix-shell scripts, that are arcane to code anything but simple.

1) One could use LuaJIT and even work with C structures in Lua for extra performance - which would be quite fast in total, but I agree that it's (a little) wasteful in the long run.
2) So it would need to compile the algorithm every time, read: for every new task? If so, it'd be wasteful again, no?
3) I hate to write in shell too :)

Also installing anything close to a working compiler toolchain on Windows under Cygwin is an atrocious process. I can imagine under Linux those specialised stations that boot from USB would have problems too installing/keeping compiler software (think those "miner rigs" with nothing else but mobo+cpu+few gpus)
ID: 3758 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 505
Credit: 407,379
RAC: 525
Message 3759 - Posted: 4 Nov 2019, 22:23:51 UTC - in response to Message 3758.  

VADemon
LuaJIT can improve the speed, but it is pulling the shorter end. For LuaJIT to use native processor extensions (vectorization etc), it would need to be compiled with those instructions enabled and thus not compatible with other processors. Still, it is not native code, and it can't optimize for specific processor model.
Using C structures and calling C functions is nice, but those would have to be compiled into the app, making it specific for the task and version.
Loading dynamic library with those optimized functions could be a solution, but boinc effectively mandates statically linked binaries, which do not support loading dynamic libraries.
The script will first check if the real app has been already compiled with latest version version and recompile only if it is missing or outdated. This is pretty basic to figure out.
Installing compiler on windows is only little more inconvenient than on system with package manager, because you have to download the installer and click next. Have a look at MinGW or MSYS2.
Installing compiler toolchain on linux system with package manager takes one command. It is no harder than installing the boinc client itself.
Before you ask, yes, there will be a file where you can set the compiler path or override something.
ID: 3759 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Code and Servers : Lua Interpreter

©2020 Tomáš Brada