Lua coroutine threading and saved games

[BS-er]

After much research and stress I think I've finally met the necessary goals for simple threaded scripting for game missions.  The goals were as follows:

• A script system that supports saving the states of the data and script threads and properly restoring them when the saved game is loaded, so that they pick up where they left off.
• Very little brainpower required to set up these script threads properly.

The goal was accomplished using the Lua script language and library, and the Pluto addon library for Lua.

The elaborate details of saving and loading the script thread states and data are in C/C++ code and Lua scripts that the average mission designer need not even worry about.  All that is necessary is a few simple rules for setting up each thread.  This basically follows my theory for quality of game missions:

• QUALITY = (IMAGINATION + SKILL) / API_COMPLEXITY

[OvermindDL1]

I take if you have not looked at my multiplayer scriptor for BZ2 missions?  Allow me to give some pseudo-code in the BZ2 mission syntax I have for it:

Code:

from BZ2 import *

class ChilliMission(DLLBase):
    def InitialSetup(self):
        while self.gameTime<=200:
            AddToMessagesBox("%i: This is being repeated every 20 sleepings, until 200 in..."%self.gameTime)
            self.sleep(20)

CreateMission = ChilliMission

And if you saved it within 20 seconds, or someone joined in, etc... then they would all properly be synced and pick up where it left off.

Another example:

Code:

class BaseDied(Exception):
    pass

REFRESHTIME = 150 # 15 seconds
DISTANCEACTIVATE = 50 # 50 'meters'

class TestMission(DLLBase):
    def InitialSetup(self):
        self.bases = {}
        self.towers = {}
        for pathname in GetAiPaths():
            if pathname[:5] == "base_":
                self.new(self.handleBase, basePathName=pathname) # will start at next sleep or return
            elif pathname[:6] == "tower_": # form of "tower_baseId_towerId"
                bId = pathname[6]
                tId = pathname[8:]
                if not self.towers.has_key(tId):
                    self.towers[bId] = {}
                self.towers[bId][tId] = pathname

    def handleBase(self, basePathName):
        baseName = basePathName[5:]
        isNearby = 0
        Sleep(0) # Sleep for one tick to let things in BZ2 settle down
        myTowers = {}
        while True:
            myBase = BuildObject("abase", 0, pathname)
            try:
                while True:
                    if not myBase: # In C++ this is the same as if(!IsAlive2(myBase))
                        raise BaseDied()
                    nearest = GetNearestVehicleObject(myBase)
                    if nearest.GetDistance(myBase)>DISTANCEACTIVATE:
                        if nearest.teamgroup == myBase.teamgroup:
                            isNearby += 1
                        else:
                            isNearby = 0
                    else:
                        isNearby = 0
                    IFace_SetInteger(baseName, REFRESHTIME-isNearby) # Assuming already setup
                    if isNearby >= REFRESHTIME:
                        isNearby = 0
                        if myBase.team==0:
                            myBase.team = nearest.team # This literally calls SetTeam(myBase, GetTeam(nearest))
                        else:
                            for (key, pathname) in self.towers[baseName].iteritems():
                                if not myTowers.has_key(key) or (myTowers.has_key(key) and not myTowers[key]):
                                    myTowers[key] = BuildObject("aTower", myBase.team, pathname)
                                    break
            except BaseDied:
                for tower in myTowers.itervalues():
                    tower.Delete()
                myTowers.clear()

Not finished, not checked, I'm tired and just whipped it up, but should give you an idea of it.  And if a player joins then everything is replicated perfectly.  I also made a library called Pylang (an older example is in the BZU programming board "Actor Model" thread), which is pure message passing and perfect to replicate things out to as many nodes (processes, whether locally, across a network, or across the internet or what-not) made in the style of the language Erlang (which you should learn, just for the concurrency ideas it gives).

Do you have any code to demonstrate how yours goes?

Also, this site was essentially down for quite a while earlier...

[BS-er]

Thanks for posting that example.  I've caught a few threads on your multiplayer scriptor, which sounds like its just what the doctor ordered for multiplayer, and maybe general strat AI design.  I planned on poking around in it fairly soon.

I'll have to check out Erlang.  I see some resemblences to Ada, which is one of my work languages.  Erlang popped up in some of my searches while I looked for the best approach for saving coroutine states.  Lua won out by the fact that someone made an addon that seems to do the job perfectly.  For the life of me I couldn't find any other scripting language that had such a feature built in or added on.  If another language has such support I'd sure like to know.  I became fairly appalled that all sorts of authors and gurus praise coroutines as a decent alternative to state machines for mission and AI scripting, but support for flexible saving of coroutine states is practically non-existent except for a humble addon for Lua.

I'm a big fan of coroutines for single player scripting, because I saw the proof in the FE mod.  The DLL Scriptor was pretty crude and the simple threads weren't quite coroutines, but non-programmer types with expansive imaginations did some great things.

The following is my current slightly odd test case script, which should give you an idea of the simple syntax:

Code:

require "mission_interface"

-- Constants:
GoodbyeWorld = "Goodbye cruel World"

-------------------------------------------------------------------------------
-- This is a thread routine for scripting mission events.
-------------------------------------------------------------------------------
function mainRoutine()
-- If a variable parameter needs to be used in multiple routines,
-- it must start with "global." to survive a saved game.
global.TestString = "No really, goodbye"
display("Start")
wait(10)
display("Hello World")
local Attacker = getByLabel("Attacker")
Attacker:attack(getPlayer())
enableThread(global.delayedThread)
while getPlayer():distanceFrom(Attacker) > 30 do
yield()
end
display("Attacker is near.")
end

-------------------------------------------------------------------------------
-- This is another thread routine for scripting mission events.
-------------------------------------------------------------------------------
function secondRoutine()
local testLocal = "Testing locals"
wait(3)
display("Middle")
wait(10)
displayGoodbyeWorld()
yield()
display(global.TestString)
yield()
display(testLocal)
end

-------------------------------------------------------------------------------
-- This is a thread routine that will not be activated immediately.
-------------------------------------------------------------------------------
function delayedRoutine()
display("Delayed routine starting.")
wait(5)
display("End of delayed routine.")
end

-------------------------------------------------------------------------------
-- A function called from a thread routine.
-------------------------------------------------------------------------------
function displayGoodbyeWorld()
display(GoodbyeWorld)
yield()
display(GoodbyeWorld)
yield()
display(GoodbyeWorld)
end

-------------------------------------------------------------------------------
-- This is a required function that is called from C/C++ before the mission
-- starts.  The createThread() function must be called to create a separate
-- thread for each scripting routine.  The createThread() function should
-- not be called other than directly or indirectly from this function.
-------------------------------------------------------------------------------
function setup()
-- Create all threads to be used for the mission.
createThread(mainRoutine)
createThread(secondRoutine)
-- Save the thread Id for the following thread so that it can be used
-- later to enable/disable the thread:
global.delayedThread = createThread(delayedRoutine, false)
declarePermanent(displayGoodbyeWorld)
end

The last function is required overhead for setting things up.

After this I plan to explore ways to add scripting power as well, and maybe draw inspiration from your approach, but its too soon to let the cart get ahead of the horse just yet.

[OvermindDL1]

So you are doing a multi-'threaded' shared-memory model like I did in my bz2 mission scriptor.  The way Erlang and my Python library that duplicates Erlang, Pylang (yes, I put so 'much' thought in that name >.>), is they enforce a pure Actor pattern.  A little different programming style, but once I learned it I have come to like it better overall then any other concurrency style.  There is no shared memory (although you can still do so in Pylang, I've tried to make it difficult to do to help force an Actor model, ala Erlang), but everything communicates by pure message passing.  I have put up quite a few examples of Pylang around different places and can post them here if you wish to see them.  I setup Pylang so it can be as simple as an Actor model in Python, to as fully powerful as being fully distributed across as many computers as you want, such as in Erlang.  Python is definitally easier to program in then Erlang in my opinion (just from the libraries available for Python), but Erlang is faster then Python due to quite a large amount of optimizations done in it like in most pure functional languages.  However, binding to Erlang from other languages is done by creating C-Nodes that join the Erlang mesh like they are a normal Erlang node (could be possible to mix Erlang and Pylang, thought about creating an interface for that later), but the C-Node exists as another process, you cannot mix Erlang and another language in the same process (without a whole freakin ton of work, you are not supposed to do this and they made it difficult to do so, Erlang does a lot of special handling and switching so they just attempted to disallow direct interaction), but it is just as fast as anything else on the Erlang mesh.  Pylang, as stated it is slower then Erlang, but it can be purely embedded into a C/C++ application so multiple processes do not have to be spawned.  To be honest though, Erlang is more designed for the server-side, can setup a near perfect scalability, and although it works for the client-side, for a game you would probably still want that in C/C++.  So on the client you might still want another scripting language, like Python, although you can pass Erlang code objects (such as user scripted code) and execute it on an Erlang node.  Erlang really is quite powerful and decently fast, but it does require you to program in a very different way.  Pylang is the best balance of the Actor model, but with ease of use combining it with other things, although it is slower (I cannot wait till PyPy becomes usable).

[BS-er]

Yeah after my last post I did some googling of Erlang, and now I understand your discussions of "Actor model" a bit more, although further reading is planned.

Now that you've exposed me to Erlang, it's something I'll try to assimilate over the course of time.  Eventually I'll want to take a stab at an educational mini-project using it.  Thanks for bringing it to my attention.

[OvermindDL1]

I really do like it.  Some other things I have thought of doing is first, make an Erlang style library in C++ to do concurrency in a similar pattern, could even make it usable into a normal Erlang node as well (because the normal Erlang C++ Node library makes it so that the entire C++ process acts as one Erlang process, even though it (I think!) does transfer process information as well so you could emulate a full Erlang system, but written purely in C++ as well, that would allow the speed of C++ for critical functions (database, pathfinding, etc...), but allow a purely fault-tolerant and featurefull Erlang for running near everything else, only thing is that you could not be able to run arbitrary things on a C Node so any Erlang process communicating with it would need to know it is a C Node and not a normal Erlang node (which would be rather obvious since you would only do specific things, like pathfinding and database on the nodes, and not generic processing like on Erlang Nodes).

To do it in C++, the concurrency aspect could be written like a standard jump table hidden with macro's so you could literally write in C++ as you do in Erlang or Pylang, and although this would be more 'pretty', it is very difficult to do in Microsoft's compiler, whereas it is a synch in others (like GCC or MingW).  Or, could go the generic route of a class with an event loop, the event loop would not have to be a function though, but rather a function pointer so you could still do the other unique tests and flow controls like in Erlang and Pylang, just a little bit more brain-wracking in structure.

Here, this is a normal python example that I ported to Pylang, a little simulation (I'm sure you can find plenty of Erlang examples out there, I have, there is even a modeler, Wings3D, made in Erlang, so client-side apps are very possible):
And you need to install the Genshi plugin for SMF so that it can colorize code snippets if I put a language="Python" parameter in the code tag, this is far less readable without it:
I do not actually encourage this style, but it works, and this is a well known example that I ported (and it runs correctly)

Code:

"""
From "http://members.verizon.net/olsongt/stackless/why_stackless.html", remade
the 'Killer Robots!' example, not so much more scaffolding needed, eh?
Although, do not need the network stuff either...  Hmm, a distributed robot
fight, perhaps add in an input for a human or something as well on multiple
ends of the network mesh.  That will be a later project, just remaking this one
for now.
"""

from ..core import Process, getMessage, getSelfPid
from ..nameserver import NameServer
from ..stacklessExtensions import Sleep
import pygame
import pygame.locals
import os
import sys
import math
import time
import random



# Messages do not have to be classes, just makes testing faster since
# an isinstance is usually faster then most other tests, although it pickles
# up slower, a tuple, ala Erlang style, would probably be the fastest you
# could get, but just using classes here to make it clean.
class ActorMovementUpdate(object):
def __init__(self, angle, velocity, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()
self.angle = angle
self.velocity = velocity

class ActorInfo(object):
def __init__(self, name,location=(-1,-1),angle=0,
velocity=0,height=-1,width=-1,hitpoints=1,physical=True,
public=True, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()
self.name = name
self.location = location
self.angle = angle
self.velocity = velocity
self.height = height
self.width = width
self.physical = physical
self.public = public
self.hitpoints = hitpoints
def updateMovement(self, movementUpdate):
self.angle = movementUpdate.angle
self.velocity = movementUpdate.velocity

class ActorInfoJoin(object):
def __init__(self, actorInfo, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()
self.actorInfo = actorInfo

class WorldState(object):
def __init__(self, updateRate, time, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()
self.updateRate = updateRate
self.time = time
self.actors = []

class Collision(object):
def __init__(self, actor, collidedWith, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()
self.actor = actor
self.collidedWith = collidedWith

class KillActor(object):
pass

class KillMe(object):
def __init__(self, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()

class Damage(object):
def __init__(self, amt, actorPid=None):
self.actorPid = actorPid
if self.actorPid is None: self.actorPid = getSelfPid()
self.amt = amt



# Not making a class for world, probably should, would be cleaner, but I am
# trying to make things look excessively procedural here.
def worldProcess():
import math

NameServer.__nameserver__.registerNamedPid(getSelfPid(), 'world')
registeredActors = {}
updateRate = 15
maxupdateRate = 30

# Internal functions
def testForCollision(x, y, item, otherItems=[]):
if x < 0 or x + item.width > 496:
return getSelfPid()
elif y < 0 or y+ item.height > 496:
return getSelfPid()
else:
ax1,ax2,ay1,ay2 = x, x+item.width, y,y+item.height
for item,bx1,bx2,by1,by2 in otherItems:
if registeredActors[item].physical == False: continue
for x,y in [(ax1,ay1),(ax1,ay2),(ax2,ay1),(ax2,ay2)]:
if x >= bx1 and x <= bx2 and y >= by1 and y <= by2:
return item
for x,y in [(bx1,by1),(bx1,by2),(bx2,by1),(bx2,by2)]:
if x >= ax1 and x <= ax2 and y >= ay1 and y <= ay2:
return item
return None

def killDeadActors():
toKill = []
for (actor, actorInfo) in registeredActors.iteritems():
if actorInfo.hitpoints <= 0:
print "ACTOR DIED", actorInfo.name, actorInfo.hitpoints
actor(KillActor())
toKill.append(actor)
for actor in toKill:
del registeredActors[actor]

def updateActorPositions():
actorPositions = []
for (actor, actorInfo) in registeredActors.iteritems():
if actorInfo.public and actorInfo.physical:
x,y = actorInfo.location
angle = actorInfo.angle
velocity = actorInfo.velocity
VectorX,VectorY = (math.sin(math.radians(angle)) * velocity,
   math.cos(math.radians(angle)) * velocity)
x += VectorX/updateRate
y -= VectorY/updateRate
collision = testForCollision(x, y, actorInfo, actorPositions)
if collision:
#don't move
actor(Collision(actor, collision))
if collision and collision is not getSelfPid():
collision(Collision(actor, collision))
else:
actorInfo.location = (x,y)
actorPositions.append( (actor,
actorInfo.location[0],
actorInfo.location[0] + actorInfo.height,
actorInfo.location[1],
actorInfo.location[1] + actorInfo.width))

def sendStateToActors(starttime):
worldState = WorldState(updateRate, starttime)
for (actor, actorInfo) in registeredActors.iteritems():
if actorInfo.public:
worldState.actors.append(actorInfo)
for (actor, actorInfo) in registeredActors.iteritems():
actor(worldState)

while True:
e=getMessage(timeout=0)
initialStartTime = time.clock()
startTime = time.clock()
if e is None: # Timeout, no messages, so send info back out to all
killDeadActors()
updateActorPositions()
sendStateToActors(startTime)
calculatedEndTime = startTime + 1.0/updateRate
doneProcessingTime = time.clock()
percentUtilized =  (doneProcessingTime - startTime) / (1.0/updateRate)
if percentUtilized >= 1:
updateRate -= 1
print "TOO MUCH LOWERING FRAME RATE: " , updateRate
elif percentUtilized <= 0.6 and updateRate < maxupdateRate:
updateRate += 1
print "TOO MUCH FREETIME, RAISING FRAME RATE: " , updateRate

Sleep(calculatedEndTime-time.clock())
startTime = calculatedEndTime
Sleep(0.0)
elif isinstance(e, ActorInfoJoin):
print 'ADDING ', e.actorInfo.name, e.actorInfo
registeredActors[e.actorPid] = e.actorInfo
elif isinstance(e, ActorMovementUpdate):
registeredActors[e.actorPid].updateMovement(e)
elif isinstance(e, Collision):
pass # Known, but ignored
elif isinstance(e, KillMe):
registeredActors[e.actorPid].hitpoints = 0
else:
print '!!!! WORLD GOT UNKNOWN MESSAGE ', e




def display():
import pygame
import os

world = NameServer.__nameserver__.getNamedPid('world')

icons = {}
pygame.init()

window = pygame.display.set_mode((496,496))
pygame.display.set_caption("Actor Demo")

world(ActorInfoJoin(ActorInfo("display", public=False)))

datapath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data')

# Function declarations here:
def getIcon(iconName):
if icons.has_key(iconName):
return icons[iconName]
else:
iconFile = os.path.join(datapath,"%s.bmp" % iconName)
surface = pygame.image.load(iconFile)
surface.set_colorkey((0xf3,0x0a,0x0a))
icons[iconName] = surface
return surface

def updateDisplay(actors):
for event in pygame.event.get():
if event.type == pygame.QUIT: NameServer.__nameserver__.shutdown()
screen = pygame.display.get_surface()
background = pygame.Surface(screen.get_size())
background = background.convert()
background.fill((200, 200, 200))
screen.blit(background, (0,0))
for item in actors:
itemImage = getIcon(item.name)
itemImage = pygame.transform.rotate(itemImage,-item.angle)
screen.blit(itemImage, item.location)
pygame.display.flip()

while True:
e=getMessage(timeout=-1)
if isinstance(e, WorldState):
updateDisplay(e.actors)
else:
print "display: UNKNOWN MESSAGE", e





def basicRobot(location=(0,0), angle=135, velocity=1, hitpoints=20, name="basicRobot"):
world = NameServer.__nameserver__.getNamedPid('world')
world(ActorInfoJoin(ActorInfo(name,
  location=location,
  angle=angle,
  velocity=velocity,
  height=32,width=32,hitpoints=hitpoints)))
selfpid = getSelfPid()
while True:
e=getMessage(timeout=-1)
if isinstance(e, WorldState):
for actor in e.actors:
if actor.actorPid is selfpid: break
location = actor.location
angle += 3.0 * (1.0 / e.updateRate)
if angle >= 360:
angle -= 360
world(ActorMovementUpdate(angle, velocity))
elif isinstance(e, Collision):
angle += 73
if angle >= 360:
angle -= 360
hitpoints -= 1
if hitpoints <= 0:
Explosion(location, angle)
world(KillMe())
elif isinstance(e, Damage):
hitpoints -= e.amt
if hitpoints <= 0:
Explosion(location, angle)
world(KillMe())
else:
print "%s: UNKNOWN MESSAGE"%name, e



def Explosion(location=(0,0), angle=0):
time = 0.0
world = NameServer.__nameserver__.getNamedPid('world')
world(ActorInfoJoin(ActorInfo('explosion',
  location=location,
  angle=angle,
  velocity=0,
  height=32, width=32, hitpoints=1,
  physical=False)))
selfpid = getSelfPid()
while True:
e=getMessage(timeout=-1)
if isinstance(e, WorldState):
if time == 0.0:
time = e.time
elif e.time >= time + 3.0:
world(KillMe())
else:
print "Explosion: UNKNOWN MESSAGE", e




def Mine(location=(0,0)):
world = NameServer.__nameserver__.getNamedPid('world')
world(ActorInfoJoin(ActorInfo('Mine',
  location=location,
  angle=0,
  velocity=0,
  height=2,width=2,hitpoints=1)))
selfpid = getSelfPid()
while True:
e=getMessage(timeout=-1)
if isinstance(e, WorldState):
pass # Known about, but unused
elif isinstance(e, Collision):
if e.collidedWith is selfpid:
other = e.actor
else:
other = e.collidedWith
other(Damage(25))
world(KillMe())
print "MINE COLLISION"
else:
print "Mine: UNKNOWN MESSAGE", e




def minedropperRobot(location=(0,0), angle=135, velocity=1, hitpoints=20, name="minedropperRobot"):
world = NameServer.__nameserver__.getNamedPid('world')
selfpid = getSelfPid()
height=32.0
width=32.0
delta = 0.0
deltaDirection = "up"
nextMine = 0.0
world(ActorInfoJoin(ActorInfo(name,
  location=location,
  angle=angle,
  velocity=velocity,
  height=height,width=width,hitpoints=hitpoints)))
while True:
e=getMessage(timeout=-1)
if isinstance(e, WorldState):
for actor in e.actors:
if actor.actorPid is selfpid:
break
location = actor.location
if deltaDirection == "up":
delta += 60.0 * (1.0/e.updateRate)
if delta > 15.0:
delta = 15.0
deltaDirection = "down"
else:
delta -= 60.0 * (1.0/e.updateRate)
if delta < -15.0:
delta = -15.0
deltaDirection = "up"
if nextMine <= e.time:
nextMine = e.time + 1.0
mineX,mineY = (location[0] + (width / 2.0),
   location[1] + (width / 2.0))
mineDistance = (width / 2.0 ) ** 2
mineDistance += (height / 2.0) ** 2
mineDistance = math.sqrt(mineDistance)
VectorX,VectorY = (math.sin(math.radians(angle + delta)),
   math.cos(math.radians(angle + delta)))
VectorX,VectorY = VectorX * mineDistance, VectorY * mineDistance
x,y = location
x += width / 2.0
y += height / 2.0
x -= VectorX
y += VectorY
Process(Mine)((x, y))
world(ActorMovementUpdate(angle+delta, velocity))
elif isinstance(e, Collision):
angle += 73
if angle >= 360:
angle -= 360
hitpoints -= 1
if hitpoints <= 0:
Explosion(location, angle)
world(KillMe())
elif isinstance(e, Damage):
hitpoints -= e.amt
if hitpoints <= 0:
Explosion(location, angle)
world(KillMe())
else:
print "%s: UNKNOWN MESSAGE"%name, e



def spawner(location=(0,0), robots=[basicRobot, minedropperRobot]):
world = NameServer.__nameserver__.getNamedPid('world')
t = time.time()+0.5
world(ActorInfoJoin(ActorInfo('spawner',
  location=location,
  angle=0,
  velocity=0,
  height=32,width=32,hitpoints=1,
  physical=False)))
while True:
e=getMessage(timeout=t-time.time())
if e is None:
t = time.time()+(random.random()*5.0)+3.0
angle = random.random() * 360.0
velocity = random.random() * 1000.0
newRobot = random.choice(robots)
Process(newRobot)(location, angle, velocity)
elif isinstance(e, WorldState):
pass # Known about, but unused




# My generic process that will completely kill a system to make sure a test app
# does not run excessivly long.
def tempKillAll(waitTime=50, ns=None):
if waitTime==-1: return # never die by this process
if not ns:
ns = NameServer.__nameserver__
Sleep(waitTime%10)
for i in xrange(int(waitTime)-(waitTime%10), 0, -10):
print "%i seconds until death" %(i)
Sleep(10)
print "dieing"
ns.shutdown()

def runTest(timeout=60):
NameServer()
Process(tempKillAll)(timeout)
Process(worldProcess)()
Process(display)()
Process(spawner)( (32,32) )
Process(spawner)( (432,32) )
Process(spawner)( (32,432) )
Process(spawner)( (432,432) )
Process(spawner)( (232,232) )
NameServer.__nameserver__.run()


""" Run this at a command prompt at the directory that contains Pylang if Pylang is not properly installed, or just run it anywhere, like the interpreter after you add the Pylang path to import sys;sys.path.append(pathToPylangParentDirectory)
F:\Python25\ODL1 Temp Scripts>..\python -c "from Pylang.tests.KillerRobots import runTest;runTest()"
"""

[Avatar]

You guys scare me sometimes...

 

-Av-

[JonathanS]

You beat me to saying it, Avatar.

[BS-er]

I'm sure we don't scare you any more than we scare ourselves .

[OvermindDL1]

This is my life, I know nothing else.