Author Topic: Finishing what we've started...  (Read 3315 times)

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Finishing what we've started...
« on: 25 August 2009, 04:58:45 »
I've been looking back a lot, especially on the wiki.

Lots has been started, but not finished. Just look at the wiki page for the GAE features. Some great things that are implimented into the XML, but are not yet implimented. As well, the effects and emanations seem very unstable, and crashed military so bad sometimes that I had to completely remove most of them. I can't figure out why, but some parts just give an access error or something.

As well, many features that were planned for FPM are still waiting, or even halfway done.

Forgive my arrogance, as I understand the hard, pestering work of coding, having delved into the source several times myself, but I think we should finish the OLD before we get on with the NEW.

As I will be reviving and taking over the FPM developement soon, I hope that by the time I finish (maybe a few months?) that at least some of the features needed for FPM will be implimented, as well as the work started by Daniel described in the wiki Reference page will be done.

As well, the AI will most likely not be able to handle the subfactions yet. They should treat subfactions just like any normal upgrade (or whatever). As well, the flags should be finished, and added to every command, not just effects. They will greatly boost AI. Of course, I understand AI is probably the second hardest thing to code (GUI looks pretty nasty to me, just looking at that source), but if hailstone can tackle the GUI, surely simply adding the ability to use a subfaction could be done within, say, a month.

Of course, I place much in the hands of coders. May god (or whatever deities you may worship, or none, if none) be with you.
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: Finishing what we've started...
« Reply #1 on: 25 August 2009, 09:56:16 »
You'll probably get at least some of them, when I get my computer back I have a little bit of work to do on the path finder, but thereafter I'll be wrapping up the refactoring as quickly as possible, and getting back to the 'fun stuff'

Here's a rough idea of what I'm planning, and in what order...

1. Finish the path finder.  I've been doing a lot of work over the last few days on graph paper and notepads, I've got some pretty pictures now and a page of well refined pseudo code, so hopefully this won't keep me occupied for too long.

2. Wrap up the refactoring. This brings the end user nothing, but make our lives easier developing it.

3. Finish support for shipyards.  the mechanism and placement 'rules' are already in there, but there was a bug in the building placement code, which I fixed before my computer went away. There is still the rotating to sort out of course.

4. Re-engineer the move skill.  This one might take a while, but I plan 'in the one hit' to add support for controlling how much speed is effected by slope, maximum slope (maybe up & down hill separately), and speed modifiers for different terrain types [these would all be optional parameters in the unit XML], to remove calculations for unit height, and just store the value [big performance gain], which will also make #4 easier.

5. Surface Props, the mechanism to support transports, walls, and garrisoning unit's in towers, (and probably other stuff, siege towers anyone?).  This would include 'ramps' initially, to walk onto walls with, with ladders and a climb skill with animation to come later, if enough people scream and yell for it :)

If you feel some other things are more important, or some broken features should be fixed first, speak up! I am listening.

(GUI looks pretty nasty to me, just looking at that source)
This is largely why Hailstone is tackling the new GUI, the current system is... errr, not very nice.

Glest Advanced Engine - Code Monkey

Timeline | Downloads

modman

  • Guest
Re: Finishing what we've started...
« Reply #2 on: 25 August 2009, 19:30:55 »
Right now, if this is the place, I would like to yell as hard as I can for LUA AIs.  LUA is the next best thing to C++ in programming AIs right now for Glest, so I would really like that.  Maybe I'll type out a long explanation of exactly how they should work when I get the time, but for now, school is about to start for me and I'm pretty busy.

And if this feature is implemented, I am willing to work out a template AI for everybody to use, so that it will work with all factions' simpler features.  This is kind of like what we have now, because the AI does well with just about all factions, but there are things that can trip it up a bit.

The next step is changing the template to be specific to a particular faction, so you can be more accurate when you tell it to do something.  For example, if you are telling it to build military, if I was working for a specific faction, depending on the time in the game (progress made by the faction so far), I could better tell the AI which units to build.

Mark

  • Guest
Re: Finishing what we've started...
« Reply #3 on: 25 August 2009, 20:52:59 »
I would like to yell my hardest for LUA AIs, but when that is done, I would REALLY love to have units walking on towers and such.  Those are just so cool!

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: Finishing what we've started...
« Reply #4 on: 26 August 2009, 05:26:19 »
Walking on towers DOES seem cool, but not a priority to me. To me, everything that is needed to finish the planned out FPM is priority, as well as AI.

I'm all with modman for lua AI, since scripting will save hours of time from recompiling. As well, it will stop the AI from getting all horrible when it encounters a new style of faction, which has happened on several occasions. It will also improve the AI for EVERY faction, since the AI plays some better than others (the good ol' magic vs tech conversation). I also volunteer to help with such a feature if ever implimented.

Summing this all up, I say finish what we've started (all I've listed, as well as GUI, refactoring, pathfinder, shipyards) and then go onto new things (move skill, ai, etc).
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

modman

  • Guest
Re: Finishing what we've started...
« Reply #5 on: 27 August 2009, 19:05:17 »
I'm all with modman for lua AI, since scripting will save hours of time from recompiling.
As well as months learning C++ basics for some.  Lua is so simple I think, that someone who is a little more than moderately computer-savvy would be able to pick it up within 3 hours of concentration.

In fact, I am willing to drop absolutely everything Glest-related I am doing if they were implemented.  This is assuming that Dark Magic is finished, because I don't want to delay that any more.  One+ years is enough for me.

Maybe when I get bored in school I'll do some pseudo-code AI instructions so when I can start using Lua I don't start from scratch.
« Last Edit: 27 August 2009, 19:08:42 by modman »

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: Finishing what we've started...
« Reply #6 on: 31 August 2009, 02:20:34 »
Actually, they will probably be too hard for the average user, but I think that a few of the biggest modders, all the coders, and most of the people who really make a difference wouldn't have too much trouble. You need to remember though, AI is not like a scenario.
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: Finishing what we've started...
« Reply #7 on: 31 August 2009, 04:05:32 »
Indeed, AI is complicated... very complicated. Using LUA instead of C++ will help the development cycle, but developing an AI is not going to be easy, in any language.

I'll slot this in at 3.5 then :)

Support will be 'primitive' initially, I'll try to get a draft interface together and start a new topic for discussion soon.
Glest Advanced Engine - Code Monkey

Timeline | Downloads

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,240
    • View Profile
    • http://www.titusgames.de
Re: Finishing what we've started...
« Reply #8 on: 31 August 2009, 18:23:48 »
I once added a new AI(ultra ultra) in the original glest. No not a real new AI, but I made a third AI selectable which had another ressource multiplier.
This was not too hard and it  think it would be easily possible to put some of the currently hardcoded parameters in the AI code into an xml file.
Parameters like "ressource multiplier" or other. Probably one can add some more things like "don't give more than X production commands to a building".

The big change with a lua supported AI is really something for later, but this tiny little changes I had in mind should be really easily possible for someone who knows how to use c++ better than me and it would give us so much fun.
BUT although this might sound like a good idea, please first try to create a stable release before starting anything else!

« Last Edit: 31 August 2009, 18:25:38 by titi »
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: Finishing what we've started...
« Reply #9 on: 1 September 2009, 17:57:53 »
Ah, that was YOU! All along I was thinking it was hailstone ;D

Can you show me ALL the parts you changed, or upload your source so I can compare to an original source? I was trying to do so in military, with an Expert and an easy version. I could get the option, but was having some trouble with adding the harvesting resources multiplier.

By the way, how far is v0.3? We are version 0.2.12 now. 0.2.13 should fix the bugs caused by implimenting glest 3.2.2 and impliment the new GUI.

I agree with Titi for the AI topic. Smaller changes might be more practical at first. A possible completely referbished AI can be done WAY later, if it is even deemed necessary (I'm busy imaginating Titi's possibilities. I mean, that could be all that's needed. I doubt most people would want to change too deep.
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

daniel.santos

  • Guest
Re: Finishing what we've started...
« Reply #10 on: 7 September 2009, 23:08:37 »
I just first want to say that silnarm is kicking a**!! =) I'm sorry I haven't gotten to catch up enough on the forums to understand all of the precise details of what you are doing, silnarm.

Water Units and Movement
My initial thought on the shipyard and water-born units was to base them on min & max depth, so a cruiser could operate in a minimum depth of 5 and a max depth of infinity while a battleship could require a minimum depth of 10.  Then, taller units (Goliath, etc) could walk through a maxiumum depth of 4 while regular humans could only walk through a maximum depth of 2.  Finally, you could have different move skills (maybe this is what you are doing?) so that units could walk/run, wade, and swim.  Normal surface movement is one move skill, wading is a second (for depth of 1 to 2 or so) and swimming a final, slower skill for depths over 2 -- this is just an example of the idea.  I think this encompasses the entirety of what was important in the paradigm I came up with for it last year, but deal with these numbers as "altitude" (negative for depth "below sea level") and then you automatically also have support for aerial units with restricted altitude (maybe we'll have caves some day).

As far as the AI and LUA, I still haven't dug into the Lua code to write too intelligently about this, but I presume what would have to happen is that all of the C++ objects that are associated with the AI would have to have their interfaces exported to LUA.  However, I would like to emphasize that most of what you are looking for has already been analyzed and partially specified, from a "requirements" level, just not implemented.  This was discussed in the Improved AI thread from 2008.  It should probably edited together and put in Wikia or something, but here's (what I think is) the significant portions of that thread to save you some digging.  Note that aside from where you explicitly tell it to in the first three settings for resources and map knowledge, this AI concept is does not involve cheating! (I don't like cheating AIs).

Quote from: modman
...because their only advantage is that they mine and harvest and research and build faster
Actually, I believe that they research, upgrade and build at the same speed.  They even execute harvesting skills at the same speed, they just produce 180%-ish of normal (i.e., cheating AI).  They do not have any awareness of other player's units, but they do always know the location of enemy bases. (some of this could be slightly inaccurate, but I don't think so).

Quote from: titi
We have CPU and CPU-Ultra now, but it would be nice to have some more steps. Some kind of cpulevel would be good!
I'm thinking along these same lines and have been working up some requirements for this.  Here's a few settings I have in mind (some of these apply to human and computer players alike)
  • Starting resources: 0%-2000% - a percentage of what is specified in <faction>.xml, normal is 100%
  • Resources Gathered: 10%-2000% - how much resources are gained when a harvester returns their load to a unit (building) that stores them, normal is 100%
  • AI Map Knowledge: 0%-100% - how well AI knows the map, 0% is never seen it before, 100% is remembers it perfectly.
  • AI Attentiveness: 0-1 - how well the AI stays on top of making sure they are producing units, keeping workers busy and responding to attacks.
  • AI Aggressiveness: 0-1 - zero is a home boy that never adventures out, 1 is reckless aggressor that only has military units at their base because it was either just produced or he's massing for an attack and happens to choose his base as a location for massing units (and only because it makes strategic sense offensively, not defensively).
  • AI Cunning: 0-1 - how likely AI is to employ special tactics discussed here and elsewhere and how well those tactics are executed -- this will be an ongoing work for some time. (see Randomize attack paths, Harassment, Distraction, Survivability and Sneak Attack below)
The first part of my new AI changes are going to be path finding AI followed by target assessment.  The path finding effects both human and computer players, but will be an important part of improving the CPU opponent because it will tally enemy units and be able to apply a target assessment on each one.  Thus, when the AI asks for a path, it doesn't just get a path, it gets a path with lots of meta data (risk assessment, time to traverse, how much it'll probably have to fight, how likely it is to die, how much is unknown about the path, etc.).

A few more ideas for more settings to tune the AI.  Like the aggression setting, the growth, upgrade to military, scouting settings are more for personality than directly saying how effective the AI should be although a middle number is typically the most effective.  In a campaign, however, especially one that's scripted, tweaking these for various teams in the AI can make a huge difference.
  • Harvesting Efficiency: 0-1 -- A zero doesn't plan where it builds resource storing buildings (as the current AIs) and also arbitrarily decides what to harvest, where as one would put build storage buildings in the best possible places (near the largest and closest clump of resources) and also harvest based upon an analysis of it's needs (i.e., it figures out what units, buildings & upgrades it is going to produce in advance and tries to plan resources for them)
  • Upgrade to Military Unit Production: 0-1 -- The ideal for this would be a 0.5, the zero puts upgrades at priority over producing military units at all costs, only producing military units once all either all upgrades are completed or the resources to produce a military unit are available and no upgrades can be done at the moment (because the building is doing another upgrade, etc.) while one will never produce an upgrade unless it has the resources to do so at a time is producing military units as fast as it can (with at least one extra in each build queue) and there are still resources left over.  In some campaigns or other situations where there are other friendly factions nearby(especially if the base is "deeper into" the map, being protected by other friendly bases) , it may also be optimal to use a higher lower setting for this (.8 or so 0.2 or so, i.e., upgrades faster) so that the full might of the faction is realized sooner, having a friendly faction nearby for protection during that vulnerable growth state.
  • Growth: 0-1 -- Zero is growing the base only as needed, one supports using recon to find new resources and take changes chances on building new camps, even when the base is unstable and it has no military units to defend the new base.  The optimal setting here is 0.5, only build new camps when the base is stable and a descent military force has been amassed to provide at least partial protection of the new base.  A setting of 0.25 would only build new camps when a very strong military has been amassed to protect the new base and setting of 0.75 would build new camps if there is at least a small number of military units to defend it.
  • Scouting: 0-1 --  Zero never scouts and one puts a very high priority on scouting, at the expense of not keeping a force large enough to protect the base.  This is tied to military unit production, as it's usually the military units who do the scouting.  In the current AI, this is also directly tied to aggression, because the "attack" and "massive attack" AI rules are triggered by a scout obtaining visual on an enemy unit and the scout also always attacks.  However, the new AI should have the scout behave in a fashion that is most likely to help him scout -- by staying alive.  Thus, good recon is available to support the final setting.
  • Situation-Based Attenuation (of other settings): 0-1 -- Zero is none (all other settings pretty much remain static through the life time of the AI), one allows for the AI to use threat assessment to adjust growth and upgrade to military production settings.  Thus, as the AI will always have a "threat assessment" level that rates the overall threat it expects and a "confidence level" that rates how confident it is in it's threat assessment based upon time alive, reconnaissance and the knowledge (or lack thereof) of how many enemy factions it's playing against.  When the AI first starts out, having experienced no hostile units, the "threat assessment" will always be very green (the world is safe) but it's confidence will be very low, because it hasn't been alive long and it hasn't explored much.  As the game progresses, it will slowly raise the threat assessment anticipating that it's enemies are building up their offensive capabilities.  As the AI (or it's allies) scouts more of the terrain and gains greater visibility, the AI's confidence level in it's assessment rises but the threat assessment wont change much until an actual enemy base is spotted and reconed, this will cause an appropriate adjustment to the thread assessment level and a significant rise in the confidence level.  If the base turns out to be more fortified or protected by more military units than previously thought, the threat assessment level rises sharply.  If it turns out that the base appears under developed, appears to have been heavily attacked by another enemy, or especially if it's being currently attacked by another enemy, the confidence level rises sharply threat assessment level drops (correction). Using a combination of the threat assessment level and the confidence level, the AI can adjust growth and upgrade to military unit production dynamically.

And here were my ideas I came up with for advanced AI techniques, actually from my 1st post to this board (and I would personally implement after the above before the following):
  • Randomize attack paths:  Make the approach that is taken for the massive attack rule vary.  It appears (in the 2.0.1 AI at least) that attacks generally come from a predictable path once the enemy base discovers your camp.  This makes it easier to anticipate the attacks and setup strategic defenses on those borders, leaving other borders unguarded with impunity. The scouting/patrol rule should endeavor to discover more routes to the enemies' camp(s).
  • Harassment: When military units are limited, but an enemy base has been located, small contingents sent in carefully can moderately disrupt harvesting.  The main goal of this type of attack is to target workers and other vulnerable units, but trying not to get killed unless the pay-off (i.e., disruption) appears high.
  • Distraction: This is a bit of an extension from randomizing the direction of attack.  Once alternate routes to an enemy base are discovered, and preferably at opposite ends of their base, a small  "suicide-mission" force can be sent in from one side, shortly followed by a massive attack on another.  The opponent will likely send the bulk of their military units to meet the distraction and leave other borders to their base vulnerable. Addendum: After re-reading this it occurred to me that it may not be obvious why this can be a good thing.  By being able to penetrate a base's perimeter, an invading force can relatively quickly destroy vulnerable targets, like workers, resource producing units (e.g., cows) and such before the defending force can return to meet them even from a short distance away.  This effect is compounded if the distraction force attempts to flee after the initial strike, drawing the defending forces further away from the base, although this wont be very effective against human players  (i.e., believable)  unless the Survivability (below) is implemented.  By destroying such vulnerable units early, the defender is denied the resources they would have otherwise produced during the siege.
  • Survivability: when regeneration is possible, the AI may want to sometimes have a single unit retreat from a battle when they are damaged to allow them time to regenerate or get healed/repaired by another unit.  This would be a pretty complex rule however, because there are a lot of factors should get weighed: the value of the unit (i.e., experience level), the importance of pressing the attack, how much the battle would suffer by the unit retreating, the likelihood that an attempt to retreat will still result in death, etc.
  • Sneak Attack: This is another high-complexity rule and would require spy units, which don't exist in the game now (the 2.0.1 at least).  The way this strategy works is that a small to mid-sized force culminates fairly near the enemy base, but not between their home base and the enemies and a spy unit observes the enemy base.  When the enemy deploys a large military force (to engage in a massive attack) and after a small delay, the sneak attack force attacks the relatively undefended base.  This has the dual effect of greatly damaging their base while potentially foiling their attack -- forcing them to withdraw back to defend their base.  Alternately, instead of a small force gathering somewhere and trying to remain hidden, the scouts in the area could be used.

As well, the effects and emanations seem very unstable, and crashed military so bad sometimes that I had to completely remove most of them. I can't figure out why, but some parts just give an access error or something.

I'll try to look at your Military thread this week and see if I can figure out what was causing the crashes.  If you didn't post details about what caused the crashes, please post them on that thread and/or create a bug report (once codemonger is back up, grrr, sorry).

As well, many features that were planned for FPM are still waiting, or even halfway done.

Right you are, sorry I haven't been around enough to get a lot of this stuff finished already.

Forgive my arrogance, as I understand the hard, pestering work of coding, having delved into the source several times myself, but I think we should finish the OLD before we get on with the NEW.

I agree.  However, as a "free software" project, none of us are getting paid and I can't tell hailstone or silnarm what to do.  In fact, I kinda think it's better if they focus on what they are most excited about because they'll do a better job on it than otherwise.

As I will be reviving and taking over the FPM developement soon, I hope that by the time I finish (maybe a few months?) that at least some of the features needed for FPM will be implimented, as well as the work started by Daniel described in the wiki Reference page will be done.

w00t!!

5. Surface Props, the mechanism to support transports, walls, and garrisoning unit's in towers, (and probably other stuff, siege towers anyone?).  This would include 'ramps' initially, to walk onto walls with, with ladders and a climb skill with animation to come later, if enough people scream and yell for it :)

One of the things I've been doing with my downtime is reading a book called "Besieged" about siege warfware from about 500 BC to 400 AD.  Most of the basics of this never changed through the medieval area and it's been quite the interesting read (we were really barbaric!!).

EDIT: Made a few more changes to quotes (corrections & clarifications), with changes in blue.
« Last Edit: 7 September 2009, 23:35:06 by daniel.santos »

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: Finishing what we've started...
« Reply #11 on: 8 September 2009, 01:56:34 »
Your Ideas are perfect. I've read them all before, but it was refreshening to go through again!

This is what I think silnarm and hailstone (and even you?) should put at priority number 1. Of course, as you stated, no one is being payed, so thus, it is truly the choice of the developers, and since I'm not really helping much, I shouldn't talk :D .

The AI concept is flawless, but I still wonder about the problems some have had in the past with the AI not producing types of resources, such as the trees in elves.
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

daniel.santos

  • Guest
Re: Finishing what we've started...
« Reply #12 on: 8 September 2009, 02:35:40 »
The AI concept is flawless, but I still wonder about the problems some have had in the past with the AI not producing types of resources, such as the trees in elves.
I'm glad you like the AI concept and I'm sure that it can be refined and improved even still -- mostly by adding more "cunning" mechanisms (like distractions, harassment, etc.) and improving the existing ones.  However, I'm not up on the trees/elves resource issue, just because I'm not caught up with the message board.  I would have to run Glest in a debugger to figure it out.  I do remember having some problems once where a computer-controlled faction wasn't producing properly.  I debugged and experimented a lot and, in the end, I think I adjusted values in the AI (somewhere in the 0.2 branch, probably around 0.2.8 or so).  I can't remember the exact nature of the problem then, off the cuff, but I do remember that it was resource related (it was trying to do things in an order that left no resources for building units).

If you added a new resource, it could be that the AI is hard-coded to a certain number of resources, that would be unfortunate and hopefully not too difficult to change.  As soon as I get codemonger.org back up, maybe you can enter a bug report for this.  I'm probably going to replace the CPU & mobo and if that doesn't solve the problem, we'll know it's definitely the kernel or compiler, although that would have to mean a long-standing bug because I'm using the stable Hardened Gentoo, which gets a lot of scrutiny and is used in a lot of production systems (because it's a very difficult OS to compromise).

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: Finishing what we've started...
« Reply #13 on: 8 September 2009, 02:55:48 »
Water Units and Movement

The stats for min and max depths would indeed be the ideal way of doing it, when I was initially working on this John made that very suggestion, and I thought it could work, with a second annotation in the annotated map for depth (or altitude if you like).  Unfortunately it's not as feasible as I first thought.  It could be done easily with the current search algorithms, but the hierarchical abstraction would become an absolute nightmare, so I went with the simpler option, deep or shallow.

Different move skills depending on the terrain type is a top notch idea, and indeed one I had not thought of.

... AI and LUA ...

I think most of that is on the tracker, and I have seen it.  Some very good ideas... I haven't given the LUA interface much thought as yet, but these are the kind of things I'd like to be able to do in LUA, obviously some of the more processor intensive things will need to be done in C++, but those functions would just form a part of the LUA interface.  You've got some pretty complex behaviours in there, I wouldn't want to be trying to implement it in C++, if just for the testing turn around time... Lua is actually quite fast for what it is, and with some decent error handling and logging (partially done now) it will be a delight for developing this kind of thing.

Quote from: daniel.santos
One of the things I've been doing with my downtime is reading a book called "Besieged" about siege warfware from about 500 BC to 400 AD.  Most of the basics of this never changed through the medieval area and it's been quite the interesting read (we were really barbaric!!).

Some might argue we still are :-X
Glest Advanced Engine - Code Monkey

Timeline | Downloads

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: Finishing what we've started...
« Reply #14 on: 8 September 2009, 20:52:00 »
Pros and Cons of Lua AIs
ProsCons
Easy to update, as no compiling is needed for every one character change.Requires major rewrites to the engine to impliment.
Easier use, which may prompt people with lua, but no C++ experience to helpNot as powerful as pure C++.
Very fast language.Increased filesize from lua scripts.
Allows for mods to bend the AI to play it better for that mod alone.If accidently edited, could screw up AI
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

daniel.santos

  • Guest
Re: Finishing what we've started...
« Reply #15 on: 8 September 2009, 23:36:58 »
...  Unfortunately it's not as feasible as I first thought.  It could be done easily with the current search algorithms, but the hierarchical abstraction would become an absolute nightmare, so I went with the simpler option, deep or shallow.

Ahh, I hadn't considered the implications for the path finder and I haven't seen your re-write yet, so I don't know what optimizations you are using that would get screwed with.  Too bad, otherwise, you could tell the pathfinder what your travel restrictions are (altitude limits, etc.).  But the pathfinder has been hurting for a long time and was desperately needed optimization, so I'm not feeling too jumpy to suggest screwing with that if you have it working better right now!

At some point in the future, if I get caught up with a lot of other things then I'll look again at the path finder design I was working out last year (that was part of the AI enhancement specification).  I don't remember if it's mentioned in either of those two threads I referred to earlier, but it was essentially a more robust pathfinder that made better use of cached paths, but also (optionally) returned other meta-information about the path and accepted more complex parameters (like "I want a path that will keep me X distance from any known enemy units").  The trick is that I wanted to have a mechanism to restrict these types of more complex queries so that they aren't generated several times a second for the same unit (because their path became invalidated by another unit moving into one of the spaces they intended to).

Another thing that annoys me (in several game, not just Glest) is when a faster unit is stuck behind a slower unit and the faster unit can't get around them.  I guess I'm mostly thinking of StarCraft.  In Glest (with the older path finder) they had a chance to get around them, but they wasted a lot of time.  I was trying to work out a solution to this problem that didn't out-right cheat, by trying to anticipate where other units were going to be X distance in the future, but I never came up with anything that I considered a clean solution.  In the end, I started visualizing all of this without the "units live in cells" concept, where the pathfinder only deals in terrain with objects of X size and works out a path in floating point.  I never got terribly far with that either, but it's still in my mind! =)

Among the things I did come up with, none of them were simple.  It mostly involved the division of the map into hierarchical areas of increasingly smaller sizes, with 4 areas at the top, 16 below that, 32 (EDIT: I mean 64) below that, etc to some depth that was appropriate.  Then, as paths are discovered, it's recorded (cached) rather or not there is a path from one area to the next in the grid at each level in the hierarchy with the values of "true", "false" or "unknown" along with some type of mechanism for measuring how "dirty" the data is.  This gives you a very fast reference as to rather or not there is a path between two cells in a map or rather or not you need to calculate it.  I never worked out the semantics for determining when a cached path is invalidated or not.  To implement what I was talking about before (with restrictions on the path based upon required width, depth/altitude, max incline/decline, specially tagged "terrain", rather or not there are enemies on the path, etc.), this tree would have to be duplicated for each unique set of parameters passed.  I suppose that how deep the map goes would depend upon the break-even point where the cache optimization in CPU cycles meets the memory usage (and thus, CPU cache hit ratio), but generally, I figured that the lowest level of the tree would describe an area of 8x8 cells, of which calculating a path within that should be pretty fast.

I guess the more I think about it, each node in such a tree would have to have an array of open paths that lead to another side described by the border cells (for the bottom nodes) or the border sibling nodes (for all upper nodes).  I really need to look at your code to make sure I'm not duplicating any effort (at least in analysis, as I'm not planning on coding any of this any time soon), but here's the general idea:
Code: [Select]
class AbstractNode {
    enum PathValue {
        PV_UNKNOWN = 0;
        PV_OPEN  = 1;
        PV_CLOSED = 2;
    };

    Node *parent;      // NULL for top level nodes
    Node *siblings[4]; // The nodes above, right, below and left of this one.  NULL when the node is a border node and doesn't have a sibling on that side.

    // In practice, these could be stored as bitfields, since PathValue only needs 2 bits.
    PathValue connectsToSiblings[4]; // the path value for going from this node to each sibling
}

/** Any node that isn't on the bottom is an "upper node" */
class UpperNode : public AbstractNode {
    Node *children[4];  // Each upper node will have exactly 4 children
    PathValue paths[6]; // The PathValue this node provides for each pair of siblings (query children for details)
}

/** These nodes directly represent a grid of cells, in this case, 8x8 */
class BottomNode : public AbstractNode {
    class Path {
        // This also wouldn't really be implemented this way, but essentially, this describes which cells of each sibling form this path
        bool openA[8];
        bool openB[8];
        // this class should actually provide some distance info somehow.
    }

    std::vector<Path> paths[6]; // This holds the paths that connect each pair of siblings.  There can be zero, one or more paths connecting two siblings.
}

Now this does nothing to address actually finding the quickest route by storing distance information for each entry/exit between nodes, but it's the base idea of the data structure.  It also doesn't address how we decide when a node is invalidated or any of the concepts of nodes/cells being closed permanently (permanent map object, like a rock), semi-permanently (resource nodes and buildings, which don't tend to move) or temporarily (a movable unit is blocking the path), but I suppose PathValue could have PV_CLOSED_PERM, PV_CLOSED_SEMI_PERM and PV_CLOSED_TEMP or some such.  So in the end, this would get represented by something like this (if you could pass C-style arrays to stl):

Code: [Select]
std::map<PathSpecification, UpperNode[4]> pathCache; // or just have a single top level node with no siblings.

In this example, PathSpecification describes the parameters I discussed earlier (min path width, min/max altitude, etc.) and implements operator==(const T&) or whatever STL maps need (I forget).

Quote from: daniel.santos
One of the things I've been doing with my downtime is reading a book called "Besieged" about siege warfware from about 500 BC to 400 AD.  Most of the basics of this never changed through the medieval area and it's been quite the interesting read (we were really barbaric!!).

Some might argue we still are :-X

ick! Too true!  And I thought that after I hit "post" it too!
« Last Edit: 8 September 2009, 23:40:33 by daniel.santos »

daniel.santos

  • Guest
Re: Finishing what we've started...
« Reply #16 on: 9 September 2009, 00:04:53 »
Oh, along these, lines I wanted to mention one more minor optimization idea.  When running GAE through gprof, I found that after path finding, the next thing that was eating CPU was searching for units in what used to be the UnitUpdater::unitOnRange() set of functions (and I still haven't seen your re-write of this yet, sorry).  The longer a unit's sight or attack skills are, the greater this burden.  I had the idea of dividing up each map into 16x16 areas and caching each unit living in that grid in a linked list that is updated on the World::moveUnitCells() function, where they are removed from any areas they leave and added to any they enter.  Because units can occupy more than one cell, they can be in more than one area.  Then, when you search for units, rather than iterating over each cell on the map in the area (which for a 12x12 area is 144 cells), you can iterate through the list of whatever 16x16 area(s) is in the radius you are looking.  I never did any work towards implementing this (that I recall anyway :) , but since you've addressed path finding, if this ends up taking the number one spot for CPU consumption then it may be worth a look.  I did try to improve performance by creating the PosCircularIterator class, but it ended up being more expensive for all operations except those that only care about the closest match, where it was faster (occasionally much faster).

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: Finishing what we've started...
« Reply #17 on: 9 September 2009, 08:49:32 »
... so I don't know what optimizations you are using that would get screwed with.
Actually, none yet. There's an Annotated A*, and an Annotated (Greedy) Best First, this could work quite well with them both... while I've made the path finder much faster, it still isn't fast enough to lift the node limit high enough to make it actually good (in terms of path quality), so abstraction is going to be needed.

Or maybe not... Whilst half way through writing the abstract map class, I had a brainwave, and completely rewrote the A* and Annotated Map.  It uses more memory, but still very reasonable (if I did my sums right, 1.2 MB for a 256*256 cell map (a 128*128 Tile map (oh, I've renamed SurfaceCell to Tile btw)). I haven't timed it yet, the A* is debugged and works but the new annotated map it searches on is a bit screwy, and I haven't yet implemented the interface I need to get my debugging textures for it. The new annotated map replaces virtually ALL of the new stuff, the 'node pool' object and it's helper structures, the double marker array, pointer array, and the binary heap are all gone.

This hasn't been committed yet, I want to get the annotated map fixed first, so if you're going to look at any of the new stuff, just look at the AnnotatedMap, and ignore the A* in it. The initialisation method is the only thing that has changed greatly, and there's a comment in the old code explaining exactly what should be done, I finally got around to doing it :)

Add in some 'local repair' to existing paths, so once you have a long path, you keep it! if you get blocked, trim off the next 10 (or whatever) steps, and get a new path to there, and splice that into your old path. This should make it 'fast' enough for the size maps we currently deal with.  Abstraction ultimately is still a must, I want bigger maps (and/or larger cell scale, which is effectively the same thing only it would make all current maps bigger too).

Quote
Too bad, otherwise, you could tell the pathfinder what your travel restrictions are (altitude limits, etc.).
I'm going to assume you've heard of influence maps, I was thinking it would be nice to have a 'generic' influence map, which could be the typical 'military strength' type, or anything else you cared to construct one for (favour high ground, not near buildings, etc). These could then be passed to the pathfinder with a search request, of course the hierarchical thing does make it a bit awkward, but I think we should try to figure something out...

Quote
... Among the things I did come up with, none of them were simple.  It mostly involved the division of the map into hierarchical areas of increasingly smaller sizes, with 4 areas at the top, 16 below that, 32 (EDIT: I mean 64) below that, etc to some depth that was appropriate. ...
This is a bit like Hierarchical A*, the 'original' abstracted A*. You're controlling the division much more tightly but it's basically the same idea.  I don't like the excessive layering myself, and suspect it wouldn't be much faster than standard A*, searching through all those hierarchies. 

I have however decided to adapt HAA* to be more like this. I'm just using the abstraction technique from HAA* and using the result to guide the low level A* (ie, to provide better heuristics) in the same way HA* does.  But I only want one abstract level, maybe two if we get really huge maps going, and the 'cluster' of low level cells would probably be 32*32, which I think will be fine, even that can be searched very quickly.

Quote
I guess the more I think about it, each node in such a tree would have to have an array of open paths that lead to another side described by the border cells (for the bottom nodes) or the border sibling nodes (for all upper nodes).  I really need to look at your code to make sure I'm not duplicating any effort (at least in analysis, as I'm not planning on coding any of this any time soon), but here's the general idea:
Well, kind of... but I haven't committed that code yet... read on...

Quote
Code: [Select]
class AbstractNode {
    // omitted
}
Now you're talking my language  ;D

Ok, here's what I have, you'll have to excuse me.. I figured this all out on graph paper and notepads while my computer was away, there was no prototyping, this is 'production' code....
Code: [Select]
// This will actually be 'packed' using bitfields...
struct Entrance {
int clearance;
int position;
};

struct Border {
// Entrances of max clearance for each field
Entrance transitions[FieldCount];

/* intra cluster edge weights
for vertical borders...        for horizontal borders...
       2      3        1
   +------+------+    +------+ [0] = NW
   |      |      |    |      | [1] = N
1 |  C1  |  C2  | 4 0 |  C1  | 2 [2] = NE
   |      |      |    |      | [3] = SE
   +------+------+    +------+ [4] = S
       0      5    |      | [5] = SW
   [0] == SW 5 |  C2  | 3
   [1] == W    |      |
   [2] == NW    +------+
   [3] == NE        4
   [4] == E
   [5] == SE
*/
float weights [FieldCount][6];
};

class AbstractMap {

Border *vertBorders, *horizBorders, sentinel;
int w, h; // width and height, in clusters
AnnotatedMap *aMap;

public:
AbstractMap ( AnnotatedMap *aMap );
~AbstractMap ();

// 'Initialise' a cluster (evaluates north and west borders)
void initCluster ( Vec2i cluster );

// Update a cluster, evaluates all borders
void updateCluster ( Vec2i cluster );

// compute intra-cluster path lengths
void evalCluster ( Vec2i cluster );

bool search ( SearchParams params, list<Vec2i> &apath );

void getBorders ( Vec2i cluster, vector<Border*> &borders, Border *exclude = NULL );

// Border getters
Border* getNorthBorder ( Vec2i cluster );
Border* getEastBorder ( Vec2i cluster );
Border* getSouthBorder ( Vec2i cluster );
Border* getWestBorder ( Vec2i cluster );
};

So we have had essentially the same idea.  I have no abstract node, I only store the borders, the maximum clearance value for each field along the border, what position along the border that is, and the edge weights to it's neighbour borders. I will also be caching results from searches in the abstract map, haven't given to much consideration to the exact nature of this, you've given me a bit to think about there actually :) (ie, 'dirtiness' of those results).

The above is only partially implemented, the initialisation and evaluation functions... the search I haven't bothered with, as again, I need debugging textures so I can see what's going on, in this case I need entirely new ones, and a new interface, code in the renderer, etc etc etc....

Yikes... I'll be taking a break now :)
Glest Advanced Engine - Code Monkey

Timeline | Downloads

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: Finishing what we've started...
« Reply #18 on: 9 September 2009, 22:01:22 »
... Then, when you search for units, rather than iterating over each cell on the map in the area (which for a 12x12 area is 144 cells), you can iterate through the list of whatever 16x16 area(s) is in the radius you are looking.  I never did any work towards implementing this (that I recall anyway :) , but since you've addressed path finding, if this ends up taking the number one spot for CPU consumption then it may be worth a look.

I have had the same idea, but for different reasons :) All I've done to the unitOnRange() family is rename them along the lines unitInRange(), trimmed the parameter lists a bit, and moved them (and this is only in the refactor branch for now).  I did however look over it, and it looked a processor hungry function... I certainly wanted to come back and have a closer look. 

I thought about the map division only very recently, while thinking about location based triggers for LUA, as this is something that is going to need to be done in an efficient manner.  In short, I think this will be a very worthwhile endeavour.  Of course, to keep in the spirit of this topic, we'll have to put it on the back-burner for the moment :-)
Glest Advanced Engine - Code Monkey

Timeline | Downloads