"Different" mods sharing development through Git

[jda]

Specific context of this topic: Sharing data between the Dwarves Aglarond vanilla Glest-based faction and the Dwarves faction in GAE-based Project Red mod.
General context: The questions (and expected answers) in this thread may be useful to any other projects wanting to share data between themselves.

Presumptions (no need to reply on these but you can correct me if I'm wrong ):
P1. I (think I) understand Git is a development tool in the lines of old CVS or SVN.
P2. Github is a frontend to he Git backend mechanics, allowing you to freely host Giy projects.
P3: You can, in a somewhat automated way, build a stable public release from the data in your Git project. (I would need guidance on this too)

Questions:
Q1: (Not really related to data-sharing between projects yet): Can I build a stable release from selected file types/exentsions? E.g. I would have the full faction tree hosted there with the units'  current .g3d's but also the WIP .blend's in the same folders, but build the stable release using only the .g3d and .tga files, without packaging the .blend and .xcf files together?
Q2: Can I create my own Dwarves Aglarond project site on Github and (generally speaking) share data with the Dwarves faction in John.D.H.'s Project Red Git site?
Q3. If so, can there be something like John getting an announcemnt that I e.g. published a new Goat animation that he might want to use in his Dwarves faction? (as opposed to subscribing to all changes in his project)
Q4: And on a variation of Q3, can it be scripted to automatically add the new .g3d to his repository? (and vice-versa as well ofc; everything vice-versa on this thread )
Q5: Could either or both Q3 and Q4 be defined either by folder (and optionally subfolder) or by single file / sets of files? (this way, I might choose to have some improvement John might make to say a goat_walking.g3d file be automatically pushed to my own repo but not the goat.xml that might be using stats specific to balancing his Dwarves inside his GAE mod as oposed to my balancing to vanilla Magitech.)
Q6: Can I define some upcoming edits to be pushed to my own repo as e.g. new_edit.g3d.john or inverselly have a old_version.g3d.bak ?
Q7: I believe Git would keep track of any contribution of mine to John as being authored by myself. But suppose it wasn't me (site owner) who made a change but someone else (uploading directly to my site), would that someone else be credited instead (this would make things soooo much easier crediting-wise )?
Q8: Can you teach me HOW TO do all of the above?
Q9: Do you have any other questions or tips that you want to post? If so, please do! 

Thanks for reading and I hope this might be useful to someone else as well! 

[Yggdrasil]

Funny. I thought about similar things the other day. How can we get more modders to use an "open development model", sharing their source files in a public repository so that others can easily contribute. Most (all?) mods are "open source" so why not make the development open too. Git is a nice tool to do that.

Most of your questions sound like you haven't really understood what a version control system is. See wikipedia (git is a distributed/decentral vcs):
http://en.wikipedia.org/wiki/Revision_control

I can't really comment on github as i never used it myself. I can only talk about git.

Q1: you could try will's script:
https://forum.megaglest.org/index.php?topic=6207.0

When i worked on the exporter for blender i thought about some kind of build system for mods where you could also generate addon-compatible zip files for releases. I only have a batched export for blend files atm, only exporting the modified blends ofcourse. It's only a proof-of-concept as i don't know if modders need/want something like this. This could be also done for textures, converting xcf to tga or better png. You'll never forget to convert again and you can easily share your source files with others.

Q2-4: If you both work much on it, it's probably better to only use one repo and both have write access to it instead of merging every day. Not sure if github supports this. Ofcourse you can also make a fork and later make a pull request to get your changes into John's repo.
Q6: Sounds like you want different branches. Yes, you can have different versions of the same file.
Q7: git requires an author for every commit, so yes.

[Omega]

This should likely be very helpful: http://help.github.com/

[jda]

Thank you very much, Yggdrasil! Trying to educate myself through the links provided by yourself (checked Will's thread already as it seemed "easier"  ) and by Omega too. Thanks!

[jda]

Funny. I thought about similar things the other day. How can we get more modders to use an "open development model", sharing their source files in a public repository so that others can easily contribute. Most (all?) mods are "open source" so why not make the development open too. Git is a nice tool to do that.

Agree.

Most of your questions sound like you haven't really understood what a version control system is. See wikipedia (git is a distributed/decentral vcs):
http://en.wikipedia.org/wiki/Revision_control

The ISO 9000 series (Quality Management Systems)  I've been working in and out with since 2004 are not such a bad background to understand those. Including to understand this: it must be coherent and work for you. How? That's what you have to demonstrate to the Certifying Auditors, i.e. your way!
In this case, Git(hub)'s way. 

Q1: you could try will's script:
https://forum.megaglest.org/index.php?topic=6207.0

Checked it out, posted a comment, will post more (isn't working for me in the simplest test; also seems too restrictive - rules enforced that the engines themselves don't require). Could be a good starting point though.

When i worked on the exporter for blender i thought about some kind of build system for mods where you could also generate addon-compatible zip files for releases. I only have a batched export for blend files atm, only exporting the modified blends ofcourse. It's only a proof-of-concept as i don't know if modders need/want something like this. This could be also done for textures, converting xcf to tga or better png. You'll never forget to convert again and you can easily share your source files with others.

Speaking only for myself, I rather be the one making the conversion. My most recent change may not be the one I want to release, might only be an alterntative I pursued for a while, concluding I would not keep pursuing it, at least now, mayble later... Does this make sense?

Q2-4: If you both work much on it, it's probably better to only use one repo and both have write access to it instead of merging every day. Not sure if github supports this. Ofcourse you can also make a fork and later make a pull request to get your changes into John's repo.

Concerning the single repo for both my factions and John's/Wciow's one, at this point, I am unsure of either John's, Wciow's or even my own plans in this regard.
However, the "pull requests" you mentioned may at least partially lead to the answer I'm looking for as to whether me putting my Dwarves Aglarond on its own Github site might benefit John's work as well - I had planned and indeed declared I wanted to move the Dwarves into GAE later on, after some base-work relevant for the faction to be played on any of the three engines. At this point though, it might make more sense for me to keep working on common areas while John and Wciow focus more on the GAE aspects (no exclusivity on either part, necessarily of course). But what if ... ? A certain mutual independence might be mutually beneficial at this point and in the future.

Q6: Sounds like you want different branches. Yes, you can have different versions of the same file.

That doesn't really answer my question, or maybe I didn't phrase it that well... I mean, of course this whole topic is about branches! Now, I can have two de facto branches on two separate Git projects or two branches 'proper'  under one single Git project - that, in terms of the human work and organisation put into is basically a question of wording.
What I was asking is about a more concise issue: whether I can define beforehand, say per file how I want to pull it in! E.g.:
Case A: I ask John to do a new walking animation for my goat and I know I'm pretty sure it will be better than what I have now. I just want a backup of my old one, just in case.
Case B: John sends me a pull-request for something he did on his own, but we hadn't discussed it yet. So I pull his changes in but I don't want to make them the working-copy yet! They are there and I can easily make them the working copy after I review them - this would allow me to review them in my own time, possibly accepting them only after John made some other change which, this later, I might not want. Was this too confusing?

Q7: git requires an author for every commit, so yes.

Yes, that does make sense, being computer-controlled, it's easy so obviously implemented. But that was not the question. In other words, yes, that would be taken care of if we were under the same Git project.
But assuming we were on different Git projects, would everything John pulls from my project attributed to me (the project owner) or, in case it was so, would it be attributed to the registered contributor to my project?
Or is there not even the figure of a "project owner" in Git?

Thanks again for your reply and for reading this one. 

[Omega]

Q7: git requires an author for every commit, so yes.

Yes, that does make sense, being computer-controlled, it's easy so obviously implemented. But that was not the question. In other words, yes, that would be taken care of if we were under the same Git project.
But assuming we were on different Git projects, would everything John pulls from my project attributed to me (the project owner) or, in case it was so, would it be attributed to the registered contributor to my project?

I'm not entirely sure if I'd be answering this right, or if I even understand the question properly, but may as well take a shot. Exhibit A: a screenshot from a github repository (this is part of a failed attempt to make a git repository for all Glest mods).



As you see, each file has a message from its last commit (you can also view the history of commits). The message ends with the username of the person who committed the file last (in this case, John and me -- "omega" was taken). However, it's possible to credit a different author in the message by setting the message (which is done with git) to state the actual author. However, when downloading the file, the message is (obviously) not downloaded too, which would fail the requirements of many licenses, thus requiring you to still distribute a read me or authors file crediting everyone. Generally, though, git does a good job of keeping track of who changes what.

On the note of question 1 from your first post, you could use rsync to make a direct copy of the techtree, but give exclude filters for the file extensions you don't want, such as *.blend and *.xcf (Blender and the GIMP files).

On question 8, don't look at me, I'm actually fairly nooby at git. Just really good at googling.

[will]

You do not need to "package" your mod on github; people can easily download a zip of the mod directly from github using the 'download' button, and you can easily pass around the URL of the downloads there too.

To decide what's in a release and what isn't is just to make a 'branch' in git and put what you want there in it.

[Yggdrasil]

My most recent change may not be the one I want to release, might only be an alterntative I pursued for a while, concluding I would not keep pursuing it, at least now, mayble later... Does this make sense?

Then don't commit it or use another branch.

Case A: I ask John to do a new walking animation for my goat and I know I'm pretty sure it will be better than what I have now. I just want a backup of my old one, just in case.

You can access any commit afterwards, rollback to any previous version or whatever. No need for a separate backup.

Case B: John sends me a pull-request for something he did on his own, but we hadn't discussed it yet. So I pull his changes in but I don't want to make them the working-copy yet! They are there and I can easily make them the working copy after I review them - this would allow me to review them in my own time, possibly accepting them only after John made some other change which, this later, I might not want.

You can review the changes beforehand or merge it into a temporary branch for testing. If there's a commit you don't want just rollback or revert it explicitly.

Get away from thinking about backups and copies of a file. The whole history of the files, all versions are still accessible. You can branch of at any point and merge different states together.

Or is there not even the figure of a "project owner" in Git?

Not really. You're only the author of your commit. If you pull in commits from John it's still attributed to John.

However, when downloading the file, the message is (obviously) not downloaded too, which would fail the requirements of many licenses, thus requiring you to still distribute a read me or authors file crediting everyone.

Ofcourse a release shouldn't contain the git information. They're useless for players. Use 'git shortlog -s' to quickly get a list of authors.

To decide what's in a release and what isn't is just to make a 'branch' in git and put what you want there in it.

That's not really a good idea. If your master branch, where your development happens, contains all source files like *.blend and *.xcf and you just branch of releases, you have to remove all source files with every release again (ok, it's just a few commands, still). Using some script to get a release package and upload it somewhere else sounds better to me. Only make a tag in git.

[jda]

Exhibit A: a screenshot from a github repository (this is part of a failed attempt to make a git repository for all Glest mods).

Too bad it failed. I assume the modders didn't join up as expected...? It does remind me of a discussion there was on the Wiki about the infos to include in the Mods section, and what infos the respective authors would update or not - of course, many if not most just wouldn't update. It's a community of people contributing what they want pretty much as they want (which is great though!), not a corporation demanding said objectives and reports of their paid employees... Not at all saying you were demanding anything but we often forget it's kinda hard to have strict expectations when you can't demands...
Still, maybe little by little it will rise back and grow to some grandeur?

As you see, each file has a message from its last commit (you can also view the history of commits). The message ends with the username of the person who committed the file last (in this case, John and me -- "omega" was taken).

Yeah, had gotten that from looking at Project Red, and looking at webpage listings of SVN commits, maybe even CVS - I never actually used any myself but do recall browsing on occasions.

However, it's possible to credit a different author in the message by setting the message (which is done with git) to state the actual author.

Now that's interesting. Always good to have a manual override. 

However, when downloading the file, the message is (obviously) not downloaded too, which would fail the requirements of many licenses, thus requiring you to still distribute a read me or authors file crediting everyone.

That's ok, I can do that. What would bother me would be I'd fail to credit someone who had made a commit to Project Red or, reverselly, have to be saying all the time "Hey John, last commit I made wasn't me! I mean, it was, but the actual author was Omega!" - you get the picture...

On the note of question 1 from your first post, you could use rsync to make a direct copy of the techtree, but give exclude filters for the file extensions you don't want, such as *.blend and *.xcf (Blender and the GIMP files).

I see. Had forgotten completelly about rsync!  I had a bookmark to a pretty nice guide a while back, will look it up. Thanks for the tip.

On question 8, don't look at me, I'm actually fairly nooby at git. Just really good at googling.

My most recent change may not be the one I want to release, might only be an alterntative I pursued for a while, concluding I would not keep pursuing it, at least now, mayble later... Does this make sense?

Then don't commit it or use another branch.

I was just trying to provide some feedback on what you said about your batch exporter for .blend files. Then again, you did mention a good bit more than that... Well, nevermind!

Case A: I ask John to do a new walking animation for my goat and I know I'm pretty sure it will be better than what I have now. I just want a backup of my old one, just in case.

You can access any commit afterwards, rollback to any previous version or whatever. No need for a separate backup.

Case B: John sends me a pull-request for something he did on his own, but we hadn't discussed it yet. So I pull his changes in but I don't want to make them the working-copy yet! They are there and I can easily make them the working copy after I review them - this would allow me to review them in my own time, possibly accepting them only after John made some other change which, this later, I might not want.

You can review the changes beforehand or merge it into a temporary branch for testing. If there's a commit you don't want just rollback or revert it explicitly.

Well, this two-case question really was one question. But as you gave me different sollutions for each, you did give me the answer I was looking for: I can decide beforehand (or afterwards) what to do, one case at a time.

Get away from thinking about backups and copies of a file. The whole history of the files, all versions are still accessible. You can branch of at any point and merge different states together.

K, done! 

Or is there not even the figure of a "project owner" in Git?

Not really. You're only the author of your commit. If you pull in commits from John it's still attributed to John.

Right, but what if I pull a commit you made in John's repo? Would I instantly get that info or would I have to look it up on John's site? (for what I've read so far, I tend to think the author of that said commit I receive from John's repo would be 'Yggdrasil'... Right?

However, when downloading the file, the message is (obviously) not downloaded too, which would fail the requirements of many licenses, thus requiring you to still distribute a read me or authors file crediting everyone.

Ofcourse a release shouldn't contain the git information. They're useless for players. Use 'git shortlog -s' to quickly get a list of authors.

Nice tip! Thanks!

To decide what's in a release and what isn't is just to make a 'branch' in git and put what you want there in it.

That's not really a good idea. If your master branch, where your development happens, contains all source files like *.blend and *.xcf and you just branch of releases, you have to remove all source files with every release again (ok, it's just a few commands, still). Using some script to get a release package and upload it somewhere else sounds better to me. Only make a tag in git.

Well, the said script should be able to upload it back to a release-only Git branch (I do believe rsync can do that, ftp-wise or http-wise, AFAIR, secure versions of those too), pretty much the same as uploading it to some other place...?

Thanks for the thought, too, Will!

[will]

You do not need to "package" your mod on github; people can easily download a zip of the mod directly from github using the 'download' button, and you can easily pass around the URL of the downloads there too.

To decide what's in a release and what isn't is just to make a 'branch' in git and put what you want there in it.

To clarify, when I was doing this for the sci-fi pack, I had a release branch that, periodically, I'd pull the develop branch into and then run megaglest --validate-techtrees=x=purgeunused=purgeduplicates=svndelete=hideduplicates or whatever the exact commandline is

[Omega]

Exhibit A: a screenshot from a github repository (this is part of a failed attempt to make a git repository for all Glest mods).

Too bad it failed. I assume the modders didn't join up as expected...?

Actually, it failed largely because it was a lot more work than expected without large community support (git is complex, and rather few members of the community even know the basics). However, the main reason it failed is because it didn't hold up to the initial expectation of easy hosting. As mentioned, you could package the files as zip for download. Great. As someone with internet likely in the bottom 5%, the difference between 7z and zip is very noticable. Bear that in mind, as if you want to distribute the mod as a 7z file, you'll have to upload a 7z file as well.

[will]

At the moment, megaglest bundles 7z.exe and GAE has the 7z lib.

If it bundled git or libgit2 then it could all be transparent.

There's no need for all this talk of zipping stuff up and finding download servers for it all.  Still, I think its a hard sell.