Projects > cpp > Issues > Bug #4674

We are migrating issue tracker of Cocos2d-x Project to Github, please create new issue there. Thanks.

Create Issue on Github

cocos2d-x github repository is huge (and slow)

Bug #4674 [Closed]
ricardo 2014-04-03 23:43 . Updated over 9 years ago

Context:
If you try to clone cocos2d-x it will take a lot of time.
And this is because we are adding binary files (libraries in particular) to the github repo.

And when you clone the github repo, it downloads all the history... so it downloads all the binary libraries that we added.

Problem 1:
Cloning the repo takes a lot of time. Not good for developers, and not good for contributors.

Problem 2:
Since it takes a lot of time, using cocos2d-x as a subrepo is not an option for many projects.

Solution:

  • It is forbidden to add binary files in github. The only exception are sound and images for testing purposes. But no binary libraries should be added.
  • Binary libraries should be downloaded from a CDN or something like that
  • There should be an script in the repo that downloads the dependencies (like in Gameplay)
  • Binary files should be removed from the history of github
  • But before doing that, we should clone cocos2d-x repo and make it a backup... but the new repo should not have any binary lib in the history.

Additionally:

zhangxm 2014-04-04 08:36

I think it is ok for developing with git developers. But we should include the binaries in released zip file.

ricardo 2014-04-04 20:32
zhangxm 2014-04-05 02:41
  • Assignee set to dumganhar
  • Priority changed from Normal to Urgent
dumganhar 2014-04-28 09:37

I finally compressed .git folder from 730.8 MB to 285.8 MB
The way is:

java -jar ~/Software/bfg.jar -b 1M -D '.{obj,o,exe,dll,apk,so,a,ipa,pdb,ilk,exp,lib,suo,ncb,sdf,sbr,opensdf,swp,swo,png,jpg,jpeg,tiff,mp3,mp4,ogg,wav,caf,pvr,zip,rar,ccz,gz,webp,ttf,bmp}'
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Did this data fix our needs?

dumganhar 2014-04-28 10:02

But the SHA id will be changed for every commits. :(

ricardo 2014-04-28 15:07

285 is much better than 730. It is 2.5 times smaller.

For sure, that is a step in the right direction.
but would it be possible to make it under 100mb ?

Thanks!

ricardo 2014-04-28 15:08

Should we run that command in all the branches ? or that command automatically performs that operation on all the branches ?

thanks!

dumganhar 2014-04-29 01:53

I only checked out a v3 branch in my clean repo for testing.
I will test again for v1, v2 and v3.

ricardo 2014-04-29 02:01

ok, thanks. yes, we need to remove the huge binaries from all all the branches.

dumganhar 2014-04-29 02:03

but would it be possible to make it under 100mb ?

Maybe it could be achieved by cutting some oldest commits off.

dumganhar 2014-04-29 02:13

ok, thanks. yes, we need to remove the huge binaries from all all the branches.

I used that tool to remove file which is larger than 1M and which has extension of exe,dll,apk,so,a,ipa,pdb,ilk,exp,lib,suo,ncb,sdf,sbr,opensdf,swp,swo,png,jpg,jpeg,tiff,mp3,mp4,ogg,wav,caf,pvr,zip,rar,ccz,gz,webp,ttf,bmp

ricardo 2014-04-29 03:02

I wouldn't remove old commits.
In any case I would remove the "v1" branch. Nobody is using it.

dumganhar 2014-04-29 14:43

Updated:

Finally, I made the repo size reduced to 146M. It seems that there are no more gaps to make it smaller.

You could clone my repo to test it.

git clone https://github.com/angeltown/cocos2d-x.git
Cloning into 'cocos2d-x'...
remote: Counting objects: 203896, done.
remote: Compressing objects: 100% (65972/65972), done.
remote: Total 203896 (delta 135728), reused 195255 (delta 130873)
Receiving objects: 100% (203896/203896), 146.59 MiB | 672.00 KiB/s, done.
Resolving deltas: 100% (135728/135728), done.
Checking connectivity... done.

The command i used is:

java -jar ~/Software/bfg.jar -b 200K -D '*.{plist,json,ExportJson,z,obj,o,exe,dll,apk,so,a,ipa,pdb,ilk,exp,lib,suo,ncb,sdf,sbr,opensdf,swp,swo,atlas,bmp,caf,ccb,ccbi,ccbproj,ccbresourcelog,ccz,dds,fnt,fsh,gz,jpeg,jpg,ktx,mid,mp3,ogg,pkm,png,ppng,pvr,pvrraw,raw,tga,tiff,tmx,tps,tsx,ttf,wav,webp,zss,ztp}'
git reflog expire --expire=now --all
git gc --prune=now --aggressive
dumganhar 2014-04-29 15:01

Maybe it could be better. I will try another way. :)

ricardo 2014-04-29 15:32

thanks. not bad :)
but it is still a bit slow for me :(

I'm wondering if the test assets could be improved.
As an example, I know that we moved resources from directory to another many times.
The move operations (remove + add) were done in one single commit ? or in two (remove first, and then add) ?

If those operations were done one commit, then git will will treat it as a "rename"... otherwise, it will tread it as a "remove", and then "add".... and that will make the git history pretty big.

dumganhar 2014-04-29 15:35

yes, i think it should work too.
I'm trying to strip the history for cpp-test resources

dumganhar 2014-04-30 07:39

Eventually, repo size is about 50M now. :)
You could use https://github.com/angeltown/cocos2d-x.git to test.
But I will re-add resources which was removed. The size may be increased to about 80M.

dumganhar 2014-04-30 11:41
git clone git://github.com/angeltown/cocos2d-x.git
Cloning into 'cocos2d-x'...
remote: Counting objects: 197081, done.
remote: Compressing objects: 100% (57377/57377), done.
remote: Total 197081 (delta 132402), reused 184353 (delta 131784)
Receiving objects: 100% (197081/197081), 56.79 MiB | 614.00 KiB/s, done.
Resolving deltas: 100% (132402/132402), done.
Checking connectivity... done.
ricardo 2014-04-30 15:29

Ok. thanks. well done!

zhangxm 2014-05-20 06:07
  • Status changed from New to Closed

Atom PDF

Status:Closed
Start date:2014-04-03
Priority:Urgent
Due date:
Assignee:dumganhar
% Done:

0%

Category:-
Target version:v3.1