Login | Register
My pages Projects Community openCollabNet

Discussions > users > Temporary files size

fsvs
Discussion topic

Hide all messages in topic

All messages in topic

Re: Temporary files size

Author pmarek
Full name P.Marek
Date 2009-08-31 02:52:17 PDT
Message Hello Bogdan!

>> Please try the fsvs-1.2.x branch from the subversion repository
>> (r2370).
>>
> I try it, and it work significially faster (perhaps, because /mnt/fsfs
> was about 1M every time).
That's fine!

I still don't know why these files were over 300k for you.


> But now there are another bug: when I try to commit whole /home, fsvs
> fails 3 times with messages like:
>
> An error occurred: Device or resource busy (16)
> in ci__directory:
> add_file(./cosmetr/p​ublic_html/orifleym.​org.ru/Z_A_344/b�z​nes-plan-magazin-kos​metika.htm,
> source="(null)"@HEAD) returns 160005
...
> It was caused by cyrillic 'i' symbol encoded in cp1251 (8bit) charset in
> file names. I recoded file names using convmv, but I never sure which
> symbols will be in file name tommorow, users are unpredictable and
> cp1251 is standart encoding for Windows.
>
> Server locale is ru_RU.UTF8
That might be the problem.

If the server thinks it has an UTF8 locale, but local filenames are not encoded as such,
subversion will reject the names.

Depending on the way users put the files there, you could try to reconfigure
samba/apache (or whatever is used), so that they put UTF8 filenames there; or you could
tell FSVS that the filenames have to be encoded.

I think the right locale would then be ru_RU.CP1251.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Temporary files size

Author "Bogdan B dot Rudas" <bogdan dot rudas at nivalnetwork dot com>
Full name "Bogdan B dot Rudas" <bogdan dot rudas at nivalnetwork dot com>
Date 2009-08-31 01:05:08 PDT
Message On Fri, 28 Aug 2009 11:08:52 +0200 (CEST)
"Philipp Marek" <philipp at marek dot priv dot at> wrote:

> Hello Bogdan!
>
> I just found the bug.
> Please try the fsvs-1.2.x branch from the subversion repository
> (r2370).
>

I try it, and it work significially faster (perhaps, because /mnt/fsfs
was about 1M every time).

But now there are another bug: when I try to commit whole /home, fsvs
fails 3 times with messages like:

An error occurred: Device or resource busy (16)
  in ci__directory:
add_file(./cosmetr/p​ublic_html/orifleym.​org.ru/Z_A_344/b�zne​s-plan-magazin-kosme​tika.htm,
source="(null)"@HEAD) returns 160005



An error occurred: Device or resource busy (16)
  in ci__directory:
add_file(./cosmetr/p​ublic_html/tush.org.​ru/M_m_1044/konturn�​y-makiyaj.htm,
source="(null)"@HEAD) returns 160005



An error occurred: Device or resource busy (16)
  in ci__directory:
add_file(./cosmetr/p​ublic_html/tush.org.​ru/M_m_1710/denniy-k​rem-p�d-mak�yaj.htm,​
source="(null)"@HEAD) returns 160005

It was caused by cyrillic 'i' symbol encoded in cp1251 (8bit) charset in
file names. I recoded file names using convmv, but I never sure which
symbols will be in file name tommorow, users are unpredictable and
cp1251 is standart encoding for Windows.

Server locale is ru_RU.UTF8


--
Bogdan Rudas
System Administrator
Nival Network
bogdan dot rudas at nivalnetwork dot com

23, Matusevicha st., office 103,
Minsk, Belarus,
Tel: +375 17 2538272
http://www.nivalnetwork.com

Re: Temporary files size

Author pmarek
Full name P.Marek
Date 2009-08-28 02:08:57 PDT
Message Hello Bogdan!

>> The md5s are normal ... but do you have special user-defined
>> properties on all other files?
>>
>
> I do not specify any properties.
Ok.

> :/home/j2# /opt/fsvs/bin/fsvs pl -v public_html/j2.ru/j2.sql
> public_html/j2.ru/j2.sql has no properties.
> :/home/j2# /opt/fsvs/bin/fsvs pl -v .
> . has no properties.
> hosto:/home/j2# /opt/fsvs/bin/fsvs pl -v public_html/j2.ru/j2.sql
> public_html/j2.ru/j2.sql has no properties.
> :/home/j2# /opt/fsvs/bin/fsvs pl -v
> public_html/j2.ru/im​ages/tel.gif public_html/j2.ru/im​ages/tel.gif has
> no properties.and
>
> /home/j2 is "working copy" root.
What's in these files, then?
Could you take a look whether the files you tried really have a "prop" file stored?
Ie.
    /opt/fsvs/bin/fsvs pl info public_html/j2.ru/im​ages/tel.gif
should give a "WAA-Path" line; does this entry have a "prop" file?

If you find such a file, please re-run the "pl" (prop-list) command with "-d", and send
me the output. Maybe there are some hidden properties?

>> Which version of gdbm are you using?
>
> # dpkg -l | grep gdbm
> ii libgdbm-dev
> 1.8.3-3 GNU dbm database routines
> (development files ii libgdbm3
> 1.8.3-3 GNU dbm database routines
> (runtime version)
>
> It is Debian 4.0 amd64 system.
> FSVS, SVN and sqlite built from latest stable sources.
Nearly the same for me ...

I just found the bug.
Please try the fsvs-1.2.x branch from the subversion repository (r2370).

And you should be able to safely remove these files ... although empty gdbm hashes are
12kB for me, so I don't really know why they got so big for you.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Temporary files size

Author pmarek
Full name P.Marek
Date 2009-08-27 10:02:44 PDT
Message Hello Bogdan!

>> How many files are in your WC? Could you send me (privately?) a
>> find /mnt/fsvs -type f -ls
>> to examine?
>
> Yes, file attached.
Thank you.
There are ~73k files for user-defined properties, and 2400 md5s.

The md5s are normal ... but do you have special user-defined properties on all other files?

And the next strange thing ... they are all really big!

-rwxr-xr-x 1 root root 393216 Apr 27 01:21
   /mnt/fsvs/00/02/402c​05a9582284cb2382ad7b​c26a/prop

They should be more of the few kB size ...


You can look at these properties with
   fsvs pl -v <filename>

If there's no privacy problem I'd be interested in what is stored there...


I know that http:// and https:// needs some things to be stored, but that shouldn't be
that much.

Which version of gdbm are you using?


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Temporary files size

Author pmarek
Full name P.Marek
Date 2009-08-27 03:10:29 PDT
Message Hello Bogdan!

> I just commit one of my sites - HTML + Images,
>
> File size on disk ~= 6.5Gb
>
> Size of repo and temporary files:
> 6,2G /mnt/svnroot/
That the repository is only a bit smaller is ok - the pictures are already compressed.

> 27G /mnt/fsvs/
But this is strange ... that is a bit much space used.

How many files are in your WC? Could you send me (privately?) a
  find /mnt/fsvs -type f -ls
to examine?

> I'd like to commit whole /home - about 60Gb of text and images.
That shouldn't be a problem per se.
  (http://fsvs.tigris.o​rg/ds/viewMessage.do​?dsForumId=3928&​dsMessageId=2359487)​

> Now many free temporary space will I need for initial commit?
Well, it shouldn't be that much, actually ... I'm wondering why so much is used for you.

> Can I remove everything from my FSVS_WAA after each commit ?
No; at least some parts are needed.

You could try
  find /mnt/fsvs -type f -iname "md5s" -ls
to see whether the manber/MD5 hashes take so much space.
(These could be removed, although checking for changes might be slower then - but for
images it shouldn't really be a problem, as they seldom keep the exact size when
edited.)


Is this with 1.2.0?


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Temporary files size

Author "Bogdan B dot Rudas" <bogdan dot rudas at nivalnetwork dot com>
Full name "Bogdan B dot Rudas" <bogdan dot rudas at nivalnetwork dot com>
Date 2009-08-27 00:16:32 PDT
Message Hello!

I just commit one of my sites - HTML + Images,

File size on disk ~= 6.5Gb

Size of repo and temporary files:
6,2G /mnt/svnroot/
27G /mnt/fsvs/
 
I'd like to commit whole /home - about 60Gb of text and images.
Now many free temporary space will I need for initial commit?
Can I remove everything from my FSVS_WAA after each commit ?


--
Bogdan Rudas
System Administrator
Nival Network
bogdan dot rudas at nivalnetwork dot com

23, Matusevicha st., office 103,
Minsk, Belarus,
Tel: +375 17 2538272
http://www.nivalnetwork.com
Messages per page: