Login | Register
My pages Projects Community openCollabNet

Discussions > dev > Recursive Revert

fsvs
Discussion topic

Hide all messages in topic

All messages in topic

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-10-03 08:42:55 PDT
Message On Tuesday 02 October 2007 Philipp Marek wrote:
> On Friday 28 September 2007 Simon Spr√ľnker wrote:
> > I am using svn+ssh://.
>
> With svn+ssh:// and a 100MB random-bytes file I can reproduce the
> memory usage. The only problem is ... it doesn't seem to be a problem
> in my functions, but in subversion:
Just in case someone's interested:
    http://marc.info/?l=​subversion-dev&m​=119132816314689​&w=2


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-10-02 05:04:16 PDT
Message On Friday 28 September 2007 Simon Spr√ľnker wrote:
> I am using svn+ssh://.

With svn+ssh:// and a 100MB random-bytes file I can reproduce the
memory usage. The only problem is ... it doesn't seem to be a problem
in my functions, but in subversion:


Breakpoint 1, 0x00002b4c96ff7540 in sbrk () from /lib/libc.so.6
(gdb) bt
#0 0x00002b4c96ff7540 in sbrk () from /lib/libc.so.6
#1 0x00002b4c96fa1649 in __default_morecore () from /lib/libc.so.6
#2 0x00002b4c96f9d997 in ?? () from /lib/libc.so.6
#3 0x00002b4c96f9ed63 in malloc () from /lib/libc.so.6
#4 0x00002b4c973b84dc in apr_palloc () from /usr/lib/libapr-1.so.0
#5 0x00002b4c965bcea6 in svn_stringbuf_ensure () from /usr/lib/libsvn_subr-1.so.1
#6 0x00002b4c965bcf32 in svn_stringbuf_appendbytes () from /usr/lib/libsvn_subr-1.so.1
#7 0x00002b4c9844758f in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#8 0x00002b4c9844764d in svn_ra_svn_read_item () from /usr/lib/libsvn_ra_svn-1.so.1
#9 0x00002b4c98440acc in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#10 0x0000000000413a91 in rev__get_file (sts=0x63baf0, revision=10, fetched=0x7fff147299d8, only_tmp=0x0, pool=<value optimi
#11 0x0000000000414505 in rev__action (sts=0x63baf0, path=0x63d178 "./asdf") at revert.c:445
#12 0x0000000000404024 in ac__dispatch (sts=0x63baf0, path=0x63d178 "./asdf") at actions.c:53
#13 0x000000000041a762 in waa__update_tree (root=<value optimized out>, cur_block=0x63bbe0) at waa.c:1894
#14 0x000000000041a97b in waa__partial_update (root=0x7fff14729b80, argc=1, normalized=0xfffffffffffff000, orig=0x7fff14729d
#15 0x000000000041c84c in waa__read_or_build_tree (root=0x7fff14729b80, argc=1, normalized=0x632350, orig=0x7fff14729d88, re
#16 0x0000000000413546 in rev__work (root=0x7fff14729b80, argc=1, argv=0x7fff14729d88) at revert.c:556
#17 0x000000000040c78c in main (argc=4, args=0x7fff14729d78) at fsvs.c:991


The memory gets allocated to increase a stringbuf ... although revert.c:176 says
    stream=svn_stream_fr​om_aprfile(a_stream,​ subpool);
so it should go directly into a file.

(BTW, the same happens on update - and I can reproduce it there, too.)

I've got libsvn1=1.4.4dfsg1-1.


So, for a summary -- I can see the problem, but as yet I don't
know where it happens.


Regards,

Phil


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-28 16:54:09 PDT
Message Philipp Marek wrote:
>> Update: If I remove the 400 MB Swap, so that the VM only has 800 MB RAM,
>> fsvs gets killed again. And I can confirm now, that the memory only gets
>> filled up, when big files are processed. In my case big means 304 MB.
>> Fsvs gets killed after writing 240 MB of it.
>
> Hmmm ... I now tried with files of 100M (random values) and 300M (zeroes),
> and fsvs would just get them ... with ~100M virtual memory, but actually
> only 4.4M memory used.
>
> top says:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8406 flip 18 0 104m 4384 3024 R 99.9 0.5 0:01.48 fsvs
>
> time says something similar - 1200 faults, times 4096 bytes.
>
> $ /usr/bin/time -v ~/svk/direct-2/fsvs/src/fsvs revert ag -d -D main
> 09:12:42.614 main[fsvs.c:812] no argument at optind=6 of 6
> 09:12:42.617 main[fsvs.c:948] optind=2 per_sts=224 action=revert rec=1
> 09:12:42.617 main[fsvs.c:951] argument 1: revert
> 09:12:42.617 main[fsvs.c:951] argument 2: ag
> 09:12:42.617 main[fsvs.c:960] filter has mask 0xFFFFFFFF (new, removed, changed, owner, group, mtime, umode, props, child, likely)
> Reverting to revision 7:
> .mC. 100000000 ag
> 09:12:45.429 main[fsvs.c:1005] memory stats: 0x652000 to 0x6df000, 564 KB
> Command being timed: "/home/flip/svk/dire​ct-2/fsvs/src/fsvs revert ag -d -D main"
> User time (seconds): 2.62
> System time (seconds): 0.19
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.82
> Average shared text size (kbytes): 0
> Average unshared data size (kbytes): 0
> Average stack size (kbytes): 0
> Average total size (kbytes): 0
> Maximum resident set size (kbytes): 0
> Average resident set size (kbytes): 0
> Major (requiring I/O) page faults: 0
> Minor (reclaiming a frame) page faults: 1285
> Voluntary context switches: 1
> Involuntary context switches: 196
> Swaps: 0
> File system inputs: 0
> File system outputs: 0
> Socket messages sent: 0
> Socket messages received: 0
> Signals delivered: 0
> Page size (bytes): 4096
> Exit status: 0
>
>
> I'm not sure what is causing this.
>
> Which URL scheme are you using? I'm testing with file://.

I am using svn+ssh://.

Best Regards,
Simon

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-28 09:32:49 PDT
Message Philipp Marek wrote:
>> Update: If I remove the 400 MB Swap, so that the VM only has 800 MB RAM,
>> fsvs gets killed again. And I can confirm now, that the memory only gets
>> filled up, when big files are processed. In my case big means 304 MB.
>> Fsvs gets killed after writing 240 MB of it.
>
> Hmmm ... I now tried with files of 100M (random values) and 300M (zeroes),
> and fsvs would just get them ... with ~100M virtual memory, but actually
> only 4.4M memory used.
>
> top says:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8406 flip 18 0 104m 4384 3024 R 99.9 0.5 0:01.48 fsvs
>
> time says something similar - 1200 faults, times 4096 bytes.
>
> $ /usr/bin/time -v ~/svk/direct-2/fsvs/src/fsvs revert ag -d -D main
> 09:12:42.614 main[fsvs.c:812] no argument at optind=6 of 6
> 09:12:42.617 main[fsvs.c:948] optind=2 per_sts=224 action=revert rec=1
> 09:12:42.617 main[fsvs.c:951] argument 1: revert
> 09:12:42.617 main[fsvs.c:951] argument 2: ag
> 09:12:42.617 main[fsvs.c:960] filter has mask 0xFFFFFFFF (new, removed, changed, owner, group, mtime, umode, props, child, likely)
> Reverting to revision 7:
> .mC. 100000000 ag
> 09:12:45.429 main[fsvs.c:1005] memory stats: 0x652000 to 0x6df000, 564 KB
> Command being timed: "/home/flip/svk/dire​ct-2/fsvs/src/fsvs revert ag -d -D main"
> User time (seconds): 2.62
> System time (seconds): 0.19
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.82
> Average shared text size (kbytes): 0
> Average unshared data size (kbytes): 0
> Average stack size (kbytes): 0
> Average total size (kbytes): 0
> Maximum resident set size (kbytes): 0
> Average resident set size (kbytes): 0
> Major (requiring I/O) page faults: 0
> Minor (reclaiming a frame) page faults: 1285
> Voluntary context switches: 1
> Involuntary context switches: 196
> Swaps: 0
> File system inputs: 0
> File system outputs: 0
> Socket messages sent: 0
> Socket messages received: 0
> Signals delivered: 0
> Page size (bytes): 4096
> Exit status: 0
>
>
> I'm not sure what is causing this.
>
> Which URL scheme are you using? I'm testing with file://.

I am using svn+ssh://.

Best Regards,
Simon

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-28 00:16:34 PDT
Message > Update: If I remove the 400 MB Swap, so that the VM only has 800 MB RAM,
> fsvs gets killed again. And I can confirm now, that the memory only gets
> filled up, when big files are processed. In my case big means 304 MB.
> Fsvs gets killed after writing 240 MB of it.

Hmmm ... I now tried with files of 100M (random values) and 300M (zeroes),
and fsvs would just get them ... with ~100M virtual memory, but actually
only 4.4M memory used.

top says:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 8406 flip 18 0 104m 4384 3024 R 99.9 0.5 0:01.48 fsvs

time says something similar - 1200 faults, times 4096 bytes.

$ /usr/bin/time -v ~/svk/direct-2/fsvs/src/fsvs revert ag -d -D main
09:12:42.614 main[fsvs.c:812] no argument at optind=6 of 6
09:12:42.617 main[fsvs.c:948] optind=2 per_sts=224 action=revert rec=1
09:12:42.617 main[fsvs.c:951] argument 1: revert
09:12:42.617 main[fsvs.c:951] argument 2: ag
09:12:42.617 main[fsvs.c:960] filter has mask 0xFFFFFFFF (new, removed, changed, owner, group, mtime, umode, props, child, likely)
Reverting to revision 7:
.mC. 100000000 ag
09:12:45.429 main[fsvs.c:1005] memory stats: 0x652000 to 0x6df000, 564 KB
        Command being timed: "/home/flip/svk/dire​ct-2/fsvs/src/fsvs revert ag -d -D main"
        User time (seconds): 2.62
        System time (seconds): 0.19
        Percent of CPU this job got: 100%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.82
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 0
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1285
        Voluntary context switches: 1
        Involuntary context switches: 196
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0


I'm not sure what is causing this.

Which URL scheme are you using? I'm testing with file://.


Regards,

Phil


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-27 22:59:26 PDT
Message Philipp Marek wrote:
> Hello Simon!
>
> Sorry it took so long - r1151.

Update: If I remove the 400 MB Swap, so that the VM only has 800 MB RAM,
fsvs gets killed again. And I can confirm now, that the memory only gets
filled up, when big files are processed. In my case big means 304 MB.
Fsvs gets killed after writing 240 MB of it.

Best Regards,
Simon

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-27 10:31:18 PDT
Message Philipp Marek wrote:
> Hello Simon!
>
> Sorry it took so long - r1151.

Hi Phil,

not a problem, thanks for solving my problems all the time :)

I already did a recursive revert and it worked! It got past the point
where fsvs used to die the last times and memory usage until this point
was moderate (~ 100 MB). But on that point, which is the revert of a
file of about 300 MB in size, memory (800 MB) was filled up quickly and
fsvs is swapping since then.
When another big file was hit, it took fsvs quite some time to do the
revert, because of all the swapping, but it succeeded.

I will test later what happens if there are bigger files to revert.

Thanks a lot!
Simon

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-27 06:41:35 PDT
Message Hello Simon!

Sorry it took so long - r1151.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <simon at spruenker dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <simon at spruenker dot de>
Date 2007-09-20 22:26:30 PDT
Message Philipp Marek wrote:
> Hello Simon!
>
>> the report didn't find anything regarding fsvs, then some things
>> distracted me from looking further into it.
>>
>> I did however test version 1.1.9 in my virtual machine. A complete
>> revert of / still used too much memory and got killed by the system.
>> Initially the VM had 300 MB, then I increased to 800 MB, but the problem
>> persisted.
>>
>> Here is what happens in the file system: When the revert starts, it
>> processes some man-pages, then /var/lib, then three linux sources (which
>> take really long), and I think then a quite big file is read from the
>> repository. During this operation the fsvs process gets killed, but when
>> the big file is starting to be processed, fsvs uses already about 80% of
>> system memory. Does this help you?
> Well, I'll try to reproduce this problem.
>
> Which versions of libapr and libsvn are you using?

libapr: 1.2.7-8.2
libsvn: 1.4.2dfsg1-2
libaprutil: 1.2.7+dfsg-2

The system is an up to date Debian 4.0r1. If you need anything else,
just let me know.

Best regards,
Simon

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-20 22:09:29 PDT
Message Hello Simon!

> the report didn't find anything regarding fsvs, then some things
> distracted me from looking further into it.
>
> I did however test version 1.1.9 in my virtual machine. A complete
> revert of / still used too much memory and got killed by the system.
> Initially the VM had 300 MB, then I increased to 800 MB, but the problem
> persisted.
>
> Here is what happens in the file system: When the revert starts, it
> processes some man-pages, then /var/lib, then three linux sources (which
> take really long), and I think then a quite big file is read from the
> repository. During this operation the fsvs process gets killed, but when
> the big file is starting to be processed, fsvs uses already about 80% of
> system memory. Does this help you?
Well, I'll try to reproduce this problem.

Which versions of libapr and libsvn are you using?


Regards,

Phil


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-20 12:59:44 PDT
Message Ph. Marek wrote:
>> I don't have much experience using it, but I am doing a
>> check with it on recursive revert right now. It will probably take a few
>> hours. Are you interested in the report?
> Of course I am! If you take a look into tests/Makefile, you'll see the
> default parameters I normally use.

Hi Phil,

the report didn't find anything regarding fsvs, then some things
distracted me from looking further into it.

I did however test version 1.1.9 in my virtual machine. A complete
revert of / still used too much memory and got killed by the system.
Initially the VM had 300 MB, then I increased to 800 MB, but the problem
persisted.

Here is what happens in the file system: When the revert starts, it
processes some man-pages, then /var/lib, then three linux sources (which
take really long), and I think then a quite big file is read from the
repository. During this operation the fsvs process gets killed, but when
the big file is starting to be processed, fsvs uses already about 80% of
system memory. Does this help you?

Best regards,
Simon

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-05 22:19:17 PDT
Message Hello Simon!

> Seems like it's still leaking. Do you use valgrind to check memory
> management?
I did, but all I saw was the file:/// URL using more than 1 MB ... which I
couldn't remove.

> I don't have much experience using it, but I am doing a
> check with it on recursive revert right now. It will probably take a few
> hours. Are you interested in the report?
Of course I am! If you take a look into tests/Makefile, you'll see the
default parameters I normally use.

There's even a simple way to run it:
  make run-tests CHECKER=valgrind
(although that has the problem that the reports will be deleted with the
wc directories ;-(

This was meant more or less as a reminder how to call it ...
I have to check against current versions, whether the parameters still exist.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-05 09:50:55 PDT
Message On 15:51:34 you wrote:
> Sure, just tell me when you are ready.
Today morning I committed :-)


Regards,

Phil


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-05 06:51:34 PDT
Message Ph. Marek wrote:
> Hello Simon,
>
>> I just built from Revision 1101 and got:
>>
>> An error occurred: No such file or directory (2)
>> in rev__action: The directory structure was changed; the path
>> ./usr/share/bug/exim4/script is no longer accessible.
>>
>> Shouldn't recursive revert create that path then?
> Yes, of course. While testing for the memory leak I found that, too.
>
> Could you please try the version coming soon? Mostly I'd be interested
> whether the memory leak is resolved.

Sure, just tell me when you are ready.

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-04 22:05:23 PDT
Message Hello Simon,

> I just built from Revision 1101 and got:
>
> An error occurred: No such file or directory (2)
> in rev__action: The directory structure was changed; the path
> ./usr/share/bug/exim4/script is no longer accessible.
>
> Shouldn't recursive revert create that path then?
Yes, of course. While testing for the memory leak I found that, too.

Could you please try the version coming soon? Mostly I'd be interested
whether the memory leak is resolved.


Thank you very much for your efforts!


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-04 22:05:07 PDT
Message Hello Simon,

> I just built from Revision 1101 and got:
>
> An error occurred: No such file or directory (2)
> in rev__action: The directory structure was changed; the path
> ./usr/share/bug/exim4/script is no longer accessible.
>
> Shouldn't recursive revert create that path then?
Yes, of course. While testing for the memory leak I found that, too.

Could you please try the version coming soon? Mostly I'd be interested
whether the memory leak is resolved.


Thank you very much for your efforts!


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-04 11:16:01 PDT
Message Ph. Marek wrote:
> Hello Simon!
>> I was testing fsvs revert -R on a Virtual Machine and ran into problems:
>> I versioned a machine's / and wanted to deploy the backup of this
>> machine to a Virtual Machine.
>>
>> When doing "fsvs revert -R /" with 1.1.7 on the VM, my process got
>> killed after running for an hour and reverting files. I assume because
>> it was eating all the memory of the VM, because memory usage was
>> constantly growing.
> I'll take a look at that.
>
>> So I tried 1.1.8 and got:
>> 15:56:00.790 ops__traverse[est_ops.c:1129] INTERNAL BUG
>> !path[0]
>>
>> sh: line 1: 7061 Segmentation fault sudo fsvs revert -R /
>>
>> It didn't even start reverting anything. Are you aware of this?
> Such a bug is already fixed in the current version; I'll release 1.1.9
> shortly.

I just built from Revision 1101 and got:

An error occurred: No such file or directory (2)
   in rev__action: The directory structure was changed; the path
./usr/share/bug/exim4/script is no longer accessible.

Shouldn't recursive revert create that path then?

Best Regards,
Simon

Re: Recursive Revert

Author pmarek
Full name P.Marek
Date 2007-09-03 23:31:40 PDT
Message Hello Simon!
> I was testing fsvs revert -R on a Virtual Machine and ran into problems:
> I versioned a machine's / and wanted to deploy the backup of this
> machine to a Virtual Machine.
>
> When doing "fsvs revert -R /" with 1.1.7 on the VM, my process got
> killed after running for an hour and reverting files. I assume because
> it was eating all the memory of the VM, because memory usage was
> constantly growing.
I'll take a look at that.

> So I tried 1.1.8 and got:
> 15:56:00.790 ops__traverse[est_ops.c:1129] INTERNAL BUG
> !path[0]
>
> sh: line 1: 7061 Segmentation fault sudo fsvs revert -R /
>
> It didn't even start reverting anything. Are you aware of this?
Such a bug is already fixed in the current version; I'll release 1.1.9
shortly.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Recursive Revert

Author =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Full name =?UTF-8?B?U2ltb24gU3Byw7xua2Vy?= <fsvs at semjon dot de>
Date 2007-09-03 15:41:30 PDT
Message Hi Phil,

I was testing fsvs revert -R on a Virtual Machine and ran into problems:
I versioned a machine's / and wanted to deploy the backup of this
machine to a Virtual Machine.

When doing "fsvs revert -R /" with 1.1.7 on the VM, my process got
killed after running for an hour and reverting files. I assume because
it was eating all the memory of the VM, because memory usage was
constantly growing.

So I tried 1.1.8 and got:
15:56:00.790 ops__traverse[est_ops.c:1129] INTERNAL BUG
   !path[0]

sh: line 1: 7061 Segmentation fault sudo fsvs revert -R /

It didn't even start reverting anything. Are you aware of this?

Best Regards,
Simon
Messages per page: