Login | Register
My pages Projects Community openCollabNet

Discussions > users > [ANNOUNCE] FSVS 1.2.0 released

fsvs
Discussion topic

Hide all messages in topic

All messages in topic

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 23:44:11 PDT
Message >> I presume if I want to use the new md5 namespace mode
>> (--with-waa_md5=N) I would have to
> --with-waa_chars=N
My fingers too fast again ...

You're right, it's

  --with-waa_md5=NUMBER Specifies how many hex characters of the MD5 of the
                          working copy root should be used to address the data
                          in the WAA. This may be increased if you have a lot
                          of different working copies on a single machine. The
                          default is 0; useful values are 0, and from 6 to 32.


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 23:43:01 PDT
Message Hello Mark!


> Well, I'm pleased to see you write the same type of bug as
> I do - enough effect to panic
> the user and a tiny change required to correct the problem!
> That fixes my issue; I hope it is useful to others. Thanks!
You're welcome.
Sadly I seem to have tested on my installation *before* the
URL-in-WAA change ...
I should have found that, as that breaks most of the announced
compatibility!


> I presume if I want to use the new md5 namespace mode
> (--with-waa_md5=N) I would have to
--with-waa_chars=N
> remove my existing WAA areas and rebuild them with sync-repos?
Yes.

But why would you do that? Do you have nested working copies,
or a lot of them?


Regards,

Phil


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

RE: [ANNOUNCE] FSVS 1.2.0 released

Author elsabio
Full name mark j hewitt
Date 2009-07-09 07:01:10 PDT
Message Phil,

Well, I'm pleased to see you write the same type of bug as I do - enough effect to panic the user and a tiny change required to correct the problem!

That fixes my issue; I hope it is useful to others. Thanks!

I presume if I want to use the new md5 namespace mode (--with-waa_md5=N) I would have to remove my existing WAA areas and rebuild them with sync-repos?

Thank you again!

Mark

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 06:28:35 PDT
Message Hello Mark!

...
>> Please send me the output of
>> sudo /usr/local/bin/fsvs -v -V
>
> [mjh@tardis backup]$ sudo /usr/local/bin/fsvs -v -V
> FSVS (licensed under the GPLv3), (C) by Ph. Marek; version fsvs-1.2.0:2282
> compiled on Jul 8 2009 16:05:00, with options:
> ... WAA_WC_MD5_CHARS=0
Fine, this is the compatibility mode.


> [mjh@tardis backup]$ sudo /usr/bin/fsvs -o waa=/backup/fsvs info .
...
> WAA-Path: /backup/fsvs/8b/21/1​1f9c8152001cc8eed28d​fa77907/
> Conf-Path: /etc/fsvs/8b2111f9c8​152001cc8eed28dfa779​07/

> You have new mail in /var/spool/mail/mjh
;-)


> [mjh@tardis backup]$ sudo /usr/local/bin/fsvs -o waa=/backup/fsvs info .
...
> WAA-Path: /backup/fsvs/8b/21/1​1f9c8152001cc8eed28d​fa77907/
> Conf-Path: /etc/fsvs/8b2111f9c8​152001cc8eed28dfa779​07/
These are identical, good.

>> sudo /usr/local/bin/fsvs urls dump -d
...
> 13:55:14.252 url__load_list[url.c:879] No file; upgrading?
> No URLs defined for "/backup".
Here we are.

Please try the attached patch.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!
Attachments

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 05:17:30 PDT
Message Hello Mark!

> Thanks for rapid answer.
> My comments were mainly addressed to the "migrate.py" script
> and the notes in the announcement regarding the
> changes to the WAA areas.
The storing of the current revision per-URL is done silently;
as soon as you do "update" or "commit", the new file in WAA is
written (and used later on).

Only if you change the URL list the file in CONF is changed.


> I do have a problem though...
>
> New version (1.2.0)
> [mjh@tardis backup]$ sudo /usr/local/bin/fsvs urls dump
> No URLs defined for "/backup".
Did you compile 1.2.0 yourself? Did you use the --with-waa_chars option?
Please send me the output of
    sudo /usr/local/bin/fsvs -v -V

> But I read that the URLs were now (partially) kept with the WAA:
>
> [mjh@tardis backup]$ sudo /usr/local/bin/fsvs -o waa=/backup/fsvs urls dump
> No URLs defined for "/backup".
>
> So how do these get migrated?
The URLs should be done completely transparently.

> Old version (1.1.17)
> [mjh@tardis backup]$ sudo /usr/bin/fsvs urls dump
> N:,P:0,svn://tardis/​media/Elements-01/fs​vsdata/tardis/backup​/trunk
I don't know off-hand why FSVS doesn't find the working copies.

Would you please send me the output of
    sudo /usr/bin/fsvs info .
    sudo /usr/local/bin/fsvs info .
    sudo /usr/local/bin/fsvs urls dump -d


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 05:11:18 PDT
Message ...
> This is exactly what you wrote before ... didn't my earlier message help you?
Sorry, I answered the same mail again, because I got it from the mailing list much later.

Sorry.

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 05:10:09 PDT
Message Hello Mark!


> So what is unclear to me is if there is an upgrade
> process to go from 1.1.17 to 1.2.0.
Yes, there is.

> There are some scripts in the distribution, but no notes
> as far as I can see on when and how to use them.
You may be right; there's nothing in the distribution.
I wrote all of that in the announcement.

> In fact, not only do they look incomplete, they have hardcoded paths
> which would not be appropriate for me anyway.
>
> I'm always keen to try new versions, but clearly I don't want to break my current
> versioned data.
This is exactly what you wrote before ... didn't my earlier message help you?


Regards,

Phil


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

RE: [ANNOUNCE] FSVS 1.2.0 released

Author elsabio
Full name mark j hewitt
Date 2009-07-09 04:53:58 PDT
Message Hi Phil!

Thanks for rapid answer. My comments were mainly addressed to the "migrate.py" script and the notes in the announcement regarding the changes to the WAA areas.

I do have a problem though...

New version (1.2.0)
[mjh@tardis backup]$ sudo /usr/local/bin/fsvs urls dump
No URLs defined for "/backup".

But I read that the URLs were now (partially) kept with the WAA:

[mjh@tardis backup]$ sudo /usr/local/bin/fsvs -o waa=/backup/fsvs urls dump
No URLs defined for "/backup".

So how do these get migrated?

Old version (1.1.17)
[mjh@tardis backup]$ sudo /usr/bin/fsvs urls dump
N:,P:0,svn://tardis/​media/Elements-01/fs​vsdata/tardis/backup​/trunk

RE: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-07-09 04:20:21 PDT
Message Hello Mark!

> So what is unclear to me is if there is an upgrade process
> to go from 1.1.17 to 1.2.0.
>
> There are some scripts in the distribution, but no notes as
> far as I can see on when and how to use them. In fact, not
> only do they look incomplete, they have hardcoded paths
> which would not be appropriate for me anyway.

From the announcement, about the ignore patterns:
> The distribution includes a perl script to convert to the
> new version; the easiest way might be to do
> $FSVS_SRC/pre-1.2-m​igration/convert-ign​ores.pl /etc/fsvs/*/Ign
> and you're done, at least if you're using the default $FSVS_CONF.

What that means is that if you're using a different FSVS_CONF,
you should just run
  $FSVS_SRC/pre-1.2-m​igration/convert-ign​ores.pl $FSVS_CONF/*/Ign

In which ways do they look incomplete?
Maybe the other files in that directory, that I've used for testing,
serve only to confuse people ... Sorry about that.

Would you mind elaborating a bit?

> I'm always keen to try new versions, but clearly I don't want
> to break my current versioned data.
Of course.
Well, the repository can't be broken from the client side anyway,
and if you hose your local FSVS data the worst that can happen is
that you have to remove FSVS_CONF and FSVS_WAA, re-set the URLs,
and do a "fsvs sync-repos".


I sure hope that it "just works", but all-in-all is should be low-risk.


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

RE: [ANNOUNCE] FSVS 1.2.0 released

Author elsabio
Full name mark j hewitt
Date 2009-07-09 01:15:27 PDT
Message So what is unclear to me is if there is an upgrade process to go from 1.1.17 to 1.2.0.

There are some scripts in the distribution, but no notes as far as I can see on when and how to use them. In fact, not only do they look incomplete, they have hardcoded paths which would not be appropriate for me anyway.

I'm always keen to try new versions, but clearly I don't want to break my current versioned data.

Re: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-06-22 10:47:32 PDT
Message
Attachments

Re: [ANNOUNCE] FSVS 1.2.0 released

Author Peter Rabbitson <rabbit+list at rabbit dot us>
Full name Peter Rabbitson <rabbit+list at rabbit dot us>
Date 2009-06-22 05:35:18 PDT
Message Philipp Marek wrote:
>>> [1] First problem at
>>> http://www.nabble.co​m/Congratulations-fo​r-fsvs-1.1.17..-but-​here-are-my-first-pr​oblems-%3A)-td202309​99.html
>> I'll take a look.
> Is it that?
>
> An error occurred: Name does not refer to a filesystem directory (160016)
> in ci__directory: editor->close_file: Path 'sysconfig' not present
>
>

No, the fact that the config dir is ignored (this started in 1.1.17). Thus I
can't have differing logins for the two repositories.

Re: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-06-22 05:28:14 PDT
Message >> [1] First problem at
>> http://www.nabble.co​m/Congratulations-fo​r-fsvs-1.1.17..-but-​here-are-my-first-pr​oblems-%3A)-td202309​99.html
> I'll take a look.
Is it that?

  An error occurred: Name does not refer to a filesystem directory (160016)
  in ci__directory: editor->close_file: Path 'sysconfig' not present


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-06-22 05:15:44 PDT
Message Hello Peter!

Peter Rabbitson wrote:
> I have a setup where I backup /etc to a central repository, but backup
> /var/www to a local repository. I am actually running 1.1.16, because
> 1.1.17 does not support authentication properly[1]. Does this mean I can
> not do this anymore with 1.2.x ? What are my alternatives?
No, this is no problem.
What I tried to explain was
- if you're having multiple working copies, AND
- they're overlapping, AND
- you have files versioned in both,
THEN you might get a problem.

So, for example:
If "/" and "/etc" are two working copies, and "/etc" is not ignored for the "/", then
eg. property changes for "/etc" + "passwd" would be shared by "/etc/passwd" in the other
working copy.

In your example the paths are completely distinct, so there's no problem.


> Also in the initial announcement you say:
>
>> The delay can be explained by it being *incompatible* with the older
>> versions
>
> What exactly does this mean? Can I attempt to upgrade at some point or...?
The ignore patterns format got changed, to accomodate the grouping functionality.
Yes, you can. The distribution includes a perl script to change the files in-place;
please look at the announcement.

> [1] First problem at
> http://www.nabble.co​m/Congratulations-fo​r-fsvs-1.1.17..-but-​here-are-my-first-pr​oblems-%3A)-td202309​99.html
I'll take a look.
Thanks for the reminder!


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: [ANNOUNCE] FSVS 1.2.0 released

Author Peter Rabbitson <rabbit+list at rabbit dot us>
Full name Peter Rabbitson <rabbit+list at rabbit dot us>
Date 2009-06-22 04:31:43 PDT
Message P.Marek wrote:
> Hello Sheldon!
>
> Nice to hear from you.
>
>> On Thu, 2009-06-18 at 19:56 +0200, P.Marek wrote:
>>> The default value is 0, which is compatible with earlier versions, as
>>> my suggestion for nested WCs is to avoid them, or at least ignore the
>>> shared paths.
>> To clarify:
>>
>> Distribution packagers can ignore the working copy directory option,
>> because it is not the default.
> Yes, please.
>
> If you feel that there's a real need you could provide a package with that option set to
> 8 ... but I don't think that's necessary.
>
> I think most people just keep /etc or / versioned; and using multiple overlayed URLs
> does/will work much better than having multiple working copies.
>
> Or does someone backup each /home/* directory as its own working copy?
>

I have a setup where I backup /etc to a central repository, but backup
/var/www to a local repository. I am actually running 1.1.16, because
1.1.17 does not support authentication properly[1]. Does this mean I can
not do this anymore with 1.2.x ? What are my alternatives?

Also in the initial announcement you say:

> The delay can be explained by it being *incompatible* with the older
> versions

What exactly does this mean? Can I attempt to upgrade at some point or...?

[1] First problem at http://www.nabble.co​m/Congratulations-fo​r-fsvs-1.1.17..-but-​here-are-my-first-pr​oblems-%3A)-td202309​99.html

Re: [ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-06-22 02:36:22 PDT
Message Hello Sheldon!

Nice to hear from you.

> On Thu, 2009-06-18 at 19:56 +0200, P.Marek wrote:
>> The default value is 0, which is compatible with earlier versions, as
>> my suggestion for nested WCs is to avoid them, or at least ignore the
>> shared paths.
>
> To clarify:
>
> Distribution packagers can ignore the working copy directory option,
> because it is not the default.
Yes, please.

If you feel that there's a real need you could provide a package with that option set to
8 ... but I don't think that's necessary.

I think most people just keep /etc or / versioned; and using multiple overlayed URLs
does/will work much better than having multiple working copies.

Or does someone backup each /home/* directory as its own working copy?


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: [ANNOUNCE] FSVS 1.2.0 released

Author sheldonh
Full name Sheldon Hearn
Date 2009-06-22 00:56:49 PDT
Message On Thu, 2009-06-18 at 19:56 +0200, P.Marek wrote:
> The default value is 0, which is compatible with earlier versions, as
> my suggestion for nested WCs is to avoid them, or at least ignore the
> shared paths.

To clarify:

Distribution packagers can ignore the working copy directory option,
because it is not the default.

Yes?

Thanks,
Sheldon.

[ANNOUNCE] FSVS 1.2.0 released

Author pmarek
Full name P.Marek
Date 2009-06-18 10:57:00 PDT
Message Hello everybody!


After a long time I'm happy to release a new version.
The delay can be explained by it being incompatible with the older
versions; this is the cause for it being labeled for the new 1.2 series,
although the next big feature (mixed version working copies) is not
included yet.

For the impatient: see
   http://freshmeat.net​/projects/fsvs (once it's updated) and
   http://doc.fsvs-software.org/,
but don't complain afterwards ;-)



 About the incompatibilities
=============================

 WAA_CHARS
-----------

FSVS uses (the MD5 of) the absolute pathname as a key for storing internal
data - user-defined properties, a list of block-MD5s for faster change
detection, and things like that.

Now that has a problem if you're (very) unlucky and two of your paths hash
to the same MD5; or if you have nested working copies, eg. "/" and "/etc".
Here all files in the "/etc" working copy get the same internal key as the
ones below "etc" in the "/" working copy, and this will cause problems, as
the FSVS-managed meta-data gets partly changed from two working copies
simultaneously.

If you need this, there are several ways to do that:
* Ignore the shared paths in the upper WC
* Use different FSVS_WAA directories
And, exclusively with this newer version,
* use the "--with-waa_md5=N" switch on "configure".

This switch (with values for "N" from 6 to 32) inserts a WC-specific
directory in the WAA hierarchy, so that the absolute path MD5s do not
collide anymore.

This might also be useful if you need to create and remove WCs often (and
need to clean up the space), as there's no "wc disconnect" command [1] yet.

The default value is 0, which is compatible with earlier versions, as my
suggestion for nested WCs is to avoid them, or at least ignore the shared
paths.


Looking for a sane way to convert the old-format WAA to the newer one is
one of the things that delayed the new release, until I settled for the
common default value [2].


 Grouping patterns
-------------------

The ignore patterns been transformed to grouping patterns, and their format
has changed; it is a bit more verbose now, but much more readable this way.

The distribution includes a perl script to convert to the new version; the
easiest way might be to do

$FSVS_SRC/pre-1.2-m​igration/convert-ign​ores.pl /etc/fsvs/*/Ign
and you're done, at least if you're using the default $FSVS_CONF.

The advantage is that you can now use groups to give properties to entries
automatically when they're being committed; see
http://doc.fsvs-soft​ware.org/gif/groups.​html for details.
(This is like the auto-props in subversion.)

The "ignore" command still has the default for the "ignore" group, so the
command line interface doesn't change that much.


 URL list
----------

Some people (including me) found out that it's a bit unaesthetic that after
a commit in "/etc" FSVS would show the URL lists as changed.
They have been split in two; the user-defined, nearly constant values
still live in "/etc/fsvs/*/Urls" (but don't get written unless that's really
necessary), but the current revision number of the URLs is stored in the
WAA.

So you get the best of both worlds; you keep your list versioned, but
(normally) see no changes.

The code handling that is backward compatible; it can use the old lists
without any manual intervention.



 Getting the new release
=========================

You can get the sources (soon) from http://freshmeat.net​/projects/fsvs,
but quite a few distributions include packages, so that you don't have
to compile it yourself.

If you want to do that, see http://fsvs.tigris.o​rg/install.html for a bit
of help.


 Changes
=========

There are too many changes to show them all; for the complete list please
go to
http://fsvs.tigris.o​rg/source/browse/fsv​s/tags/fsvs-1.2.0/fs​vs/CHANGES?view=mark​up
and look through the >50 changelist entries.

Here I'll just present the most important points.


New features:
* An additional level in the WAA (optionally, on compile time), to make
  nested WCs non-overlapping.
* Using lchown()/lutimes() to set ownership for symlinks, too. Thank you
  for noticing, Nebojsa.
* Pattern "groups"; with the new subcommand "groups test" you can test
  patterns and group assignments, and with the option "group_stats" see
  pattern match frequencies.
* The read-only commands now only test for read access to CONF; so,
  depending on the permissions of the wc entries, other users than UID 0
  might be able to do "status", "prop-list", "prop-get" and "info", and (if
  the user has repository access) "diff", too.
* Splitted "-v" and "-q" into a more detailed "verbose" option, which can
  now show the group and URL on status, too.
* New option "debug_buffer" allows to capture debug information in RAM and
  only print it if an error happens.
* New option "empty_message", to avoid commits without a message.
* New option "mkdir_base", to create non-existing but needed directories
  in the repository above the given URL on commit.
* "fsvs add" (and "fsvs prop-set") can now store a destination URL, which
  is very useful in multi-URL setups.
* "fsvs log" can now report for other defined URLs.
* Updating to revision 0 should now work, and means to remove that URL.
* Readonly flag for URLs, so that no commit is allowed.

Performance:
* A "status" after test 019 is at about 41% cpu cycles compared to the
  old version (blame sscanf(); now calls strtoul()).
* An fsync() per property-less entry on commit avoided.
* Copied entries that are unchanged at commit time got no manber-hashes, so
  the entire file had to get read to detect a change.

Important fixes:
* Bugfix: Using a commit-pipe on an empty file would hang in a tight loop.
* Fixed a memleak in "checkout" (and "export"). This was needed to recover
  a 56GB installation [3]. Thank you, Omar.
* "revert" now works even if sockets or fifos are in the way.
* "urls load" now tries to keep the internal numbers by matching against an
  old URL list.

Visible behavior changes:
* Changed the default for the "conflict" option to "merge".
* "fsvs log -v" now sorts the filenames.
* Handling symlinks to directories on the commandline fixed,
  eg. for "fsvs st symlink-to-dir/file".
* URLs can be now modified by name, too, eg. to set the target revision.
* Changed the default URL dump format to something more readable.


Dear packagers, please create the $FSVS_CONF/auth hierarchy in your
packages, so that automatic password storage works.

Ideas, hints, tips & tricks, bug reports and all other feedback is welcome.

Does anybody have some hints whether SSL with client certificates works
currently?


Regards,

Phil



Ad 1:
http://fsvs.tigris.o​rg/ds/viewMessage.do​?dsForumId=3928&​dsMessageId=996569

Ad 2:
The mixed-WC parts need a SQLite database for tracking the revision
numbers; and then it's likely that I'll put some other data in the DB as
well, so that the multitude of WAA directories is likely to vanish, and be
replaced by a smaller list.
Using SQLite is not such a problem, because subversion 1.6 will require it,
too.

Ad 3:
http://article.gmane​.org/gmane.comp.sysu​tils.backup.fsvs.gen​eral/738
"Restore finished in 82 minutes with 56 GB of data restored."

--
Versioning your /etc, /home or even your whole installation?
           Try fsvs (fsvs.tigris.org)!
Messages per page: