Login | Register
My pages Projects Community openCollabNet

Discussions > dev > Re: migration script 1.2

fsvs
Discussion topic

Hide all messages in topic

All messages in topic

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2009-02-09 09:25:50 PST
Message
Attachments

Re: migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2009-02-09 04:27:56 PST
Message Ok, thanks for feedback. Sorry your waiting for script, will push that
one forward.
migration.sh did not changed, will look for the python script for now
because i find the getopt in sh way hard. also implementing something
like progress bar is *arghh*

> I took a look ... but all I can offer is "fine indentation, nice colors" ;-/
>
It's ok, seems you dont use strict in perl either ;-)

Will look to have the script ready till end of next week.

Cheers,
Gunnar

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2009-02-08 05:08:02 PST
Message
Attachments

Re: migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2009-02-08 04:03:36 PST
Message Hi Phil,

very busy at the moment. I have asked my uncle to make an approach in
python based on my shell script and he has something ready that needs only
some fixing in naming, integrating the lib into main file ( have to go
through that with him as soon as I can ) and integrating the fsvs
relevant functions.
I am also more known to python than perl hope thats ok with you? Do you
have some kind of release plan for new version?

Current state of script is here: http://thielebein.ne​t/trac/browser/fsvs

Best,
Gunnar


P.Marek wrote:
> Hello Gunnar!
>
> How are you? Everything all right?
>
> I just wanted to take a look whether something happened to you, as I haven't
> heard from you for so long.
>
>
> On Friday 09 January 2009 P.Marek wrote:
>
>> Regarding the performance problem above ... do you know perl? I think it
>> would get *much* faster this way - just put all existing files into some
>> hash (possibly by using FSVS to get the list), and then running through the
>> reported entries (from "fsvs st -v") should be fast enough ...
>>
> I'm sorry if that question causes a lot of work to you. That was not my
> intention.
>
>
>
>> BTW - do you have some place for versioning this script, or if you don't,
>> how about the FSVS repository? I could give you commit access, if you like.
>>
> That offer still exists.
>
>
>
>> Thank you very much for your effort!
>>
> And that's more true than ever.
>
>
> Regards,
>
> Phil
>
>
>

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2009-02-08 01:05:41 PST
Message
Attachments

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2009-01-09 09:46:27 PST
Message
Attachments

Re: migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2009-01-08 09:23:04 PST
Message Hi Phil,

First I hope you had a good start in 2009, did you?
Regarding "reverse search" (looking for md5 of every file in fs) as in
the last script, should I revert that? It consumes much more time this
way and its no problem for replacing that to the use of fsvs st -v, I
only tried doing this way because of sorting files to the belonging WC.
That it's not possible by design to have underlaying $WAA i wasnt aware
off. When using "fsvs status -v $WC" will this work - new version of
fsvs on old WAA style?

Best Rgds,
Gunnar

Philipp Marek wrote:
> Hello Gunnar!
>
> On Saturday 13 December 2008 Gunnar Thielebein wrote:
>
>> good to know that the script works in some way like you want it to.
>> Regarding DEBUG output thats a good idea and I implemented that first.
>>
> Fube,
>
>
>> For compatibility I would like to keep the script sh conform.
>>
> Ok.
>
>
>> Can you say what needs to be done for rewriting ignore patterns.
>>
> The lowercase letters at the start must be substituted by
> the keywords; I have attached the current documentation for the patterns,
> please look at "Modifiers".
>
>
>> Regarding the PREFIX_ONE / PREFIX_TWO i used this, because first
>> approach was of moving PREFIX_ONE directory. This is now removed, as
>> i've rewritten some parts.
>>
> Ok.
>
>
>> What I do now is to check all files/dirs in wc-path to dir-structure in
>> FSVS_WAA. Before I do a reverse sort so that deepest path in tree is
>> taken first.
>> So i can keep sure that files are not moved to the wrong (parental)
>> working copy. I also added some exclude directory handling for speed-up
>> the search.
>>
> I don't think that's necessary - as soon as two working copies are nested, and
> have the same file versioned, the WAA paths will have collided - and then
> there's already a problem.
>
> So, if you find two WCs where one is above another, and files are shared, you
> could print a warning. (But that's low priority, lets get the other things
> working.)
>
>
>> For some things like animation :-) and a better option handling I need a
>> little more time. Its only a draftstate for me to get sure this is right
>> direction, and my experience is still script-kiddie ;-) .
>> Although many thanks for honoring the work.
>>
> Well, it seems "good enough" :-)
> And I'm happy if I can offload some work, so that I can blame others, too ;-)
>
>
> Many thanks again!
>
>
> Regards,
>
> Phil
>
>

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2008-12-13 11:12:16 PST
Message Hello Gunnar!

On Saturday 13 December 2008 Gunnar Thielebein wrote:
> good to know that the script works in some way like you want it to.
> Regarding DEBUG output thats a good idea and I implemented that first.
Fube,

> For compatibility I would like to keep the script sh conform.
Ok.

> Can you say what needs to be done for rewriting ignore patterns.
The lowercase letters at the start must be substituted by
the keywords; I have attached the current documentation for the patterns,
please look at "Modifiers".

> Regarding the PREFIX_ONE / PREFIX_TWO i used this, because first
> approach was of moving PREFIX_ONE directory. This is now removed, as
> i've rewritten some parts.
Ok.

> What I do now is to check all files/dirs in wc-path to dir-structure in
> FSVS_WAA. Before I do a reverse sort so that deepest path in tree is
> taken first.
> So i can keep sure that files are not moved to the wrong (parental)
> working copy. I also added some exclude directory handling for speed-up
> the search.
I don't think that's necessary - as soon as two working copies are nested, and
have the same file versioned, the WAA paths will have collided - and then
there's already a problem.

So, if you find two WCs where one is above another, and files are shared, you
could print a warning. (But that's low priority, lets get the other things
working.)

> For some things like animation :-) and a better option handling I need a
> little more time. Its only a draftstate for me to get sure this is right
> direction, and my experience is still script-kiddie ;-) .
> Although many thanks for honoring the work.
Well, it seems "good enough" :-)
And I'm happy if I can offload some work, so that I can blame others, too ;-)


Many thanks again!


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!
Attachments

Re: migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2008-12-13 05:30:29 PST
Message Hi Phil, good to know that the script works in some way like you want it to. Regarding DEBUG output thats a good idea and I implemented that first. For compatibility I would like to keep the script sh conform. Can you say what needs to be done for rewriting ignore patterns. Regarding the PREFIX_ONE / PREFIX_TWO i used this, because first approach was of moving PREFIX_ONE directory. This is now removed, as i've rewritten some parts. What I do now is to check all files/dirs in wc-path to dir-structure in FSVS_WAA. Before I do a reverse sort so that deepest path in tree is taken first. So i can keep sure that files are not moved to the wrong (parental) working copy. I also added some exclude directory handling for speed-up the search. For some things like animation :-) and a better option handling I need a little more time. Its only a draftstate for me to get sure this is right direction, and my experience is still script-kiddie ;-) . Although many thanks for honoring the work. Rgds, Gunnar script is now in svn: svn http://thielebein.net/repo/fsvs/migration.sh http://thielebein.net/trac/browser/fsvs/migration.sh P.Marek wrote: > I took a quick look, and ran it on my system - and it seems to work fine. > > Here are two patches for the cleanup part, and renaming the URLs file - both > untested. > > What's left to do is the rewriting of the ignore patterns. > > And I'd propose changing your progress messages to have a "# " in front, > the "echo " statements to "$DEBUG ", and setting $DEBUG to "echo" > or "", depending on some commandline parameter. > Then people can take a look what would happen, and possibly even redirect the > output into a file to be used with "bash " directly. > > Furthermore I'd suggest to change the PREFIX_ calculation to something like > this: > # Only the path below $FSVS_WAA > find $FSVS_WAA -type f -printf "%P\n" | > # Do streaming, as this could be several thousand files. > while IFS=/ read PREFIX_ONE PREFIX_TWO SUB_PATH > do > j=$FSVS_WAA/$PREFIX_ONE/$PREFIX_TWO/$SUB_PATH > > although I have to admit I don't quite understand why PREFIX_ONE etc. are > really used - you do a "mkdir -p", so the levels in the middle are created > anyway. > If you don't need them, you could replace that with > find ... | while read j > > > If it gets a "help" or "usage" parameter (which prints some text), and > possibly a "doitreally" (which sets DEBUG to "") ... that would make it > nearly perfect, only a fine ASCII-art animation could be displayed while > doing the work ;-) > > Nice work; good that you're including some documentation inline. > > > Gunnar, *thank you very much*. You'll receive (at least) a honorable mention; > maybe you'll even have a free wish (feature-wise) - and for the moment I'll > just assume that you'd like http/https authentication running, so I'll try to > get that working (although I have no real test environment). > > > Regards, > > Phil > >

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2008-12-12 11:10:20 PST
Message Finger quicker than the brain ...
(but that's not *that* hard.)


--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!
Attachments

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2008-12-12 11:09:11 PST
Message I took a quick look, and ran it on my system - and it seems to work fine. Here are two patches for the cleanup part, and renaming the URLs file - both untested. What's left to do is the rewriting of the ignore patterns. And I'd propose changing your progress messages to have a "# " in front, the "echo " statements to "$DEBUG ", and setting $DEBUG to "echo" or "", depending on some commandline parameter. Then people can take a look what would happen, and possibly even redirect the output into a file to be used with "bash " directly. Furthermore I'd suggest to change the PREFIX_ calculation to something like this: # Only the path below $FSVS_WAA find $FSVS_WAA -type f -printf "%P\n" | # Do streaming, as this could be several thousand files. while IFS=/ read PREFIX_ONE PREFIX_TWO SUB_PATH do j=$FSVS_WAA/$PREFIX_ONE/$PREFIX_TWO/$SUB_PATH although I have to admit I don't quite understand why PREFIX_ONE etc. are really used - you do a "mkdir -p", so the levels in the middle are created anyway. If you don't need them, you could replace that with find ... | while read j If it gets a "help" or "usage" parameter (which prints some text), and possibly a "doitreally" (which sets DEBUG to "") ... that would make it nearly perfect, only a fine ASCII-art animation could be displayed while doing the work ;-) Nice work; good that you're including some documentation inline. Gunnar, *thank you very much*. You'll receive (at least) a honorable mention; maybe you'll even have a free wish (feature-wise) - and for the moment I'll just assume that you'd like http/https authentication running, so I'll try to get that working (although I have no real test environment). Regards, Phil -- Versioning your /etc, /home or even your whole installation? Try fsvs (fsvs.tigris.org)!

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2008-12-12 09:22:31 PST
Message Hello Gunnar!

On Thursday 11 December 2008 Gunnar Thielebein wrote:
> one question.
I see at least two :-)

> When I have a folder like in your example:
> > No - the next version will have a per-WC prefix in the WAA structure, so
> > it will be
> > /var/spool/6666/66/66/..../
> > Other files will be like this:
> > MD5(/etc/services) = cdd8f7859d9a2306f50d​599fd2a26edf
> > /var/spool/6666/cd/d​8/f7859d9a2306f50d59​9fd2a26edf/
>
> Do i need too check if cdd8f7859d9a2306f50d​599fd2a26edf (/etc/services)
> is an folder under control of the working copy
> 6666cd76f96956469e7b​e39d750cc710 (/) ? If yes, I dont know how to do
> determine pathname of e.g. cd/d8/f7859d9a2306f5​0d599fd2a26edf .
You can't - directly.
That's why I said to do a "status -v $WC", so you'll get all (known) entries
of the WC; then you "simply" (;-) do an MD5 of all of them, so you'll see
which path belongs to which MD5 hash.

> Do all relevant files (also for subfolders) lay in the WAA_PATH of
> current WC? If yes ignore previous question.
Yes; but if there are two WCs on a machine, the files should be sorted to the
corresponding WC-specific WAA-paths.

> For my approach I have left this out. Also left out usage of fsvs.
> Have a look at the script, I have echoed all relevant parts out so this
> is save to try. I left out cleanup this should be another step after
> migration is finished.
I'll take a look.

Thank you very much in advance!


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2008-12-11 10:22:07 PST
Message Phil,
one question.
When I have a folder like in your example:

> No - the next version will have a per-WC prefix in the WAA structure, so it
> will be
> /var/spool/6666/66/66/..../
> Other files will be like this:
> MD5(/etc/services) = cdd8f7859d9a2306f50d​599fd2a26edf
> /var/spool/6666/cd/d​8/f7859d9a2306f50d59​9fd2a26edf/

Do i need too check if cdd8f7859d9a2306f50d​599fd2a26edf (/etc/services)
is an folder under control of the working copy
6666cd76f96956469e7b​e39d750cc710 (/) ? If yes, I dont know how to do
determine pathname of e.g. cd/d8/f7859d9a2306f5​0d599fd2a26edf .

Do all relevant files (also for subfolders) lay in the WAA_PATH of
current WC? If yes ignore previous question.

For my approach I have left this out. Also left out usage of fsvs.
Have a look at the script, I have echoed all relevant parts out so this
is save to try. I left out cleanup this should be another step after
migration is finished.

Rgds,
Gunnar

> Philipp Marek wrote:
>> Hello Gunnar,
>>
>> I saw that there are some problems for mails with attachments to tigris' mailings lists.
>> Had your original mail some version of your script attached, to get feedback?
>> If yes, then please resend - the list (and I) got no attachment.
>>
>> Regards,
>>
>> Phil
>>
>>
> Hi Phil,
>
> unfortunately not yet, the things asked I did need for understanding the
> goal and looking for a free hour now.
>
> Cheers,
> Gunnar
>
>
>
Attachments

Re: migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2008-12-11 02:34:13 PST
Message Philipp Marek wrote:
> Hello Gunnar,
>
> I saw that there are some problems for mails with attachments to tigris' mailings lists.
> Had your original mail some version of your script attached, to get feedback?
> If yes, then please resend - the list (and I) got no attachment.
>
> Regards,
>
> Phil
>
>
Hi Phil,

unfortunately not yet, the things asked I did need for understanding the
goal and looking for a free hour now.

Cheers,
Gunnar

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2008-12-10 23:32:20 PST
Message Hello Gunnar,

I saw that there are some problems for mails with attachments to tigris' mailings lists.
Had your original mail some version of your script attached, to get feedback?
If yes, then please resend - the list (and I) got no attachment.

Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

Re: migration script 1.2

Author pmarek
Full name P.Marek
Date 2008-12-08 05:38:30 PST
Message Hello Gunnar!

On Monday 08 December 2008 Gunnar Thielebein wrote:
> some questions about the script.
>
> When looking in /var/spool/fsvs/dd i see a lot of folder labeled with
> 2digixt hexcode that if understand correctly represents dirs in filesystem.
Not exactly.
The WAA names are MD5(path), but with two directory levels - like you write
below.

If it's just in the "dd" folder, then you got lucky :-)

> In /etc/fsvs there is one folder present which is configuration folder
> of the path i am monitoring with fsvs (for me this is root).
> First task is now to move "Urls" out of all pathes e.g. like
> /etc/fsvs/6666cd76f9​6956469e7be39d750cc7​d9/ and into
> /var/spool/66/66/cd7​6f96956469e7be39d750​cc7d9, correct?
No - the next version will have a per-WC prefix in the WAA structure, so it
will be
  /var/spool/6666/66/66/..../
Other files will be like this:
  MD5(/etc/services) = cdd8f7859d9a2306f50d​599fd2a26edf
  /var/spool/6666/cd/d​8/f7859d9a2306f50d59​9fd2a26edf/


> This pathes I
> have here in example are from debian packges dir-structures and may differ.
> Then you want to clean up WAA? I dont understand what you mean with
> cleanup, can you specify?
> Some other comments in org. post.
Older FSVS versions created too many directories in the WAA structure, and
didn't remove them cleanly in all cases.
So there might be empty directory structures which could get cleaned up.

> > Well, then please see whether you get something up and running.
> >
> > Short description:
> > - The $FSVS_WAA has an additional level, that's determined by a few (per
> > default 4) characters of the MD5 of the WC path.
> > I'd suggest just running "fsvs info <wc>" to get the correct paths.
> > Extra points if you're using the $FSVS_CONF/*/_base links to do that
> > for all used WCs on the machine.
>
> I see no other way than the $FSVS_CONF/*/_base approach, please correct me.
I meant to look for the _base links, and get the correct WAA path by looking
at the output of "fsvs info <wc-path>" - that would show whether the binary
was compiled with the default of 4 characters for the WC-distinction in the
WAA area, or maybe some other value.


> > - The URLs file has to be moved to the WAA
> > - And the WAA has either to be cleaned up, or the data moved correctly.
> > Please see that at least the "dir" files is moved; although the
> > properties
>
> Can you specify "dir" file? Where to move?
All of the files in the WAA have to be moved to the "new" specific
WAA-directories, ie. the ones with the WC-path-prefix in them.

Like your example above - the first 4 characters of MD5(/) = 6666, so it would
be
  /var/spool/6666/ + structure in original WAA.

> > would be nice - else a "fsvs sync-repos" run would be necessary to
> > avoid the loss of the properties.
>
> fsvs sync-repo - before we start migration?
> Should all fsvs relevant tasks be done with the new version of fsvs or
> should migration be done in a pre-upgrade step?
> I mean, does the fsvs 1.2.x own the same syntax or should this be used
> for testing script. I am still with 1.1.16 atm.
All operations should be done with the new version.
If we could just move the "dir" file, we wouldn't need the "sync-repos" - then
the user wouldn't loose all local user changes like "add", "prop-set" etc.

> > Maybe looking for all "prop" files in the WAA, and correlating them to
> > the working copy (by getting a list of all known entries by "fsvs st -v")
> > would be best.
> > The "md5s" and "cflct" files would be nice, too, but are not as
> > important.
>
> I cant find "prop" files. In which step did they get created? Where
> should they be moved too?
They get created if the user defines additional, user-specific properties.
Like the fsvs:commit-pipe.


Thank you *very much* for your effort and the time you're putting into this!


Regards,

Phil

--
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

migration script 1.2

Author tekknokra
Full name Gunnar Thielebein
Date 2008-12-08 05:10:06 PST
Message Hi Phil,

some questions about the script.
When looking in /var/spool/fsvs/dd i see a lot of folder labeled with
2digixt hexcode that if understand correctly represents dirs in filesystem.
In /etc/fsvs there is one folder present which is configuration folder
of the path i am monitoring with fsvs (for me this is root).
First task is now to move "Urls" out of all pathes e.g. like
/etc/fsvs/6666cd76f9​6956469e7be39d750cc7​d9/ and into
/var/spool/66/66/cd7​6f96956469e7be39d750​cc7d9, correct? This pathes I
have here in example are from debian packges dir-structures and may differ.
Then you want to clean up WAA? I dont understand what you mean with
cleanup, can you specify?
Some other comments in org. post.

> Well, then please see whether you get something up and running.
>
> Short description:
> - The $FSVS_WAA has an additional level, that's determined by a few (per
> default 4) characters of the MD5 of the WC path.
> I'd suggest just running "fsvs info <wc>" to get the correct paths.
> Extra points if you're using the $FSVS_CONF/*/_base links to do that
> for all used WCs on the machine.
>
I see no other way than the $FSVS_CONF/*/_base approach, please correct me.
> - The URLs file has to be moved to the WAA
> - And the WAA has either to be cleaned up, or the data moved correctly.
> Please see that at least the "dir" files is moved; although the properties
>
Can you specify "dir" file? Where to move?

> would be nice - else a "fsvs sync-repos" run would be necessary to avoid
> the loss of the properties.
>

fsvs sync-repo - before we start migration?
Should all fsvs relevant tasks be done with the new version of fsvs or
should migration be done in a pre-upgrade step?
I mean, does the fsvs 1.2.x own the same syntax or should this be used
for testing script. I am still with 1.1.16 atm.
> Maybe looking for all "prop" files in the WAA, and correlating them to the
> working copy (by getting a list of all known entries by "fsvs st -v") would
> be best.
> The "md5s" and "cflct" files would be nice, too, but are not as important.
>
I cant find "prop" files. In which step did they get created? Where
should they be moved too?
> That all with a nice short README, disclaimer, and your email addresses (for
> bug reports :-) , and I'd be *very* happy.
>
>
> Don't hesitate to ask if there are any more questions.
> If you like you're welcome to use the dev@ list - so that others (including
> me) see some progress, and can help if there are any questions.
>
>
> Thank you *very* *much*!
>
>
> Regards,
>
> Phil
Best Regds,
Gunnar
Messages per page: