Re: quick dpkg-scanpackages

[Ben Collins <bcollins@debian.org>]

On Thu, Apr 06, 2000 at 12:09:21PM -0600, Jason Gunthorpe wrote:
> 
> On Thu, 6 Apr 2000, Nick Cabatoff wrote:
> 
> > I hacked dpkg-scanpackages the next day to do this, and then realized
> > it was pointless.  The md5sums are what take by far the most time.
> > For a while I decided that was okay and just looked at file
> 
> The FTP archive actually uses a special version of dpkg-scanpackages that
> cooperates with dsync to cache MD5s. Performance is dominated by the time
> it takes to unpack the control tar.

Saving having to install dsync, you could just apply the attached patch
which uses Digest::MD5 instead of the md5sum program (make sure you have
libdigest-md5-perl installed). I tested it and got a 50% increase in
speed (which probably says more about the fact that the archive is nfs 
mounted than what the real speed could be).

(scanning 249 packages)
22.83 secs (old way)
15.89 secs (new way)

Ben

-- 
 -----------=======-=-======-=========-----------=====------------=-=------
/  Ben Collins  --  ...on that fantastic voyage...  --  Debian GNU/Linux   \
`     bcollins@debian.org  --  bcollins@openldap.org  --  bmc@visi.net     '
 `---=========------=======-------------=-=-----=-===-======-------=--=---'

--- /usr/bin/dpkg-scanpackages.dpkg~	Thu Apr  6 17:18:37 2000
+++ /usr/bin/dpkg-scanpackages.dpkg	Thu Apr  6 17:22:53 2000
@@ -1,5 +1,7 @@
 #! /usr/bin/perl
 
+use Digest::MD5;
+
 $version="1.6.11"; # This line modified by Makefile
 
 %kmap= ('optional','suggests',
@@ -82,10 +84,10 @@
     
     $tv{'Filename'}= "$pathprefix$fn";
 
-    open(C,"md5sum <$fn |") || die "$fn $!";
-    chop($_=<C>); close(C); $? and die "\`md5sum < $fn' exited with $?\n";
-    /^[0-9a-f]{32}$/ or die "Strange text from \`md5sum < $fn': \`$_'\n";
-    $tv{'MD5sum'}= $_;
+    open(C,"$fn") || die "$fn $!";
+    binmode(C);
+    $tv{'MD5sum'} = Digest::MD5->new->addfile(*C)->hexdigest;
+    close(C);
 
     @stat= stat($fn) or die "Couldn't stat $fn: $!\n";
     $stat[7] or die "$fn is empty\n";