Base revisions

Author: jgg
Date: 1998-07-02 02:58:12 GMT
Base revisions

1 files changed, 491 insertions, 0 deletions

diff --git a/doc/files.sgml b/doc/files.sgml
new file mode 100644
index 000000000..d98d6f68a
--- /dev/null
+++ b/doc/files.sgml
@@ -0,0 +1,491 @@
+<!doctype debiandoc system>
+<!-- -*- mode: sgml; mode: fold -*- -->
+<book>
+<title>APT Files</title>
+
+<author>Jason Gunthorpe <email>jgg@debian.org</email></author>
+<version>$Id: files.sgml,v 1.1 1998/07/02 02:58:12 jgg Exp $</version>
+
+<abstract>
+This document describes the complete implementation and format of the 
+installed APT directory structure. It also serves as guide to how APT 
+views the Debian archive.
+</abstract>
+
+<copyright>
+Copyright &copy; Jason Gunthorpe, 1998.
+<p>
+"APT" and this document are free software; you can redistribute them and/or
+modify them under the terms of the GNU General Public License as published
+by the Free Software Foundation; either version 2 of the License, or (at your
+option) any later version.
+
+<p>
+For more details, on Debian GNU/Linux systems, see the file
+/usr/doc/copyright/GPL for the full license.
+</copyright>
+
+<toc sect>
+
+<chapt>Introduction
+<!-- General		                                               {{{ -->
+<!-- ===================================================================== -->
+<sect>General
+
+<p>
+This document serves two purposes. The first is to document the installed
+directory structure and the format and purpose of each file. The second
+purpose is to document how APT views the Debian archive and deals with 
+multiple package files.
+
+<p>
+The var directory structure is as follows:
+<example>
+  /var/state/apt/
+		  lists/
+		       partial/
+		  xstatus
+  /var/cache/apt/
+                  pkgcache.bin
+		  srcpkgcache.bin
+		  archives/
+		          partial/
+  /etc/apt/
+           sources.list
+	   cdromdevs.list
+  /usr/lib/apt/
+                methods/
+		       cdrom
+		       ftp
+		       http
+</example>
+
+<p>
+As is specified in the FHS 2.0 /var/state/apt is used for application 
+data that is not expected to be user modified. /var/cache/apt is used
+for regeneratable data and is where the package cache and downloaded .debs
+go.
+</sect>
+                                                                  <!-- }}} -->
+
+<chapt>Files
+<!-- Distribution Source List					       {{{ -->
+<!-- ===================================================================== -->
+<sect>Distribution Source list (sources.list)
+
+<p>
+The distribution source list is used to locate archives of the debian
+distribution. It is designed to support any number of active sources and to
+support a mix of source media. The file lists one source per line, with the 
+fastest source listed first. The format of each line is:
+
+<p>
+<var>type ui args</var>
+
+<p>
+The first item, <var>type</var>, indicates the format for the remainder 
+of the line. It is designed to indicate the structure of the distribution
+the line is talking about. Currently the only defined value is <em>deb</em>
+which indicates a standard debian archive with a dists dir.
+
+<sect1>The deb Type
+   <p>
+   The <em>deb</em> type is to be a typical two level debian distributions,
+   dist/<var>distribution</var>/<var>component</var>. Typically distribution
+   is one of stable, unstable or frozen while component is one of main, 
+   contrib, non-free or non-us. The format for the deb line is as follows:
+
+   <p>
+   deb <var>uri</var> <var>distribution</var> <var>compontent</var> 
+   [<var>component</var> ...]
+
+   <p>
+   <var>uri</var> for the <em>deb</em> type must specify the base of the 
+   debian distribution. APT will automatically generate the proper longer 
+   URIs to get the information it needs. <var>distribution</var> can specify 
+   an exact path, in this case the components must be omitted and
+   <var>distribution</var> must end in a slash.
+   
+   <p>
+   Since only one distribution can be specified per deb line it may be
+   necessary to list a number of deb lines for the same URI. APT will
+   sort the URI list after it has generated a complete set to allow 
+   connection reuse. It is important to order things in the sourcelist
+   from most prefered to least prefered (fastest to slowest).
+</sect1>
+
+<sect1>URI specification
+<p> 
+URIs in the source list support a large number of access schemes.
+
+<taglist>
+<tag>cdrom<item>
+   The cdrom scheme is special in that If Modifed Since queries are never
+   performed and that APT knows how to match a cdrom to the name it
+   was given when first inserted. It does this by examining the date
+   and size of the package file. APT also knows all of the possible 
+   prefix paths for the cdrom drives and that the user should be prompted
+   to insert a CD if it cannot be found. The path is relative to an 
+   arbitary mount point (of APT's choosing) and must not start with a 
+   slash. The first pathname component is the given name and is purely
+   descriptive and of the users choice. However, if a file in the root of 
+   the cdrom is called 'cdname' its contents will be used instead of 
+   prompting. The name serves as a tag for the cdrom and should be unique.
+   APT will track the CDROM's based on their tag and package file
+   properties.
+   <example>
+   cdrom:Debian 1.3/debian
+   </example>
+
+<tag>http<item>
+   This scheme specifies a HTTP server for the debian archive. HTTP is prefered
+   over FTP because If Modified Since queries against the Package file are
+   possible. Newer HTTP protcols may even support reget which would make
+   http the protocol of choice.
+   <example>
+   http://www.debian.org/archive
+   </example>
+
+<tag>ftp<item>
+   This scheme specifies a FTP connection to the server. FTP is limited because
+   there is no support for IMS and is hard to proxy over firewalls.
+   <example>
+   ftp://ftp.debian.org/debian
+   </example>
+
+<tag>file<item>
+   The file scheme allows an arbitary directory in the file system to be 
+   considered as a debian archive. This is usefull for NFS mounts and 
+   local mirrors/archives.
+   <example>
+   file:/var/debian
+   </example>
+   
+<tag>mirror<item>
+   The mirror scheme is special in that it does not specify the location of a
+   debian archive but specifies the location of a list of mirrors to use
+   to access the archive. Some technique will be used to determine the
+   best choice for a mirror. The mirror file is specified in the Mirror File
+   section. If/when URIs take off they should obsolete this field.
+   <example>
+   mirror:http://www.debian.org/archivemirrors
+   </example>
+   
+<tag>smb<item>
+   A possible future expansion may be to have direct support for smb (Samba 
+   servers).
+   <example>
+   smb://ftp.kernel.org/pub/mirrors/debian
+   </example>
+</taglist>
+</sect1>
+
+<sect1>Hashing the URI
+<p>
+All permanent information aquired from any of the sources is stored in the
+lists directory. Thus, there must be a way to relate the filename in the
+lists directory to a line in the sourcelist. To simplify things this is
+done by quoting the URI and treating ='s as quoteable characters and
+converting / to =. The URI spec says this is done by converting a 
+sensitive character into %xx where xx is the hexadecimal representation 
+from the ascii character set. Examples:
+
+<example>
+http://www.debian.org/archive/dists/stable/binary-i386/Packages 
+/var/state/apt/lists/www.debian.org=archive=dists=stable=binary-i386=Packages
+
+cdrom:Debian 1.3/debian/Packages
+/var/state/apt/info/Debian%201.3=debian=Packages
+</example>
+
+<p> 
+The other alternative that was considered was to use a deep directory 
+structure but this poses two problems, it makes it very difficult to prune
+directories back when sources are no longer used and complicates the handling
+of the partial directory. This gives a very simple way to deal with all
+of the situations that can arise. The equals sign was choosen on the 
+suggestion of Manoj because it is very infrequently used in filenames.
+Also note that the same rules described in the <em>Archive Directory</>
+section regarding the partial sub dir apply here as well.
+</sect1>
+
+</sect>
+                                                                  <!-- }}} -->
+<!-- Extra Status						       {{{ -->
+<!-- ===================================================================== -->
+<sect>Extra Status File (xstatus)
+
+<p>
+The extra status file serves the same purpose as the normal dpkg status file 
+(/var/lib/dpkg/status) except that it stores information unique to diety.
+This includes the autoflag, target distribution and version and any other
+uniqe features that come up over time. It duplicates nothing from the normal
+dpkg status file.  Please see other APT documentation for a discussion
+of the exact internal behavior of these fields. The Package field is
+placed directly before the new fields to indicate which package they
+apply to. The new fields are as follows:
+
+<taglist>
+<tag>X-Auto<item>
+   The Auto flag can be Yes or No and controls whether the package is in
+   auto mode. 
+
+<tag>X-TargetDist<item>
+   The TargetDist item indicates which distribution versions are offered for
+   installation from. It should be stable, unstable or frozen.
+   
+<tag>X-TargetVersion<item>
+   The target version item is set if the user selects a specific version, it
+   overrides the TargetDist selection if both are present.
+</taglist>
+</sect>
+                                                                  <!-- }}} -->
+<!-- Binary Package Cache					       {{{ -->
+<!-- ===================================================================== -->
+<sect>Binary Package Cache (pkgcache.bin)
+
+<p>
+Please see cache.sgml for a complete description of what this file is. The 
+cache file is updated whenever the contents of the lists directory changes.
+If the cache is erased, corrupted or of a non-matching version it will
+be automatically rebuilt by all of the tools that need it. 
+<em>srcpkgcache.bin</> contains a cache of all of the package files in the 
+source list. This allows regeneration of the cache when the status files 
+change to use a prebuilt version for greater speed.
+</sect>
+                                                                  <!-- }}} -->
+<!-- Downloads Directory					       {{{ -->
+<!-- ===================================================================== -->
+<sect>Downloads Directory (archives)
+
+<p>
+The archives directory is where all downloaded .deb archives go. When the
+file transfer is initiated the deb is placed in partial. Once the file
+is fully downloaded and its MD5 hash and size are verifitied it is moved
+from partial into archives/. Any files found in archives/ can be assumed 
+to be verified.
+
+<p>
+No dirctory structure is transfered from the receiving site and all .deb
+file names conform to debian conventions. No short (msdos) filename should
+be placed in archives. If the need arises .debs should be unpacked, scanned
+and renamed to their correct internal names. This is mostly to prevent
+file name conflicts but other programs may depend on this if convenient.
+Downloaded .debs must be found in one of the package lists with an exact
+name + version match..
+</sect>
+                                                                  <!-- }}} -->
+<!-- The Methods Directory					       {{{ -->
+<!-- ===================================================================== -->
+<sect> The Methods Directory (/usr/lib/apt/methods)
+
+<p>
+Like dselect, APT will support plugable acquisition methods to complement
+its internaly supported methods. The files in
+this directory are execultables named after the URI type. APT will
+sort the required URIs and spawn these programs giving a full sorted, quoted 
+list of URIs.
+
+<p>
+The interface is simple, the program will be given a list
+of URIs on the command line. The URIs will be in pairs, the first 
+being the actual URI and the second being the filename to write the data to. 
+The current directory will be set properly by APT and it is 
+expected the method will put files relative to the current directory. 
+The output of these programs is strictly speficied. The programs must accept
+nothing from stdin (stdin will be an invalid fd) and they must output 
+status information to stdout according to the format below.
+Stderr will be redirected to the logging facility.
+
+<p>
+Each line sent to stdout must be a line that has a single letter and a
+space. Strings after the first letter do not need quoting, they are taken
+as is till the end of the line. The tag letters, listed in expected order, 
+is as follows:
+
+<taglist>
+
+<tag>F - Change URI<item>
+This specifies a change in URI. All information after this will be applied
+to the new URI. When the URI is changed it is assumed that the old URI has
+completed unless an error is set. The format is <var>F URI</>
+
+<tag>S - Object Size<item>
+This specifies the expected size of the object. APT will use this to 
+compute percent done figures. If it is not sent then a kilobyte meter
+will be used instead of a percent display. The foramat is <var>S INTEGER</>
+
+<tag>E - Error Information<item>
+Exactly one line of error information can be set for each URI. The
+information will be summarized for the user. If an E tag is send before
+any F tags then the error is assumed to be a fatal method error and all URI
+fetches for that method are aborted with that error string. The format 
+is <var>E String</>
+
+<tag>I - Informative progress information<item>
+The I tag allows the method to specify the status of the connection. 
+Typically the GUI will show the last recieved I line. The format is
+<var>I String</> As a general rule an I tag should be ommitted before a
+lengthy operation only. Things that always take a short period are not
+suited for I tags. I tags should change wnenever the methods state changes.
+Some standard forms, in order of occurance, are <var>Connecting to SITE</>,
+<var>Connecting to SITE (1.1.1.1)</>, <var>Waiting for file</>, 
+<var>Authenticating</>, <var>Downloading</>, <var>Resuming (size)</>, 
+<var>Computing MD5</> <var>I</> lines should never print out information that 
+APT is already aware of, such as file names.
+
+<tag>R - Set final path<item>
+The R tag allows the method to tell APT that the file is present in the
+local file system. APT might copy it into a the download directory. The format
+is <var>R String</>
+
+<tag>M - MD5Sum of the file<item>
+The method is expected to compute the md5 hash on the fly as the download
+progresses. The final md5 of the file is to be output when the file is 
+completed. If the md5 is not output it will not be checked! Some methods
+such as the file method will not check md5's because they are most
+commonly used on mirrors or local CD-ROM's, a paranoid option may be
+provided in future to force checking. The format is <var>M MD5-String</>
+
+<tag>L - Log output<item>
+This tag indicates a string that should be dumped to some log file. The
+string is for debugging and is not ment to be seen by the user. The format
+is <var>L String</> Log things should only be used in a completed method
+if they have special relavence to what is happening.
+</taglist>
+
+<p>
+APT monitors the progress of the transfer by watching the file size. This
+means the method must not create any temp files and must use a fairly small
+buffer. The method is also responsible for If-Modified-Since (IMS) queries
+for the object. It should check ../outputname to get the time stamp but not
+size. The size may be different because the file was uncompressed after
+it was transfed. A method must <em>never</> change the file in .., it may
+only change the output file in the current directory.
+
+<p>
+The APT 'http' program is the reference implementation of this specification, 
+it implements all of the features a method is expected to do.
+</sect>
+                                                                  <!-- }}} -->
+<!-- The Mirror List						       {{{ -->
+<!-- ===================================================================== -->
+<sect> The Mirror List
+
+<p>
+The mirror list is stored on the primary debian web server (www.debian.org)
+and contains a machine readable list of all known debian mirrors. The mirror
+URI type will cause this list to be downloaded and considered. It has the
+same form as the source list. When the source list specifies mirror
+as the target the mirror list is scanned to find the nescessary parts for 
+the requested distributions and components. This means the user could 
+have a line like:
+
+<var>deb mirror:http://www.debian.org/mirrorlist stable main non-us</var>
+
+which would likely cause APT to choose two separate sites to download from,
+one for main and another for non-us.
+
+<p>
+Some form of network measurement will have to be used to gauge performance
+of each of the mirrors. This will be discussed later, initial versions
+will use the first found URI.
+</sect>
+                                                                  <!-- }}} -->
+<!-- The Release File						       {{{ -->
+<!-- ===================================================================== -->
+<sect> The Release File
+
+<p>
+This file plays and important role in how APT presents the archive to the 
+user. Its main purpose is to present a descriptive name for the source
+of each version of each package. It also is used to detect when new versions
+of debian are released. It augments the package file it is associated with 
+by providing meta information about the entire archive which the Packages
+file describes.
+
+<p>
+The full name of the distribution for presentation to the user is formed
+as 'label version archive', with a possible extended name being 
+'label version archive component'.
+
+<p>
+The file is formed as the package file (RFC-822) with the following tags
+defined:
+
+<taglist>
+<tag>Archive<item>
+This is the common name we give our archives, such as <em>stable</> or
+<em>unstable</>.
+
+<tag>Component<item>
+Referes to the sub-component of the archive, <em>main</>, <em>contrib</>
+etc.
+
+<tag>Version<item>
+This is a version string with the same properties as in the Packages file.
+It represents the release level of the archive.
+
+<tag>Origin<item>
+This specifies who is providing this archive. In the case of Debian the
+string will read 'Debian'. Other providers may use their own string
+
+<tag>Label<item>
+This carries the encompassing name of the distribution. For Debian proper
+this field reads 'Debian'. For derived distributions it should contain their 
+proper name.
+
+<tag>Architecture<item>
+When the archive has packages for a single architecture then the Architecture
+is listed here. If a mixed set of systems are represented then this should
+contain the keyword <em>mixed</em>.
+
+<tag>NotAutomatic<item>
+A Yes/No flag indicating that the archive is extremely unstable and its
+version's should never be automatically selected. This is to be used by 
+experimental.
+
+<tag>Description<item>
+Description is used to describe the release. For instance experimental would
+contain a warning that the packages have problems.
+</taglist>
+
+<p>
+The location of the Release file in the archive is very important, it must 
+be located in the same location as the packages file so that it can be 
+located in all situations. The following is an example for the current stable
+release, 1.3.1r6 
+
+<example>
+Archive: stable
+Compontent: main
+Version: 1.3.1r6
+Origin: Debian
+Label: Debian
+Architecture: i386
+</example>
+
+This is an example of experimental,
+<example>
+Archive: experimental
+Version: 0
+Origin: Debian
+Label: Debian
+Architecture: mixed
+NotAutomatic: Yes
+</example>
+
+And unstable,
+<example>
+Archive: unstable
+Compontent: main
+Version: 2.1
+Origin: Debian
+Label: Debian
+Architecture: i386
+</example>
+
+</sect>
+                                                                  <!-- }}} -->
+
+</book>