I have 2 SmartOS servers at home with a total of 17 zones running persistently - a majority of these provisioned with the latest LTS support release of pkgsrc: 2014Q4. All zones are setup to download a similar set of “bootstrap” packages to make them ready to be used. Packages like gcc, git, etc. I pull to all of my zones upon creation.

The problem, however, is my home internet is not the fastest - it usually can pull packages at around 100KB/s. git and gcc alone are responsible for over 100MB of compiled binary data, meaning these initial package downloads can take over 15 minutes per new zone.

A possible solution to speed up initial package downloads I investigated was to rsync the entire pkgsrc tree to a local server, and host it over an internal HTTP server. This, however, required a massive amount of storage dedicated to packages I am almost guaranteed to never use. I could have just pulled 2014Q4, but then I would have been required to add new releases whenever I wanted to upgrade, and keep around the old packages until all my old zones have been upgraded.

Instead, I created a Node.JS HTTP caching server to proxy GET and HEAD requests to Joyent’s pkgsrc server, and cache the data on the local filesystem. The requests will be proxied if the file does not exist locally, and will be dual-written to the requesting client, and the local filesystem. Subsequent requests for the same resource will be streamed from the local filesystem without every making an outbound request.

fs-caching-server

https://github.com/bahamas10/node-fs-caching-server

The program is written to be generic - it allows it to act as a caching proxy to any website, and also allows the user to specify a regex to test a URL when deciding if the request should be cached, or just proxied directly with no cache.

How To

To spin up a server on SmartOS to front the Joyent pkgsrc servers install the fs-caching-server program.

npm install -g fs-caching-server

Then create the directory that will be used locally to cache packages as they are lazily retrieved (any directory will work).

mkdir -p /data/pkgsrc-proxy

And finally, create an SMF manifest to start the service - all configuration parameters will be set as environmental variables.

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='application/pkgsrc-proxy' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <dependency name='dep0' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/milestone/multi-user:default'/>
    </dependency>
    <exec_method name='start' type='method' exec='fs-caching-server &amp;' timeout_seconds='10'>
      <method_context working_directory='/data/pkgsrc-proxy'>
        <method_credential user='nobody' group='nobody'/>
        <method_environment>
          <envvar name='PATH' value='/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin:/opt/custom/bin'/>
          <envvar name='FS_CACHE_PORT' value='8000'/>
          <envvar name='FS_CACHE_HOST' value='0.0.0.0'/>
          <envvar name='FS_CACHE_URL' value='http://pkgsrc.joyent.com'/>
          <envvar name='FS_CACHE_DEBUG' value='true'/>
        </method_environment>
      </method_context>
    </exec_method>
    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='30'/>
    <template>
      <common_name>
        <loctext xml:lang='C'>Joyent pkgsrc proxy and caching service</loctext>
      </common_name>
    </template>
  </service>
</service_bundle>

The service will cache data in its CWD by default, which is set in the manifest as /data/pkgsrc-proxy.

If a regex is not specified as FS_CACHE_REGEX it will default to /\.(png|jpg|jpeg|css|html|js|tar|tgz|tar\.gz)$/ which is sufficient for caching pkgsrc packages.

Finally, configure any SmartOS zones to use the new server. Modify /opt/local/etc/pkgin/repositories.conf to point to the new server.

# http://pkgsrc.joyent.com/packages/SmartOS/2014Q4/x86_64/All
http://10.0.1.50:8000/packages/SmartOS/2014Q4/x86_64/All

(replace 10.0.1.50:8000 with whatever host and port was used earlier)

And rebuild the database

rm -f /var/db/pkgin/pkgin.db
pkgin -y up

And your done! Any new packages installed will be pulled through the caching proxy.