Conary | James Tanner

Recovered from the older tannerjc.net wiki snapshot dated January 23, 2016.

FAQ

What are the caveats of addCapsule

build an empty appliance and install the rpm by hand first to see what will probably go wrong with addCapsule
pre and post rpm scripts are not inspected for dependencies and may fail for simple commands like bc
pre and post rpm scripts may create additonal files that might require more space than the image may have accounted for

list all packages in group

conary rq group-oracle-appliance –all-troves

file stream missing

   * oracle-xe:source: Error building recipe oracle-xe:source=/oracle.eng.rpath.com@r:oracle-1-devel//rmake-repository.eng.rpath.com@local:oracle-1-devel/10.2.0.1_1.0-24.1[is: x86_64]: File Stream Missing
    The following file stream was not found on the server:
    fileId: 360b8b4c6c980e6487ef29ad538944c314097935
    This could be due to an incomplete mirror, insufficient permissions,
    or the troves using this filestream having been removed from the server.

[root@rbuilder~]# /usr/share/rbuilder/scripts/rmake-reset

source file does not exist

Source file filename does not exist

The sourcefile specified does not actually exist
add –debug-all to the rbuild command on the first build, so that it does a cvc add on the file
cd packagedir; cvc add filename; cvc ci -m manual add

commit outdated sources

Use --commit-outdated-sources to commit anyway

this is a flag for the rmake command
rmake commit –commit-outdated-sources jobid

connection refused

Error occurred opening repository https://user:pwd@hostname/repos/rmake-repository: Connection refused (via Conary proxy localhost)

/etc/rmake/server.d/25_rbuilder-rapa.conf has the wrong hostname for rbuilderUrl
fix the hostname and restart the rbuilder

world-writable permission

WorldWriteableExecutables: /home/oracle/.bash_profile has executable mode 0101763 with world-writeable permission

A method, such as addSource is trying to set permissions such as 755 whereas it should be 0755

Workflows

Appliance-workflow

setup

download the rbuilder ISO: https://products.rpath.com/download/
install the rbuilder appliance
download the devimage ISO: http://docs.rpath.com/download/
install the devimage ISO
devimage: conary migrate group-devimage-appliance ; reboot
devimage: rbuild config –ask

targets

add the targets

platforms

enable a platform
http://docs.rpath.com/platforms/platform_repositories.html
The centos platform requires all repo data from every cent 5 point release and the updates

[root@sasha httpd]# fgrep xml access_log | awk '{print $7}' | sort -u
/repos/centos//5.0/os/i386/repodata/primary.xml.gz
/repos/centos//5.0/os/i386/repodata/repomd.xml
/repos/centos/5.0/os/i386/repodata/repomd.xml
/repos/centos//5.0/os/x86_64/repodata/primary.xml.gz
/repos/centos//5.0/os/x86_64/repodata/repomd.xml
/repos/centos//5.0/updates/i386/repodata/primary.xml.gz
/repos/centos//5.0/updates/i386/repodata/repomd.xml
/repos/centos//5.0/updates/x86_64/repodata/primary.xml.gz
/repos/centos//5.0/updates/x86_64/repodata/repomd.xml
/repos/centos//5.1/os/i386/repodata/primary.xml.gz
/repos/centos//5.1/os/i386/repodata/repomd.xml
/repos/centos//5.1/os/x86_64/repodata/primary.xml.gz
/repos/centos//5.1/os/x86_64/repodata/repomd.xml
/repos/centos//5.1/updates/i386/repodata/primary.xml.gz
/repos/centos//5.1/updates/i386/repodata/repomd.xml
/repos/centos//5.1/updates/x86_64/repodata/primary.xml.gz
/repos/centos//5.1/updates/x86_64/repodata/repomd.xml
/repos/centos//5.2/os/i386/repodata/primary.xml.gz
/repos/centos//5.2/os/i386/repodata/repomd.xml
/repos/centos//5.2/os/x86_64/repodata/primary.xml.gz
/repos/centos//5.2/os/x86_64/repodata/repomd.xml
/repos/centos//5.2/updates/i386/repodata/primary.xml.gz
/repos/centos//5.2/updates/i386/repodata/repomd.xml
/repos/centos//5.2/updates/x86_64/repodata/primary.xml.gz
/repos/centos//5.2/updates/x86_64/repodata/repomd.xml
/repos/centos//5.3/os/i386/repodata/primary.xml.gz
/repos/centos//5.3/os/i386/repodata/repomd.xml
/repos/centos//5.3/os/x86_64/repodata/primary.xml.gz
/repos/centos//5.3/os/x86_64/repodata/repomd.xml
/repos/centos//5.3/updates/i386/repodata/primary.xml.gz
/repos/centos//5.3/updates/i386/repodata/repomd.xml
/repos/centos//5.3/updates/x86_64/repodata/primary.xml.gz
/repos/centos//5.3/updates/x86_64/repodata/repomd.xml
/repos/centos//5.4/os/i386/repodata/primary.xml.gz
/repos/centos//5.4/os/i386/repodata/repomd.xml
/repos/centos//5.4/os/x86_64/repodata/primary.xml.gz
/repos/centos//5.4/os/x86_64/repodata/repomd.xml
/repos/centos//5.4/updates/i386/repodata/primary.xml.gz
/repos/centos//5.4/updates/i386/repodata/repomd.xml
/repos/centos//5.4/updates/x86_64/repodata/primary.xml.gz
/repos/centos//5.4/updates/x86_64/repodata/repomd.xml
/repos/centos//5.5/os/i386/repodata/primary.xml.gz
/repos/centos//5.5/os/i386/repodata/repomd.xml
/repos/centos//5.5/os/x86_64/repodata/primary.xml.gz
/repos/centos//5.5/os/x86_64/repodata/repomd.xml
/repos/centos//5.5/updates/i386/repodata/primary.xml.gz
/repos/centos//5.5/updates/i386/repodata/repomd.xml
/repos/centos//5.5/updates/x86_64/repodata/primary.xml.gz
/repos/centos//5.5/updates/x86_64/repodata/repomd.xml
/repos/centos/5/os/i386/repodata/repomd.xml

appliance

create a new appliance from an enabled platform

group

create a group to represent the package repository

image

create an image(s) that will be sent to the required target(s) [vmware, xen, amazon, etc]

package

checkout a new package
edit recipe
build package
add package to group
build group

system

deploy image to a target, which creates a system

Notes

devimage

conary update subversion –install-label=conary.rpath.com@rpl:2 –resolve

KVM

Virtio disks are not recognized by the installer ISO. Use normal IDE/scsi disks. Rebuild the initrd afterward to handle virt disks.

daemons

Daemons are commonly placed in /srv. This is not a hard requirement, but it allows you to use the servicedirs macro.

r.addArchive('http://link/%(name)s-distribution-%(version)s.0.0.Final.zip', dir='%(servicedir)s/')

info packages

info packages don’t own any files (other than the ones in /etc/conary/userinfo)
create the homedir within the package recipe which -requires- the user

init scripts

A highly configurable approach is to keep a local copy of the init script and use r.addSource to move it into place. Use macros for the destination directory instead of explicit /etc/init.d/blah so that the tag scripts will automatically chkconfig add/on the file.

Example:

r.addSource('jboss.init', dest='%(initdir)s/jboss', mode=0755)

ownership

files

http://cvs.rpath.com/conary-docs/conary.build.packagepolicy.Ownership-class.html

r.Ownership is not recursive command like chown -R on a flat filesystem. It acts more like a sed command (policy) on a file (metadata) which contains a list of files. It also does not handle directories because they are not part of the conary metadata explicitly. Always place a trailing slash on the filterexp parameter.

Example

[jtanner@devimage Development]$ cat chownme/chownme.recipe  | egrep -v -e ^\$ -e ^\#
class Chownme(PackageRecipe):
    name = 'chownme'
    version = '1'
    buildRequires = []
    def setup(r):
        r.addArchive('data.tar.gz', dir='/opt/')
        r.Ownership('jboss', 'jboss', '/opt/data/')

Result

[jtanner@devimage Development]$ conary rq chownme --lsl
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/1
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/2
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/3
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/subdir/4
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/subdir/5
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/subdir/6

directories

Directories have to be explicitly excluded so that they will be laid out with proper ownership. The user:group seems to inherit from the previous r.Ownership method. IMPORTANT: the regex for exclusion is not a single dir, it is a regex for all directories listed in the package metadata.

[jtanner@devimage Development]$ cat chownme/chownme.recipe  | egrep -v -e ^\$ -e ^\#
class Chownme(PackageRecipe):
    name = 'chownme'
    version = '1'
    buildRequires = []
    def setup(r):
        r.addArchive('data.tar.gz', dir='/opt/')
        r.Ownership('jboss', 'jboss', '/opt/data/')
        r.ExcludeDirectories(exceptions='/opt/data/')

Result

[jtanner@devimage Development]$ conary rq chownme --lsl
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/1
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/2
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/3
drwxrwxr-x    1 jboss    jboss           0 2011-06-20 16:47:42 UTC /opt/data/subdir
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/subdir/4
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/subdir/5
-rw-rw-r--    1 jboss    jboss           0 2011-06-20 15:36:35 UTC /opt/data/subdir/6

replace

The replace method seems to be finicky on modifying files that are not read/write to the rmake-chroot user in the chroot environment. Ensure that the destination file’s permissions are set with a 4 character octal value [0755] by addSource, or the replacement will produce a permission denied error:

IOError: [Errno 13] Permission denied: '/tmp/rmake/builds/jboss/_ROOT_/home/jboss/.bash_profile'

bash-3.2$ find /tmp/rmake/builds/jboss/_ROOT_/ | fgrep home | xargs ls -al
--w----r-T 1 rmake-chroot rmake-chroot  306 Jun 22 09:24 /tmp/rmake/builds/jboss/_ROOT_/home/jboss/.bash_profile

platforms

test connection

192.168.2.51 - - [29/Jun/2011:14:27:05 -0400] GET /repos/centos/5.0/os/i386/repodata/repomd.xml HTTP/1.1 200 1140 - conary-http-client/0.1

commands

rbuild

Performance

enabled virtio on the rbuilder if it is a kvm guest
echo ‘hugetlbfs /dev/hugepages hugetlbfs defaults 0 0’ /etc/fstab
echo ‘vm.nr_hugepages = 280’ /etc/sysctl.conf
reboot
virsh edit guestname; memoryBacking hugepages/ /memoryBacking
service libvirtd restart
virsh start guestname
rbuilder without hugepages enabled

[jtanner@devimage1 Development]$ time rbuild build packages git -m test
[Fri Jul  1 09:32:50 2011] Creating rMake build job for 1 items
+ found lookaside://git.kernel.org_pub_scm_git_git.git/git.git--master--v1.7.6.tar.bz2 in repository
Added Job 13
  git:source=rmake-repository.dj.edm@local:git-1-devel/1.7.6-0.11
[09:33:09] Watching job 13
[09:33:10] [13] - State: Loaded
[09:33:11] [13] - job troves set
[09:33:12] [13] - State: Building
[09:33:12] [13] - Building troves
[09:33:12] [13] - git:source{x86_64} - State: Queued
[09:33:12] [13] - git:source{x86_64} - State: Resolving
[09:33:33] [13] - git:source{x86_64} - State: Queued
[09:33:34] [13] - git:source{x86_64} - State: Creating Chroot
[09:36:52] [13] - git:source{x86_64} - State: Building
[09:41:04] [13] - git:source{x86_64} - State: Built
[09:41:10] [13] - State: Built
[09:41:10] [13] - build job finished successfully
13 git{x86_64} - Built - (Job Built) ([h]elp)
Committing changeset (11839KB (100%) of 11839KB at 10542KB/sec)...
Committed job 13:
    git:source=/git.dj.edm@r:git-1-devel//rmake-repository.dj.edm@local:git-1-devel/1.7.6-0.11[is: x86_64]{x86_64} -
       git=/git.dj.edm@r:git-1-devel/1.7.6-1-1[is: x86_64]
       git:source=/git.dj.edm@r:git-1-devel/1.7.6-1[]

real	8m40.372s
user	0m3.392s
sys	0m1.648s

rbuilder with hugepages enabled

[jtanner@devimage1 Development]$ time rbuild build packages git -m hugepages on
[Fri Jul  1 10:17:53 2011] Creating rMake build job for 1 items
+ found lookaside://git.kernel.org_pub_scm_git_git.git/git.git--master--v1.7.6.tar.bz2 in repository
Added Job 14
  git:source=rmake-repository.dj.edm@local:git-1-devel/1.7.6-1.1
[10:18:22] Watching job 14
[10:18:24] [14] Downloading 1 recipes...
[10:18:25] [14] Loading 1 out of 1: git:source
[10:18:26] [14] - State: Loaded
[10:18:27] [14] - job troves set
[10:18:28] [14] - State: Building
[10:18:28] [14] - Building troves
[10:18:28] [14] - git:source{x86_64} - State: Queued
[10:18:28] [14] - git:source{x86_64} - State: Resolving
[10:19:07] [14] - git:source{x86_64} - State: Queued
[10:19:08] [14] - git:source{x86_64} - State: Creating Chroot
[10:21:08] [14] - git:source{x86_64} - State: Building
[10:23:46] [14] - git:source{x86_64} - State: Built
[10:23:47] [14] - State: Built
[10:23:47] [14] - build job finished successfully
14 git{x86_64} - Built - (Job Built) ([h]elp)
Committing changeset (37KB (100%) of 37KB at 11453KB/sec)...
Committed job 14:
    git:source=/git.dj.edm@r:git-1-devel//rmake-repository.dj.edm@local:git-1-devel/1.7.6-1.1[is: x86_64]{x86_64} -
       git=/git.dj.edm@r:git-1-devel/1.7.6-2-1[is: x86_64]
       git:source=/git.dj.edm@r:git-1-devel/1.7.6-2[]

real	6m20.984s
user	0m3.336s
sys	0m0.532s

Recipes

dos2unix

class Dos2unix(CPackageRecipe):
    name = 'dos2unix'
    version = '5'

        '''
        #make error
        msgfmt -c po/dos2unix/de.po -o po/dos2unix/de.mo
        make: msgfmt: Command not found

        #whatprovides
        $ yum provides */msgfmt
        gettext-0.18.1.1-4.fc14.x86_64 : GNU libraries and utilities for producing multi-lingual messages
        Repo        : fedora
        Matched from:
        Filename    : /usr/bin/msgfmt
        '''

    buildRequires = ['gettext:rpm']

    def setup(r):
        r.addArchive('http://sourceforge.net/projects/dos2unix/files/dos2unix/5.3/dos2unix-5.3.tar.gz')
        r.Make()
        r.MakeInstall()

LAMP

class OverrideRecipe(FactoryRecipeClass):

    autoResolve = True

    def addRecipePackages(r):
        r.add('lsof')
        r.add('which')
        r.add('openssh')
        r.add('openssh-server')
        r.add('openssh-clients')
        r.add('vsftpd')
        r.add('vim-enhanced')
        r.add('mysql')
        r.add('mysql-server')
        r.add('php-mysql')
        r.add('php')
        r.add('httpd')

        r.addPostUpdateScript( contents = #!/bin/bash
echo password | passwd --stdin root
)

git

class Git(CPackageRecipe):
    name = 'git'
    version = '1.7.6'

    buildRequires = ['curl-devel:rpm', 'autoconf:rpm', 'gettext:rpm']

    def setup(r):
        r.addGitSnapshot('git://git.kernel.org/pub/scm/git/git.git', tag='v1.7.6')
        #r.Make(makeName = 'configure')
        r.Make('configure')
        r.ManualConfigure('--prefix=/opt/%(name)s-%(version)s')
        r.Make()
        r.MakeInstall()
        r.ExcludeDirectories(exceptions='/opt/%(name)s-%(version)s')
        r.LinkCount(exceptions=('/opt/git-1.7.6/bin/', '/opt/git-1.7.6/libexec/git-core/'))

jboss6

class info_jboss(UserInfoRecipe):
    name = 'info-jboss'
    version = '1'

    def setup(r):
    	r.User('jboss', 1001, homedir='/home/jboss', shell='/bin/bash')

class Jboss(PackageRecipe):
    name = 'jboss'
    version = '6'

    buildRequires = ['unzip:rpm', 'chkconfig:rpm']

    def setup(r):
	r.addArchive('jboss-as-distribution-6.0.0.Final.zip', dir='%(servicedir)s/')
	#r.addArchive('http://sourceforge.net/projects/jboss/files/JBoss/JBoss-6.0.0.Final/%(name)s-as-distribution-%(version)s.0.0.Final.zip', dir='%(servicedir)s/')

	#rename the jboss dir and set permissions, exceptions are required to chown dirs
	r.Move('%(servicedir)s/%(name)s-%(version)s.0.0.Final/', '%(servicedir)s/%(name)s-%(version)s')
	r.Ownership('jboss', 'jboss', '%(servicedir)s/%(name)s-%(version)s/')
	r.ExcludeDirectories(exceptions='%(servicedir)s/%(name)s-%(version)s/')

	# the solaris init script creates a bad dep on /usr/bin/bash
	r.Remove('%(servicedir)s/%(name)s-%(version)s/bin/jboss_init_solaris.sh')

	#copy over the init script
	r.addSource('jboss.init', dest='%(initdir)s/%(name)s', mode=0755)

	#copy over the jboss user's .bash_profile
	r.addSource('jboss.bash_profile', dest='/home/jboss/.bash_profile', mode=0644)
	r.Ownership('jboss', 'jboss', '/home/jboss/')

	#set the proper location of jboss in the requisite scripts
	r.Replace('JBOSS_HOME=', 'JBOSS_HOME=%(servicedir)s/%(name)s-%(version)s', '/home/jboss/.bash_profile')
	r.Replace('JBOSS_HOME:-', 'JBOSS_HOME:-%(servicedir)s/%(name)s-%(version)s', '%(initdir)s/%(name)s')
	r.Replace('JAVA_HOME=', 'JAVA_HOME=/usr/java/jdk1.6.0_26', '/home/jboss/.bash_profile')
	r.Replace('JAVAPTH:-', 'JAVAPTH:-/usr/java/jdk1.6.0_26', '%(initdir)s/%(name)s')

	#set the admin password
	r.Replace('admin=admin', 'admin=secretsauce', '%(servicedir)s/%(name)s-%(version)s/server/default/conf/props/jmx-console-users.properties')

	#set the default user and pass
	r.Replace('kermit=friend', 'user1=password1', '%(servicedir)s/%(name)s-%(version)s/server/default/conf/props/jbossws-roles.properties')

#!/bin/sh
#
# description: JBoss Start Stop Restart
# processname: jboss
# chkconfig: 345 20 80
#
# $Id: jboss_init_redhat.sh 81068 2008-11-14 15:14:35Z dimitris@jboss.org $
#
# JBoss Control Script
#
# To use this script run it as root - it will switch to the specified user
#
# Here is a little (and extremely primitive) startup/shutdown script
# for RedHat systems. It assumes that JBoss lives in /usr/local/jboss,
# it's run by user 'jboss' and JDK binaries are in /usr/local/jdk/bin.
# All this can be changed in the script itself.
#
# Either modify this script for your requirements or just ensure that
# the following variables are set correctly before calling the script.

#define where jboss is - this is the directory containing directories log, bin, conf etc
JBOSS_HOME=${JBOSS_HOME:-}

#define the user under which jboss will run, or use 'RUNASIS' to run as the current user
JBOSS_USER=${JBOSS_USER:-jboss}

#make sure java is in your path
JAVAPTH=${JAVAPTH:-}

#configuration to use, usually one of 'minimal', 'default', 'all'
JBOSS_CONF=${JBOSS_CONF:-default}

#if JBOSS_HOST specified, use -b to bind jboss services to that address
JBOSS_HOST=0.0.0.0
JBOSS_BIND_ADDR=${JBOSS_HOST:+-b $JBOSS_HOST}

#define the classpath for the shutdown class
JBOSSCP=${JBOSSCP:-$JBOSS_HOME/bin/shutdown.jar:$JBOSS_HOME/client/jnet.jar}

#define the script to use to start jboss
JBOSSSH=${JBOSSSH:-$JBOSS_HOME/bin/run.sh -c $JBOSS_CONF $JBOSS_BIND_ADDR}

if [ $JBOSS_USER = RUNASIS ]; then
  SUBIT=
else
  SUBIT=su - $JBOSS_USER -c
fi

if [ -n $JBOSS_CONSOLE -a ! -d $JBOSS_CONSOLE ]; then
  # ensure the file exists
  touch $JBOSS_CONSOLE
  if [ ! -z $SUBIT ]; then
    chown $JBOSS_USER $JBOSS_CONSOLE
  fi
fi

if [ -n $JBOSS_CONSOLE -a ! -f $JBOSS_CONSOLE ]; then
  echo WARNING: location for saving console log invalid: $JBOSS_CONSOLE
  echo WARNING: ignoring it and using /dev/null
  JBOSS_CONSOLE=/dev/null
fi

#define what will be done with the console log
JBOSS_CONSOLE=${JBOSS_CONSOLE:-/dev/null}

JBOSS_CMD_START=cd $JBOSS_HOME/bin; $JBOSSSH
JBOSS_CMD_STOP=${JBOSS_CMD_STOP:-java -classpath $JBOSSCP org.jboss.Shutdown --shutdown}

if [ -z `echo $PATH | grep $JAVAPTH` ]; then
  export PATH=$PATH:$JAVAPTH
fi

if [ ! -d $JBOSS_HOME ]; then
  echo JBOSS_HOME does not exist as a valid directory : $JBOSS_HOME
  exit 1
fi

echo JBOSS_CMD_START = $JBOSS_CMD_START

case $1 in
start)
    cd $JBOSS_HOME/bin
    if [ -z $SUBIT ]; then
        eval $JBOSS_CMD_START ${JBOSS_CONSOLE} 21
    else
        $SUBIT $JBOSS_CMD_START ${JBOSS_CONSOLE} 21
    fi
    ;;
stop)
    if [ -z $SUBIT ]; then
        $JBOSS_CMD_STOP
    else
        $SUBIT $JBOSS_CMD_STOP
    fi
    ;;
restart)
    $0 stop
    $0 start
    ;;
*)
    echo usage: $0 (start|stop|restart|help)
esac

FAQ

What are the caveats of addCapsule

list all packages in group

file stream missing

source file does not exist

commit outdated sources

connection refused

world-writable permission

Workflows

setup

targets

platforms

appliance

group

image

package

system

Notes

devimage

KVM

daemons

info packages

init scripts

ownership

files

directories

replace

platforms

commands

rbuild

Performance

Recipes

dos2unix

LAMP

git

jboss6

puppetd

oracle