r00t2/bdisk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Activity

fixing some merge issues

Browse Source
This commit is contained in:
brent s. 2021-03-28 12:43:02 -04:00
commit 75580b43cc
Signed by: bts
GPG Key ID: 8C004C2F93481F6B
25 changed files with 4122 additions and 612 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
TODO
View File

@ -1,28 +1,24 @@
- write classes/functions
- XML-based config
-x XML syntax
--- xregex btags - case-insensitive? this can be represented in-pattern:
xhttps://stackoverflow.com/a/9655186/733214
-x configuration generator
--- xprint end result xml config to stderr for easier redirection? or print prompts to stderr and xml to stdout?
-- xXSD for validation
-- Flask app for generating config?
-- TKinter (or pygame?) GUI?
--- https://docs.python.org/3/faq/gui.html
--- https://www.pygame.org/wiki/gui
- ensure we use docstrings in a Sphinx-compatible manner?
https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
at the very least document all the functions and such so pydoc's happy.
- better prompt display. i might include them as constants in a separate file
and then import it for e.g. confgen. or maybe a Flask website/app?

- locking
- for docs, 3.x (as of 3.10) was 2.4M.
- GUI? at least for generating config...
- Need ability to write/parse mtree specs (or a similar equivalent) for applying ownerships/permissions to overlay files
- xNeed ability to write/parse mtree specs (or a similar equivalent) for applying ownerships/permissions to overlay files
-- parsing is done. writing may? come later.

- SSL key gen:
import OpenSSL
k = OpenSSL.crypto.PKey()
k.generate_key(OpenSSL.crypto.TYPE_RSA, 4096)
x = OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM,
k,
cipher = 'aes256',
passphrase = 'test')

- need to package:
python-hashid (https://psypanda.github.io/hashID/,
https://github.com/psypanda/hashID,
https://pypi.org/project/hashID/)

- package for PyPI:
# https://packaging.python.org/tutorials/distributing-packages/
@ -38,7 +34,6 @@ BUGS.SQUARE-R00T.NET bugs/tasks:
#14: Use os.path.join() for more consistency/pythonicness
#24: Run as regular user? (pychroot? fakeroot?)
#34: Build-time support for only building single phase of build
#36: Allow parsing pkg lists with inline comments
#39: Fix UEFI
#40: ISO overlay (to add e.g. memtest86+ to final ISO)
#43: Support resuming partial tarball downloads (Accet-Ranges: bytes)
#43: Support resuming partial tarball downloads (Accept-Ranges: bytes)

1
bdisk/BIOS.py
View File

@ -1,3 +1,4 @@
import jinja2
import os
import shutil


90
bdisk/GPG.py
View File

@ -1,8 +1,26 @@
import datetime
import gpg
import os
import psutil
import gpg.errors


# This helps translate the input name from the conf to a string compatible with the gpg module.
_algmaps = {#'cv': 'cv{keysize}', # DISABLED, can't sign (only encrypt). Currently only 25519
'ed': 'ed{keysize}', # Currently only 25519
#'elg': 'elg{}', # DISABLED, can't sign (only encrypt). 1024, 2048, 4096
'nist': 'nistp{keysize}', # 256, 384, 521
'brainpool.1': 'brainpoolP{keysize}r1', # 256, 384, 512
'sec.k1': 'secp{keysize}k1', # Currently only 256
'rsa': 'rsa{keysize}', # Variable (1024 <> 4096), but we only support 1024, 2048, 4096
'dsa': 'dsa{keysize}'} # Variable (768 <> 3072), but we only support 768, 2048, 3072

# This is just a helper function to get a delta from a unix epoch.
def _epoch_helper(epoch):
d = datetime.datetime.utcfromtimestamp(epoch) - datetime.datetime.utcnow()
return(abs(int(d.total_seconds()))) # Returns a positive integer even if negative...
#return(int(d.total_seconds()))

# http://files.au.adversary.org/crypto/GPGMEpythonHOWTOen.html
# https://www.gnupg.org/documentation/manuals/gpgme.pdf
# Support ECC? https://www.gnupg.org/faq/whats-new-in-2.1.html#ecc
@ -60,7 +78,7 @@ class GPGHandler(object):
self._prep_home()
else:
self._check_home()
self.ctx = self.get_context(home_dir = self.home)
self.ctx = self.GetContext(home_dir = self.home)

def _check_home(self, home = None):
if not home:
@ -94,11 +112,12 @@ class GPGHandler(object):
'write to')
return()

def get_context(self, **kwargs):
def GetContext(self, **kwargs):
ctx = gpg.Context(**kwargs)
return(ctx)

def kill_stale_agent(self):
def KillStaleAgent(self):
# Is this even necessary since I switched to the native gpg module instead of the gpgme one?
_process_list = []
# TODO: optimize; can I search by proc name?
for p in psutil.process_iter():
@ -113,7 +132,64 @@ class GPGHandler(object):
# for p in plst:
# psutil.Process(p).terminate()

def get_sigs(self, data_in):
def CreateKey(self, name, algo, keysize, email = None, comment = None, passwd = None, key = None, expiry = None):
algo = _algmaps[algo].format(keysize = keysize)
userid = name
userid += ' ({0})'.format(comment) if comment else ''
userid += ' <{0}>'.format(email) if email else ''
if not expiry:
expires = False
else:
expires = True
self.ctx.create_key(userid,
algorithm = algo,
expires = expires,
expires_in = _epoch_helper(expiry),
sign = True)
# Even if expires is False, it still parses the expiry...
# except OverflowError: # Only trips if expires is True and a negative expires occurred.
# raise ValueError(('Expiration epoch must be 0 (to disable) or a future time! '
# 'The specified epoch ({0}, {1}) is in the past '
# '(current time is {2}, {3}).').format(expiry,
# str(datetime.datetime.utcfromtimestamp(expiry)),
# datetime.datetime.utcnow().timestamp(),
# str(datetime.datetime.utcnow())))
return(k)
# We can't use self.ctx.create_key; it's a little limiting.
# It's a fairly thin wrapper to .op_createkey() (the C GPGME API gpgme_op_createkey) anyways.
flags = (gpg.constants.create.SIGN |
gpg.constants.create.CERT)
if not expiry:
flags = (flags | gpg.constants.create.NOEXPIRE)
if not passwd:
flags = (flags | gpg.constants.create.NOPASSWD)
else:
# Thanks, gpg/core.py#Context.create_key()!
sys_pinentry = gpg.constants.PINENTRY_MODE_DEFAULT
old_pass_cb = getattr(self, '_passphrase_cb', None)
self.ctx.pinentry_mode = gpg.constants.PINENTRY_MODE_LOOPBACK
def passphrase_cb(hint, desc, prev_bad, hook = None):
return(passwd)
self.ctx.set_passphrase_cb(passphrase_cb)
try:
if not key:
try:
self.ctx.op_createkey(userid, algo, 0, 0, flags)
k = self.ctx.get_key(self.ctx.op_genkey_result().fpr, secret = True)
else:
if not isinstance(key, gpg.gpgme._gpgme_key):
key = self.ctx.get_key(key)
if not key:
raise ValueError('Key {0} does not exist'.format())
#self.ctx.op_createsubkey(key, )
finally:
if not passwd:
self.ctx.pinentry_mode = sys_pinentry
if old_pass_cb:
self.ctx.set_passphrase_cb(*old_pass_cb[1:])
return(k)

def GetSigs(self, data_in):
key_ids = []
# Currently as of May 13, 2018 there's no way using the GPGME API to do
# the equivalent of the CLI's --list-packets.
@ -131,3 +207,9 @@ class GPGHandler(object):
l = [i.strip() for i in line.split(':')]
key_ids.append(l[0])
return(key_ids)

def CheckSigs(self, keys, sig_data):
try:
self.ctx.verify(sig_data)
except:
pass # TODO

7
bdisk/SSL.py
View File

@ -3,3 +3,10 @@ import OpenSSL
# migrate old functions of bSSL to use cryptography
# but still waiting on their recpipes.
# https://cryptography.io/en/latest/x509/tutorial/
#import OpenSSL
#k = OpenSSL.crypto.PKey()
#k.generate_key(OpenSSL.crypto.TYPE_RSA, 4096)
#x = OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM,
# k,
# cipher = 'aes256',
# passphrase = 'test')

1
bdisk/basedistro/antergos.py Symbolic link
View File

@ -0,0 +1 @@
archlinux.py

1
bdisk/basedistro/arch.py Symbolic link
View File

@ -0,0 +1 @@
archlinux.py

96
bdisk/basedistro/archlinux.py Normal file
View File

@ -0,0 +1,96 @@
#!/usr/bin/env python3

# Supported initsys values:
# systemd
# Possible future inclusions:
# openrc
# runit
# sinit
# s6
# shepherd
initsys = 'systemd'

def extern_prep(cfg, cur_arch = 'x86_64'):
import os
import re
mirrorlist = os.path.join(cfg['build']['paths']['chroot'],
cur_arch,
'etc/pacman.d/mirrorlist')
with open(mirrorlist, 'r') as f:
mirrors = []
for i in f.readlines():
m = re.sub('^\s*#.*$', '', i.strip())
if m != '':
mirrors.append(m)
if not mirrors:
# We do this as a fail-safe.
mirror = ('\n\n# Added by BDisk\n'
'Server = https://arch.mirror.square-r00t.net/'
'$repo/os/$arch\n')
with open(mirrorlist, 'a') as f:
f.write(mirror)
return()

# This will be run before the regular packages are installed. It can be
# whatever script you like, as long as it has the proper shebang and doesn't
# need additional packages installed.
# In Arch's case, we use it for initializing the keyring and installing an AUR
# helper.
pkg_mgr_prep = """#!/bin/bash

pacman -Syy
pacman-key --init
pacman-key --populate archlinux
pacman -S --noconfirm --needed base
pacman -S --noconfirm --needed base-devel multilib-devel git linux-headers \
mercurial subversion vala xorg-server-devel
cd /tmp
sqrt="https://git.square-r00t.net/BDisk/plain/external"
# Temporary until there's another AUR helper that allows dropping privs AND
# automatically importing GPG keys.
pkg="${sqrt}/apacman-current.pkg.tar.xz?h=4.x_rewrite"
curl -sL -o apacman-current.pkg.tar.xz ${pkg}
pacman -U --noconfirm apacman-current.pkg.tar.xz
rm apacman*
"""

# Special values:
# {PACKAGE} = the package name
# {VERSION} = the version specified in the <package version= ...> attribute
# {REPO} = the repository specified in the <package repo= ...> attribute
# If check_cmds are needed to run before installing, set pre_check to True.
# Return code 0 means the package is installed already, anything else means we
# should try to install it.
#### AUR SUPPORT ####
packager = {'pre_check': False,
'sys_update': ['/usr/bin/apacman', '-S', '-u'],
'sync_cmd': ['/usr/bin/apacman', '-S', '-y', '-y'],
'check_cmds': {'versioned': ['/usr/bin/pacman',
'-Q', '-s',
'{PACKAGE}'],
'unversioned': ['/usr/bin/pacman',
'-Q', '-s',
'{PACKAGE}']
},
'update_cmds': {'versioned': ['/usr/bin/pacman',
'-S', '-u',
'{PACKAGE}'],
'unversioned': ['/usr/bin/pacman',
'-S', '-u',
'{PACKAGE}']
},
}

# These are packages *required* to exist on the base guest, no questions asked.
# TODO: can this be trimmed down?
prereqs = ['arch-install-scripts', 'archiso', 'bzip2', 'coreutils',
'customizepkg-scripting', 'cronie', 'dhclient', 'dhcp', 'dhcpcd',
'dosfstools', 'dropbear', 'efibootmgr', 'efitools', 'efivar',
'file', 'findutils', 'iproute2', 'iputils', 'libisoburn',
'localepurge', 'lz4', 'lzo', 'lzop', 'mkinitcpio-nbd',
'mkinitcpio-nfs-utils', 'mkinitcpio-utils', 'nbd', 'ms-sys',
'mtools', 'net-tools', 'netctl', 'networkmanager', 'pv',
'python', 'python-pyroute2', 'rsync', 'sed', 'shorewall',
'squashfs-tools', 'sudo', 'sysfsutils',
'syslinux', 'traceroute', 'vi']

1
bdisk/basedistro/manjaro.py Symbolic link
View File

@ -0,0 +1 @@
archlinux.py

931
bdisk/bdisk.xsd
View File

@ -1,6 +1,933 @@
<?xml version="1.0" encoding="UTF-8" ?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://bdisk.square-r00t.net"
xmlns="http://bdisk.square-r00t.net"
targetNamespace="http://bdisk.square-r00t.net/"
xmlns="http://bdisk.square-r00t.net/"
elementFormDefault="qualified">

<!-- CUSTOM TYPES -->
<!-- t_btag_uri: a string that will allow btags (xpath or variable only) or a URI string (but NOT a URN). -->
<!-- We can't use xs:anyURI because it is too loose (allows things like relative paths, etc.) -->
<!-- but ALSO too restrictive in that btags fail validation ({ and } are invalid for anyURI, -->
<!-- ironically). -->
<xs:simpleType name="t_btag_uri">
<xs:restriction base="xs:string">
<xs:pattern value="\w+:(/?/?)[^\s]+"/>
<xs:pattern value=".*\{variable%[A-Za-z0-9_]\}.*"/>
<xs:pattern value=".*\{xpath%[&quot;'A-Za-z0-9_/\(\)\.\*@\-\[\]=]+\}.*"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_btag_uri -->

<!-- t_filename: a POSIX fully-portable filename. -->
<xs:simpleType name="t_filename">
<xs:restriction base="xs:string">
<xs:pattern value="([a-z0-9._-]+){1,255}"/>
<xs:pattern value=".*\{variable%[A-Za-z0-9_]\}.*"/>
<xs:pattern value=".*\{xpath%[&quot;'A-Za-z0-9_/\(\)\.\*@\-\[\]=]+\}.*"/>
<!-- We don't allow (string)(regex) or (regex)(string) or (string)(regex)(string) or multiple regexes -->
<!-- because that's just... not feasible to manage from a parsing perspective. -->
<xs:pattern value="\{regex%.+\}"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_filename -->

<!-- t_gpg_keyid: a set of various patterns that match GPG key IDs. -->
<xs:simpleType name="t_gpg_keyid">
<xs:restriction base="xs:string">
<xs:pattern value="(none|new)"/>
<xs:pattern value="(auto|default)"/>
<xs:pattern value="(0x)?[0-9A-Fa-f]{40}"/>
<xs:pattern value="(0x)?[0-9A-Fa-f]{16}"/>
<xs:pattern value="(0x)?[0-9A-Fa-f]{8}"/>
<xs:pattern value="([0-9A-Fa-f ]{4}){5} ?([0-9A-Fa-f ]{4}){4}[0-9A-Fa-f]{4}"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_gpg_keyid -->

<!-- t_gpg_keyid_list: a type for a list of key IDs. -->
<xs:simpleType name="t_gpg_keyid_list">
<xs:list itemType="t_gpg_keyid"/>
</xs:simpleType>
<!-- END t_gpg_key_list -->

<!-- t_net_loc: a remote host. Used for PKI Subject's commonName and host for rsync. -->
<xs:simpleType name="t_net_loc">
<xs:restriction base="xs:string">
<xs:pattern
value="(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_net_loc -->

<!-- t_pass_hash_algo: used for t_password. -->
<xs:simpleType name="t_pass_hash_algo">
<xs:restriction base="xs:string">
<xs:enumeration value="des"/>
<xs:enumeration value="md5"/>
<xs:enumeration value="sha256"/>
<xs:enumeration value="sha512"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_pass_hash_algo -->

<!-- t_pass_salt: used for t_password. -->
<xs:simpleType name="t_pass_salt">
<xs:restriction base="xs:string">
<xs:pattern value="($[156]($rounds=[0-9]+)?$[a-zA-Z0-9./]{1,16}$?|auto|)"/>
<xs:pattern value="\{variable%[A-Za-z0-9_]\}"/>
<xs:pattern value="\{xpath%[&quot;'A-Za-z0-9_\(\)\.\*\-/\[\]=]+\}"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_pass_salt -->

<!-- t_password: used for rootpass and user/password elements. -->
<xs:complexType name="t_password">
<!-- The below will need some fleshing out and testing. It may not be possible strictly via XSD. -->
<!-- TODO: restrict the value further with a union or multi-group regex that checks for a valid length? -->
<!-- des: ????? -->
<!-- md5: "[a-zA-Z0-9./]{22}" -->
<!-- sha256: "[a-zA-Z0-9./]{43}" -->
<!-- sha512: "[a-zA-Z0-9./]{86}" -->
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="hash_algo" type="t_pass_hash_algo" use="optional"/>
<xs:attribute name="hashed" type="xs:boolean" use="required"/>
<xs:attribute name="salt" type="t_pass_salt" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
<!-- END t_password -->

<!-- t_path: for specifying subdirectories (either local filesystem or remote paths). -->
<xs:simpleType name="t_path">
<xs:restriction base="xs:string">
<!-- We include blank to operate on default actions (or default filepaths). -->
<xs:pattern value=""/>
<xs:pattern value="(.+)/([^/]+)"/>
<xs:pattern value="((.+)/([^/]+))?\{variable%[A-Za-z0-9_]\}((.+)/([^/]+))?"/>
<xs:pattern value="((.+)/([^/]+))?\{xpath%[&quot;'A-Za-z0-9_\(\)\.\*\-/\[\]=]+\}((.+)/([^/]+))?"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_path -->

<!-- t_pki_cert: used for pki/ca/cert and pki/client/cert. -->
<xs:complexType name="t_pki_cert">
<xs:simpleContent>
<xs:extension base="t_path">
<xs:attribute name="hash_algo" use="required">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="blake2b512"/>
<xs:enumeration value="blake2s256"/>
<xs:enumeration value="gost"/>
<xs:enumeration value="md4"/>
<xs:enumeration value="md5"/>
<xs:enumeration value="mdc2"/>
<xs:enumeration value="rmd160"/>
<xs:enumeration value="sha1"/>
<xs:enumeration value="sha224"/>
<xs:enumeration value="sha256"/>
<xs:enumeration value="sha384"/>
<xs:enumeration value="sha512"/>
<xs:enumeration value="none"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
<!-- END t_pki_cert -->

<!-- t_pki_key: used for pki/ca/key and pki/client/key -->
<xs:complexType name="t_pki_key">
<xs:simpleContent>
<xs:extension base="t_path">
<xs:attribute name="cipher" use="required">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="aes128"/>
<xs:enumeration value="aes192"/>
<xs:enumeration value="bf"/>
<xs:enumeration value="blowfish"/>
<xs:enumeration value="camellia128"/>
<xs:enumeration value="camellia192"/>
<xs:enumeration value="camellia256"/>
<xs:enumeration value="des"/>
<xs:enumeration value="rc2"/>
<xs:enumeration value="seed"/>
<xs:enumeration value="none"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="passphrase" type="xs:string"/>
<xs:attribute name="keysize"
type="xs:positiveInteger"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
<!-- END t_pki_key -->

<!-- t_pki_subject: used for pki/ca/subject and pki/client/subject -->
<xs:complexType name="t_pki_subject">
<xs:all>
<!-- .../SUBJECT/COMMONNAME -->
<xs:element name="commonName" type="t_net_loc"/>
<!-- END .../SUBJECT/COMMONNAME -->
<!-- .../SUBJECT/COUNTRYNAME -->
<xs:element name="countryName">
<xs:simpleType>
<xs:restriction base="xs:string">
<!-- We can't validate an actual ISO-3166 ALPHA-2 code, but we can validate the format. -->
<!-- TODO: maybe cron the generation of an external namespace? -->
<xs:pattern value="[A-Z]{2}"/>
<xs:pattern value=".*\{variable%[A-Za-z0-9_]\}.*"/>
<xs:pattern value=".*\{xpath%[&quot;'A-Za-z0-9_/\(\)\.\*@\-\[\]=]+\}.*"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END .../SUBJECT/COUNTRYNAME -->
<!-- .../SUBJECT/LOCALITYNAME -->
<xs:element name="localityName" type="xs:string"/>
<!-- END .../SUBJECT/LOCALITYNAME -->
<!-- .../SUBJECT/STATEORPROVINCENAME -->
<xs:element name="stateOrProvinceName"
type="xs:string"/>
<!-- END .../SUBJECT/STATEORPROVINCENAME -->
<!-- .../SUBJECT/ORGANIZATION -->
<xs:element name="organization" type="xs:string"/>
<!-- END .../SUBJECT/ORGANIZATION -->
<!-- .../SUBJECT/ORGANIZATIONALUNITNAME -->
<xs:element name="organizationalUnitName"
type="xs:string"/>
<!-- END .../SUBJECT/ORGANIZATIONALUNITNAME -->
<!-- .../SUBJECT/EMAILADDRESS -->
<xs:element name="emailAddress" type="xs:string"/>
<!-- END .../SUBJECT/EMAILADDRESS -->
</xs:all>
</xs:complexType>
<!-- END t_pki_subject -->

<!-- t_remote_file: an element that lets us define both a file pattern for remote content and flags attribute. -->
<xs:complexType name="t_remote_file">
<xs:simpleContent>
<xs:extension base="t_filename">
<xs:attribute name="flags" type="t_remote_file_flags" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
<!-- END t_remote_file -->

<!-- t_remote_file_flags: a type to match a list of known flags. -->
<xs:simpleType name="t_remote_file_flags">
<xs:list>
<xs:simpleType>
<xs:restriction base="xs:string">
<!-- Currently we only support two flags. -->
<xs:enumeration value="regex"/>
<xs:enumeration value="latest"/>
</xs:restriction>
</xs:simpleType>
</xs:list>
</xs:simpleType>
<!-- END t_remote_file_flags -->

<!-- t_username: enforce a POSIX-compliant username. Used for user/username elements. -->
<xs:simpleType name="t_username">
<xs:restriction base="xs:string">
<xs:pattern value="[a-z_]([a-z0-9_-]{0,31}|[a-z0-9_-]{0,30}$)"/>
<xs:pattern value="\{variable%[A-Za-z0-9_]\}"/>
<xs:pattern value="\{xpath%[&quot;'A-Za-z0-9_\(\)\.\*\-/\[\]=]+\}"/>
</xs:restriction>
</xs:simpleType>
<!-- END t_username -->
<!-- END CUSTOM TYPES -->

<!-- ROOT ELEMENT ("BDISK") -->
<xs:element name="bdisk">
<xs:complexType>
<!-- Should this be xs:sequence instead? -->
<xs:sequence>
<!-- BDISK/PROFILE -->
<xs:element name="profile" maxOccurs="unbounded" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/META -->
<xs:element name="meta" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/META/NAMES -->
<xs:element name="names" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/META/NAMES/NAME -->
<xs:element name="name" maxOccurs="1" minOccurs="1">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern value="[A-Z0-9]{1,8}"/>
<xs:pattern value="\{variable%[A-Za-z0-9_]\}"/>
<xs:pattern value="\{xpath%[A-Za-z0-9_\(\)\.\*\-/]+\}"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/META/NAMES/NAME -->
<!-- BDISK/PROFILE/META/NAMES/UXNAME -->
<xs:element name="uxname" maxOccurs="1" minOccurs="1">
<xs:simpleType>
<xs:restriction base="xs:string">
<!-- refer to the 2009 POSIX spec, "3.282 Portable Filename Character Set" -->
<!-- http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_282 -->
<!-- (We use this string to name some files.) -->
<xs:pattern value="([A-Za-z0-9._-]+){1,255}"/>
<xs:pattern value="\{variable%[A-Za-z0-9_]\}"/>
<xs:pattern value="\{xpath%[A-Za-z0-9_\(\)\.\*\-/]+\}"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/META/NAMES/UXNAME -->
<!-- BDISK/PROFILE/META/NAMES/PNAME -->
<xs:element name="pname" maxOccurs="1" minOccurs="1">
<xs:simpleType>
<xs:restriction base="xs:string">
<!-- TODO: Can I use UTF-8 instead? -->
<!-- https://stackoverflow.com/a/9805789/733214 -->
<xs:pattern value="\p{IsBasicLatin}*"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/META/NAMES/PNAME -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META/NAMES -->
<!-- BDISK/PROFILE/META/DESC -->
<xs:element name="desc" maxOccurs="1" minOccurs="1" type="xs:string"/>
<!-- END BDISK/PROFILE/META/DESC -->
<!-- BDISK/PROFILE/META/DEV -->
<xs:element name="dev" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/META/DEV/AUTHOR -->
<xs:element name="author" maxOccurs="1" minOccurs="1"
type="xs:normalizedString"/>
<!-- END BDISK/PROFILE/META/DEV/AUTHOR -->
<!-- BDISK/PROFILE/META/DEV/EMAIL -->
<!-- The following does NOT WORK. Shame, really. -->
<!-- It seems to be an invalid pattern per my XSD validator (xmllint). -->
<!--<xs:pattern value="([!#-&apos;*+/-9=?A-Z^-~-]+(\.[!#-&apos;*+/-9=?A-Z^-~-]+)*|&quot;([]!#-[^-~ \t]|(\\[\t -~]))+&quot;)@([!#-&apos;*+/-9=?A-Z^-~-]+(\.[!#-&apos;*+/-9=?A-Z^-~-]+)*|\[[\t -Z^-~]*])"/>-->
<xs:element name="email" maxOccurs="1" minOccurs="1"
type="xs:normalizedString"/>
<!-- END BDISK/PROFILE/META/DEV/EMAIL -->
<!-- BDISK/PROFILE/META/DEV/WEBSITE -->
<xs:element name="website" maxOccurs="1" minOccurs="1"
type="t_btag_uri"/>
<!-- END BDISK/PROFILE/META/DEV/WEBSITE -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META/DEV -->
<!-- BDISK/PROFILE/META/URI -->
<xs:element name="uri" maxOccurs="1" minOccurs="1" type="t_btag_uri"/>
<!-- END BDISK/PROFILE/META/URI -->
<!-- BDISK/PROFILE/META/VER -->
<xs:element name="ver" maxOccurs="1" minOccurs="1">
<xs:simpleType>
<xs:restriction base="xs:normalizedString">
<!-- Like ../names/uxname, this is also used to name certain files so, POSIX portable filename. -->
<xs:pattern value="([A-Za-z0-9._-]+){1,255}"/>
<xs:pattern value="\{variable%[A-Za-z0-9_]\}"/>
<xs:pattern value="\{xpath%[A-Za-z0-9_\(\)\.\*\-/]+\}"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/META/VER -->
<!-- BDISK/PROFILE/META/MAX_RECURSE -->
<xs:element name="max_recurse" maxOccurs="1" minOccurs="1">
<xs:simpleType>
<xs:restriction base="xs:positiveInteger">
<xs:maxExclusive value="1000"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/META/MAX_RECURSE -->
<!-- BDISK/PROFILE/META/REGEXES -->
<xs:element name="regexes" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/META/REGEXES/PATTERN -->
<xs:element name="pattern" maxOccurs="unbounded" minOccurs="1">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="id" type="xs:string"
use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META/REGEXES/PATTERN -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META/REGEXES -->
<!-- BDISK/PROFILE/META/VARIABLES -->
<xs:element name="variables" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/META/VARIABLES/VARIABLE -->
<xs:element name="variable" maxOccurs="unbounded" minOccurs="1">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="id" type="xs:string"
use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META/VARIABLES/VARIABLE -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META/VARIABLES -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/META -->
<!-- BDISK/PROFILE/ACCOUNTS -->
<xs:element name="accounts" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/ACCOUNTS/ROOTPASS -->
<xs:element name="rootpass" maxOccurs="1" minOccurs="1" type="t_password"/>
<!-- END BDISK/PROFILE/ACCOUNTS/ROOTPASS -->
<!-- BDISK/PROFILE/ACCOUNTS/USER -->
<xs:element name="user" maxOccurs="unbounded" minOccurs="0">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/ACCOUNTS/USER/USERNAME -->
<xs:element name="username" type="t_username" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/ACCOUNTS/USER/USERNAME -->
<!-- BDISK/PROFILE/ACCOUNTS/USER/COMMENT -->
<!-- https://en.wikipedia.org/wiki/Gecos_field -->
<!-- Through experimentation, this *seems* to cap at 990 chars. -->
<xs:element name="comment" maxOccurs="1"
minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:normalizedString">
<xs:maxLength value="990"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/ACCOUNTS/USER/COMMENT -->
<!-- BDISK/PROFILE/ACCOUNTS/USER/PASSWORD -->
<xs:element name="password" type="t_password" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/ACCOUNTS/USER/PASSWORD -->
</xs:all>
<xs:attribute name="sudo" type="xs:boolean" use="optional"/>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/ACCOUNTS/USER -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/ACCOUNTS -->
<!-- BDISK/PROFILE/SOURCES -->
<xs:element name="sources" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:sequence>
<!-- BDisk only supports two different architectures (x86/i686 and x86_64, respectively) currently. -->
<!-- TODO: future improvements may let us include e.g. two different x86_64 environments (e.g. CentOS and Debian on the same media), but this is like, still in development stages. -->
<!-- BDISK/PROFILE/SOURCES/SOURCE -->
<xs:element name="source" minOccurs="1" maxOccurs="2">
<xs:complexType>
<xs:all>
<!-- We cheat here. TECHNICALLY it should ONLY be scheme://location (no /path...), but there isn't a data type for that. -->
<!-- Currently we enforce only one item. Future BDisk versions may be able to make use of multiple <mirror>s and select best one based on speed. -->
<!-- BDISK/PROFILE/SOURCES/SOURCE/MIRROR -->
<xs:element name="mirror" type="t_btag_uri" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/SOURCES/SOURCE/MIRROR -->
<!-- BDISK/PROFILE/SOURCES/SOURCE/ROOTPATH -->
<xs:element name="rootpath" maxOccurs="1" minOccurs="1"
type="t_path"/>
<!-- END BDISK/PROFILE/SOURCES/SOURCE/ROOTPATH -->
<!-- BDISK/PROFILE/SOURCES/SOURCE/TARBALL -->
<xs:element name="tarball" maxOccurs="1" minOccurs="1"
type="t_remote_file"/>
<!-- END BDISK/PROFILE/SOURCES/SOURCE/TARBALL -->
<!-- BDISK/PROFILE/SOURCES/SOURCE/CHECKSUM -->
<xs:element name="checksum" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="t_remote_file">
<!-- There is NO way we can validate this, because it will vary based on the algorithms supported by the build host. -->
<xs:attribute name="hash_algo" type="xs:string"
use="required"/>
<xs:attribute name="explicit" type="xs:boolean"
use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SOURCES/SOURCE/CHECKSUM -->
<!-- BDISK/PROFILE/SOURCES/SOURCE/SIG -->
<xs:element name="sig" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="t_remote_file">
<!-- Required; otherwise there's no point using it. -->
<xs:attribute name="keys" type="t_gpg_keyid_list"
use="required"/>
<xs:attribute name="keyserver" type="t_btag_uri"
use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SOURCES/SOURCE/SIG-->
</xs:all>
<xs:attribute name="arch" use="required">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern value="(i686|x86(_64)?|32|64)"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SOURCES/SOURCE -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SOURCES -->
<!-- BDISK/PROFILE/PACKAGES -->
<xs:element name="packages" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/PACKAGES/PACKAGE -->
<xs:element name="package" maxOccurs="unbounded" minOccurs="1">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="version" type="xs:string" use="optional"/>
<xs:attribute name="repo" type="xs:string" use="optional"/>
<!-- Default is "both" -->
<xs:attribute name="arch" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern value="(i686|x86(_64)?|32|64|both)"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/PACKAGES/PACKAGE -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/PACKAGES -->
<!-- BDISK/PROFILE/SERVICES -->
<xs:element name="services" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/SERVICES/SERVICE -->
<xs:element name="service" maxOccurs="unbounded" minOccurs="1">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="enabled" type="xs:boolean" use="required"/>
<xs:attribute name="blacklisted" type="xs:boolean" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SERVICES/SERVICE -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SERVICES -->
<!-- BDISK/PROFILE/BUILD -->
<xs:element name="build" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/BUILD/PATHS -->
<xs:element name="paths">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/BUILD/PATHS/BASE -->
<xs:element name="base" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/BASE -->
<!-- BDISK/PROFILE/BUILD/PATHS/CACHE -->
<xs:element name="cache" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/CACHE -->
<!-- BDISK/PROFILE/BUILD/PATHS/CHROOT -->
<xs:element name="chroot" maxOccurs="1" minOccurs="1"
type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/CHROOT -->
<!-- BDISK/PROFILE/BUILD/PATHS/OVERLAY -->
<xs:element name="overlay" maxOccurs="1" minOccurs="1"
type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/OVERLAY -->
<!-- BDISK/PROFILE/BUILD/PATHS/TEMPLATES -->
<xs:element name="templates" maxOccurs="1" minOccurs="1"
type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/TEMPLATES -->
<!-- BDISK/PROFILE/BUILD/PATHS/MOUNT -->
<xs:element name="mount" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/MOUNT -->
<!-- BDISK/PROFILE/BUILD/PATHS/DISTROS -->
<xs:element name="distros" maxOccurs="1" minOccurs="1"
type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/DISTROS -->
<!-- BDISK/PROFILE/BUILD/PATHS/DEST -->
<xs:element name="dest" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/DEST -->
<!-- BDISK/PROFILE/BUILD/PATHS/ISO -->
<xs:element name="iso" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/ISO -->
<!-- BDISK/PROFILE/BUILD/PATHS/HTTP -->
<xs:element name="http" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/HTTP -->
<!-- BDISK/PROFILE/BUILD/PATHS/TFTP -->
<xs:element name="tftp" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/TFTP -->
<!-- EBDISK/PROFILE/BUILD/PATHS/PKI -->
<xs:element name="pki" maxOccurs="1" minOccurs="1" type="t_path"/>
<!-- END BDISK/PROFILE/BUILD/PATHS/PKI -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/BUILD/PATHS -->
<!-- BDISK/PROFILE/BUILD/BASEDISTRO -->
<xs:element name="basedistro"/>
<!-- END BDISK/PROFILE/BUILD/BASEDISTRO -->
</xs:all>
<xs:attribute name="its_full_of_stars" type="xs:boolean"/>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/BUILD -->
<!-- BDISK/PROFILE/ISO -->
<xs:element name="iso" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:attribute name="sign" type="xs:boolean"/>
<xs:attribute name="multi_arch">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="yes"/>
<xs:enumeration value="no"/>
<xs:enumeration value="true"/>
<xs:enumeration value="false"/>
<xs:enumeration value="x86_64"/>
<xs:enumeration value="x86"/>
<xs:enumeration value="64"/>
<xs:enumeration value="32"/>
<xs:enumeration value="i686"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/ISO -->
<!-- BDISK/PROFILE/IPXE -->
<xs:element name="ipxe" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/IPXE/URI -->
<xs:element name="uri" type="t_btag_uri" maxOccurs="1" minOccurs="1"/>
<!-- END BDISK/PROFILE/IPXE/URI -->
</xs:all>
<xs:attribute name="sign" type="xs:boolean"/>
<xs:attribute name="iso" type="xs:boolean"/>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/IPXE -->
<!-- BDISK/PROFILE/GPG -->
<xs:element name="gpg" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/GPG/KEY -->
<xs:element name="key" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/GPG/KEY/NAME -->
<xs:element name="name" type="xs:normalizedString" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/GPG/KEY/NAME -->
<!-- BDISK/PROFILE/GPG/KEY/EMAIL -->
<xs:element name="email" type="xs:normalizedString" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/GPG/KEY/EMAIL -->
<!-- BDISK/PROFILE/GPG/KEY/COMMENT -->
<xs:element name="comment" type="xs:string" maxOccurs="1"
minOccurs="0"/>
<!-- END BDISK/PROFILE/GPG/KEY/COMMENT -->
<!-- BDISK/PROFILE/GPG/KEY/SUBKEY -->
<xs:element name="subkey" maxOccurs="1" minOccurs="0">
<xs:complexType>
<!-- See below for notes on attributes. -->
<!-- TODO: convert into shared type for parent as well? -->
<xs:attribute name="algo" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="rsa"/>
<xs:enumeration value="dsa"/>
<xs:enumeration value="ed"/>
<xs:enumeration value="nist"/>
<xs:enumeration value="brainpool.1"/>
<xs:enumeration value="sec.k1"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="keysize" type="xs:positiveInteger" use="optional"/>
<xs:attribute name="expire" use="optional">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:pattern value="(0|[0-9]{10})"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/GPG/KEY/SUBKEY -->
</xs:all>
<xs:attribute name="algo" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<!-- rsa, dsa, and elgamal are "normal". Newer GnuPG supports ECC (yay!), so we have support for those in the XSD (you can get a list with gpg -with-colons -list-config curve | cut -f3 -d":" | tr ';' '\n'). -->
<!-- We test in-code if the host supports it. -->
<xs:enumeration value="rsa"/>
<xs:enumeration value="dsa"/>
<!-- The following only support encryption. The entire reason we'd be generating a key is to sign files, so we disable them. -->
<!-- <xs:enumeration value="elg"/> -->
<!-- <xs:enumeration value="cv"/> -->
<xs:enumeration value="ed"/>
<xs:enumeration value="nist"/>
<xs:enumeration value="brainpool.1"/>
<xs:enumeration value="sec.k1"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- We COULD constrain this further, but it's conditional upon the algo type. So we'll do that in BDisk itself. -->
<!-- But it may be possible? https://stackoverflow.com/a/39045446/733214 -->
<xs:attribute name="keysize" type="xs:positiveInteger" use="optional"/>
<!-- XSD doesn't have a datatype for Epoch vs. 0 (for no expire). -->
<xs:attribute name="expire" use="optional">
<xs:simpleType>
<!--This is xs:integer instead of xs:positiveInteger because 0 will fail validation then. -->
<xs:restriction base="xs:integer">
<xs:pattern value="(0|[0-9]{10})"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/GPG/KEY -->
</xs:sequence>
<xs:attribute name="keyid" type="t_gpg_keyid" use="required"/>
<xs:attribute name="publish" type="xs:boolean" use="optional"/>
<xs:attribute name="prompt_passphrase" type="xs:boolean" use="required"/>
<xs:attribute name="passphrase" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern
value="[!&quot;#$%&amp;\\'\(\)\*\+,\-\./0123456789:;&lt;=&gt;\?@ABCDEFGHIJKLMNOPQRSTUVWXYZ\[\]\^_`abcdefghijklmnopqrstuvwxyz\{\|\}~ ]+"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="gnupghome" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern value="(.+)/([^/]+)"/>
<xs:pattern
value="((.+)/([^/]+))?\{variable%[A-Za-z0-9_]\}((.+)/([^/]+))?"/>
<xs:pattern
value="((.+)/([^/]+))?\{xpath%[A-Za-z0-9_\(\)\.\*\-/]+\}((.+)/([^/]+))?"/>
<xs:pattern value="(none|)"/>
<xs:pattern value="(auto|default)"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/GPG -->
<!-- BDISK/PROFILE/PKI -->
<xs:element name="pki" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/PKI/CA -->
<xs:element name="ca" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/PKI/CA/CERT -->
<xs:element name="cert" maxOccurs="1" minOccurs="1"
type="t_pki_cert"/>
<!-- END BDISK/PROFILE/PKI/CA/CERT -->
<!-- BDISK/PROFILE/PKI/CA/CSR -->
<xs:element name="csr" maxOccurs="1" minOccurs="0" type="t_path"/>
<!-- END BDISK/PROFILE/PKI/CA/CSR -->
<!-- BDISK/PROFILE/PKI/CA/INDEX -->
<xs:element name="index" maxOccurs="1" minOccurs="0" type="t_path"/>
<!-- END BDISK/PROFILE/PKI/CA/INDEX -->
<!-- BDISK/PROFILE/PKI/CA/SERIAL -->
<xs:element name="serial" maxOccurs="1" minOccurs="0"
type="t_path"/>
<!-- END BDISK/PROFILE/PKI/CA/SERIAL -->
<!-- BDISK/PROFILE/PKI/CA/KEY -->
<xs:element name="key" minOccurs="1" maxOccurs="1"
type="t_pki_key"/>
<!-- END BDISK/PROFILE/PKI/CA/CSR -->
<!-- BDISK/PROFILE/PKI/CA/SUBJECT -->
<xs:element name="subject" maxOccurs="1" minOccurs="0"
type="t_pki_subject"/>
<!-- END BDISK/PROFILE/PKI/CA/SUBJECT -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/PKI/CA -->
<!-- BDISK/PROFILE/PKI/CLIENT -->
<xs:element name="client" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/PKI/CLIENT/CERT -->
<xs:element name="cert" maxOccurs="1" minOccurs="1"
type="t_pki_cert"/>
<!-- END BDISK/PROFILE/PKI/CLIENT/CERT -->
<!-- BDISK/PROFILE/PKI/CLIENT/CSR -->
<xs:element name="csr" maxOccurs="1" minOccurs="0" type="t_path"/>
<!-- END BDISK/PROFILE/PKI/CLIENT/CSR -->
<!-- BDISK/PROFILE/PKI/CLIENT/KEY -->
<xs:element name="key" minOccurs="1" maxOccurs="1"
type="t_pki_key"/>
<!-- END BDISK/PROFILE/PKI/CLIENT/CSR -->
<!-- BDISK/PROFILE/PKI/CLIENT/SUBJECT -->
<xs:element name="subject" maxOccurs="1" minOccurs="0"
type="t_pki_subject"/>
<!-- END BDISK/PROFILE/PKI/CLIENT/SUBJECT -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/PKI/CLIENT -->
</xs:sequence>
<xs:attribute name="overwrite" type="xs:boolean" use="required"/>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/PKI -->
<!-- BDISK/PROFILE/SYNC -->
<xs:element name="sync" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:all>
<!-- BDISK/PROFILE/SYNC/IPXE -->
<xs:element name="ipxe" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="t_path">
<xs:attribute name="enabled" type="xs:boolean" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC/IPXE -->
<!-- BDISK/PROFILE/SYNC/TFTP -->
<xs:element name="tftp" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="t_path">
<xs:attribute name="enabled" type="xs:boolean" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC/TFTP -->
<!-- BDISK/PROFILE/SYNC/ISO -->
<xs:element name="iso" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="t_path">
<xs:attribute name="enabled" type="xs:boolean" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC/ISO -->
<!-- BDISK/PROFILE/SYNC/GPG -->
<xs:element name="gpg" maxOccurs="1" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="t_path">
<xs:attribute name="enabled" type="xs:boolean" use="optional"/>
<xs:attribute name="format" use="required">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="asc"/>
<xs:enumeration value="bin"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC/GPG -->
<!-- BDISK/PROFILE/SYNC/RSYNC -->
<xs:element name="rsync" maxOccurs="1" minOccurs="1">
<xs:complexType>
<xs:sequence>
<!-- BDISK/PROFILE/SYNC/RSYNC/USER -->
<xs:element name="user" type="t_username" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/SYNC/RSYNC/USER -->
<!-- BDISK/PROFILE/SYNC/RSYNC/HOST -->
<xs:element name="host" type="t_net_loc" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/SYNC/RSYNC/HOST -->
<!-- BDISK/PROFILE/SYNC/RSYNC/PORT -->
<xs:element name="port" maxOccurs="1" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:positiveInteger">
<xs:minInclusive value="1"/>
<xs:maxInclusive value="65535"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC/RSYNC/PORT -->
<xs:choice>
<!-- BDISK/PROFILE/SYNC/RSYNC/PUBKEY -->
<xs:element name="pubkey" type="t_path" maxOccurs="1"
minOccurs="1"/>
<!-- END BDISK/PROFILE/SYNC/RSYNC/PUBKEY -->
<!-- BDISK/PROFILE/SYNC/RSYNC/PUBKEY -->
<xs:element name="password" maxOccurs="1" minOccurs="1"/>
<!-- END BDISK/PROFILE/SYNC/RSYNC/PUBKEY -->
</xs:choice>
</xs:sequence>
<xs:attribute name="enabled" type="xs:boolean" use="required"/>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC/IPXE -->
</xs:all>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE/SYNC -->
</xs:all>
<xs:attribute name="id" type="xs:positiveInteger" use="optional"/>
<xs:attribute name="name" type="xs:string" use="optional"/>
<xs:attribute name="uuid" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern
value="[0-9a-f]{8}\-[0-9a-f]{4}\-4[0-9a-f]{3}\-[89ab][0-9a-f]{3}\-[0-9a-f]{12}"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- END BDISK/PROFILE -->
</xs:sequence>
</xs:complexType>
</xs:element>
<!-- END BDISK -->
</xs:schema>

63
bdisk/confgen.py
View File

@ -1,10 +1,10 @@
#!/usr/bin/env python3.6
#!/usr/bin/env python3

# Ironically enough, I think building a GUI for this would be *cleaner*.
# Go figure.

import confparse
import crypt
import datetime
import getpass
import os
import utils
@ -134,7 +134,12 @@ class ConfGenerator(object):
self.cfg = c.xml
self.append = True
else:
self.cfg = lxml.etree.Element('bdisk')
_ns = {None: 'http://bdisk.square-r00t.net/',
'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}
_xsi = {
'{http://www.w3.org/2001/XMLSchema-instance}schemaLocation':
'http://bdisk.square-r00t.net bdisk.xsd'}
self.cfg = lxml.etree.Element('bdisk', nsmap = _ns, attrib = _xsi)
self.append = False
self.profile = lxml.etree.Element('profile')
self.cfg.append(self.profile)
@ -155,6 +160,13 @@ class ConfGenerator(object):
self.get_pki()
self.get_gpg()
self.get_sync()
# TODO: make this more specific (script? gui? web? etc.)
# and append comment to bdisk element
_comment = lxml.etree.Comment(
'Generated {0} by BDisk configuration generator'.format(
str(datetime.datetime.now())
)
)
except KeyboardInterrupt:
exit('\n\nCaught KeyboardInterrupt; quitting...')
return()
@ -442,6 +454,8 @@ class ConfGenerator(object):
tarball_elem.attrib['flags'] = 'latest'
tarball_elem.text = tarball['full_url']
print('\n++ SOURCES || {0} || CHECKSUM ++'.format(arch.upper()))
# TODO: explicit not being set for explicitly-set sums,
# and checksum string not actually added to those. investigate.
chksum = lxml.etree.SubElement(source, 'checksum')
_chksum_chk = prompt.confirm_or_no(prompt = (
'\nWould you like to add a checksum for the tarball? (BDisk '
@ -470,7 +484,7 @@ class ConfGenerator(object):
print('Invalid selection. Starting over.')
continue
chksum.attrib['hash_algo'] = checksum_type
chksum.attrib['explicit'] = "no"
chksum.attrib['explicit'] = "false"
chksum.text = checksum['full_url']
else:
# Maybe it's a digest string.
@ -502,8 +516,8 @@ class ConfGenerator(object):
print('Invalid selection. Starting over.')
continue
else:
checksum_type == checksum_type[0]
chksum.attrib['explicit'] = "yes"
checksum_type = checksum_type[0]
chksum.attrib['explicit'] = "true"
chksum.text = checksum
chksum.attrib['hash_algo'] = checksum_type
print('\n++ SOURCES || {0} || GPG ++'.format(arch.upper()))
@ -595,7 +609,7 @@ class ConfGenerator(object):
usage = (
'{0} for yes, {1} for no...\n'))
if _chk_optimizations:
build.attrib['its_full_of_stars'] = 'yes'
build.attrib['its_full_of_stars'] = 'true'
print('\n++ BUILD || PATHS ++')
# Thankfully, we can simplify a lot of this.
_dir_strings = {'base': ('the base directory (used for files that are '
@ -625,7 +639,7 @@ class ConfGenerator(object):
'created that can be used to serve iPXE)'),
'tftp': ('the TFTP directory (where a TFTP/'
'traditional PXE root is created)'),
'ssl': ('the SSL/TLS PKI directory (where we store '
'pki': ('the SSL/TLS PKI directory (where we store '
'the PKI structure we use/re-use - MAKE SURE '
'it is in a path that is well-protected!)')}
has_paths = False
@ -676,9 +690,9 @@ class ConfGenerator(object):
self.profile.append(iso)
# We have more than one arch, so we need to ask how they want to handle
# it.
_ma_strings = {'yes': ('a multi-arch ISO (both architectures on one '
_ma_strings = {'true': ('a multi-arch ISO (both architectures on one '
'ISO)'),
'no': ('separate image files for '
'false': ('separate image files for '
'{0}').format(' and '.join(_arches))}
for a in _arches:
_ma_strings[a] = 'only build an image file for {0}'.format(a)
@ -710,7 +724,7 @@ class ConfGenerator(object):
'option to configure it a bit later).\nWould you like to sign '
'the ISO/USB image files with GPG?\n'), usage = (
'{0} for yes, {1} for no...\n'))
_gpg_sign = ('yes' if _gpg_input else 'no')
_gpg_sign = ('true' if _gpg_input else 'false')
iso.attrib['sign'] = _gpg_sign
self.profile.append(iso)
return()
@ -725,21 +739,21 @@ class ConfGenerator(object):
'see the manual for more information). Would you like to '
'build iPXE support?\n'), usage = (
'{0} for yes, {1} for no...\n'))
_ipxe = ('yes' if _ipxe else 'no')
if _ipxe == 'yes':
_ipxe = ('true' if _ipxe else 'true')
if _ipxe == 'true':
print('\n++ iPXE || MINI-ISO ++')
_iso = prompt.confirm_or_no(prompt = (
'\nWould you like to build a "mini-ISO" (see the manual) for '
'bootstrapping iPXE booting from USB or optical media?\n'),
usage = ('{0} for yes, {1} for no...\n'))
ipxe.attrib['iso'] = ('yes' if _iso else 'no')
ipxe.attrib['iso'] = ('true' if _iso else 'false')
print('\n++ iPXE || SIGNING ++')
_sign = prompt.confirm_or_no(prompt = (
'\nBDisk can sign the mini-ISO and other relevant files for '
'iPXE builds using GPG. Would you like to sign the iPXE build '
'distributables? (You\'ll have the chance to configure GPG '
'later).\n'), usage = ('{0} for yes, {1} for no...\n'))
ipxe.attrib['sign'] = ('yes' if _sign else 'no')
ipxe.attrib['sign'] = ('true' if _sign else 'false')
_uri = None
while not _uri:
print('\n++ iPXE || URL ++')
@ -754,7 +768,7 @@ class ConfGenerator(object):
else:
uri = lxml.etree.SubElement(ipxe, 'uri')
uri.text = _uri
if _ipxe == 'yes':
if _ipxe == 'true':
self.profile.append(ipxe)
return()
@ -780,7 +794,7 @@ class ConfGenerator(object):
'wish to keep persistent keys and certs), you should '
'DEFINITELY answer no here.\n'),
usage = ('{0} for yes, {1} for no...\n'))
pki.attrib['overwrite'] = ('yes' if _overwrite else 'no')
pki.attrib['overwrite'] = ('true' if _overwrite else 'false')
for x in ('ca', 'client'):
print('\n++ SSL/TLS PKI || {0} ++'.format(x.upper()))
_x = None
@ -804,7 +818,7 @@ class ConfGenerator(object):
for x in _xpaths:
_x = self.profile.xpath(x)
for a in _x:
if a == 'yes':
if a == 'true':
_sigchk = True
break
if _sigchk:
@ -848,7 +862,7 @@ class ConfGenerator(object):
'\nWould you like to push the key to the SKS keyserver pool '
'(making it much easier for end-users to look it up)?\n'),
usage = ('{0} for yes, {1} for no...\n'))
gpg.attrib['publish'] = ('yes' if _gpgpublish else 'no')
gpg.attrib['publish'] = ('true' if _gpgpublish else 'false')
print('\n++ GPG || PASSWORD HANDLING ++')
_gpgpass_prompt = prompt.confirm_or_no(prompt = (
'\nWould you like BDisk to prompt you for a passphrase? If not, '
@ -856,7 +870,8 @@ class ConfGenerator(object):
'the configuration (HIGHLY unrecommended) or use a blank '
'passphrase (also HIGHLY unrecommended).\n'),
usage = ('{0} for yes, {1} for no...\n'))
gpg.attrib['prompt_passphrase'] = ('yes' if _gpgpass_prompt else 'no')
gpg.attrib['prompt_passphrase'] = ('true' if _gpgpass_prompt else
'false')
_pass = None
if not _gpgpass_prompt:
while not _pass:
@ -921,7 +936,7 @@ class ConfGenerator(object):
'\nWould you like to sync {0}?\n'.format(_syncs[s])),
usage = ('{0} for yes, {1} for no...\n'))
elem = lxml.etree.SubElement(sync, s)
elem.attrib['enabled'] = ('yes' if _item_sync_chk else 'no')
elem.attrib['enabled'] = ('true' if _item_sync_chk else 'false')
if not _item_sync_chk:
continue
if s == 'gpg':
@ -935,12 +950,12 @@ class ConfGenerator(object):
'\n\t'.join(_choices)
))).strip().lower()
if _export_type.startswith('a'):
_export_type == 'asc'
_export_type = 'asc'
elif _export_type.startswith('b'):
_export_type == 'bin'
_export_type = 'bin'
else:
print('Using the default.')
_export_type == 'asc'
_export_type = 'asc'
elem.attrib['format'] = _export_type
_path = None
while not _path:

422
bdisk/confparse.py
View File

@ -1,15 +1,20 @@
import copy
import re
import os
import pprint
import re
import utils
import validators
import lxml.etree
from urllib.parse import urlparse

etree = lxml.etree
detect = utils.detect()
generate = utils.generate()
transform = utils.transform()
valid = utils.valid()

class Conf(object):
def __init__(self, cfg, profile = None):
def __init__(self, cfg, profile = None, validate_cfg = False,
xsd_file = None):
"""
A configuration object.

@ -36,92 +41,351 @@ class Conf(object):

You can provide any combination of these
(e.g. "profile={'id': 2, 'name' = 'some_profile'}").
Non-greedy matching (meaning ALL attributes specified
must match).
"""
#self.raw = _detect_cfg(cfg) # no longer needed; in utils
if validate_cfg == 'pre':
# Validate before attempting any other operations
self.validate()
self.xml_suppl = utils.xml_supplicant(cfg, profile = profile)
self.profile = self.xml_suppl
self.xml = None
self.profile = None
# Mad props to https://stackoverflow.com/a/12728199/733214
self.xpath_re = re.compile('(?<=(?<!\{)\{)[^{}]*(?=\}(?!\}))')
self.substitutions = {}
self.xpaths = ['xpath']
try:
self.xml = etree.fromstring(self.raw)
except lxml.etree.XMLSyntaxError:
raise ValueError('The configuration provided does not seem to be '
'valid')
self.xsd = None
#if not self.validate(): # Need to write the XSD
# raise ValueError('The configuration did not pass XSD/schema '
# 'validation')
self.get_profile()
self.max_recurse = int(self.profile.xpath('//meta/'
'max_recurse')[0].text)
self.xml = self.xml_suppl.xml
for e in self.xml_suppl.xml.iter():
self.xml_suppl.substitute(e)
self.xml_suppl.get_profile(profile = self.xml_suppl.orig_profile)
with open('/tmp/parsed.xml', 'wb') as f:
f.write(lxml.etree.tostring(self.xml_suppl.xml))
self.profile = self.xml_suppl.profile
self.xsd = xsd_file
self.cfg = {}
if validate_cfg:
# Validation post-substitution
self.validate(parsed = False)
# TODO: populate checksum{} with hash_algo if explicit

def get_pki_obj(self, pki, pki_type):
elem = {}
if pki_type not in ('ca', 'client'):
raise ValueError('pki_type must be "ca" or "client"')
if pki_type == 'ca':
elem['index'] = None
elem['serial'] = None
for e in pki.xpath('./*'):
# These have attribs or children.
if e.tag in ('cert', 'key', 'subject'):
elem[e.tag] = {}
if e.tag == 'subject':
for sub in e.xpath('./*'):
elem[e.tag][sub.tag] = transform.xml2py(sub.text,
attrib = False)
else:
for a in e.xpath('./@*'):
elem[e.tag][a.attrname] = transform.xml2py(a)
elem[e.tag]['path'] = e.text
else:
elem[e.tag] = e.text
return(elem)

def get_source(self, source, item, _source):
_source_item = {'flags': [], 'fname': None}
elem = source.xpath('./{0}'.format(item))[0]
if item == 'checksum':
if elem.get('explicit', False):
_explicit = transform.xml2py(
elem.attrib['explicit'])
_source_item['explicit'] = _explicit
if _explicit:
del(_source_item['fname'])
_source_item['value'] = elem.text
return(_source_item)
else:
_source_item['explicit'] = False
if elem.get('hash_algo', False):
_source_item['hash_algo'] = elem.attrib['hash_algo']
else:
_source_item['hash_algo'] = None
if item == 'sig':
if elem.get('keys', False):
_keys = [i.strip() for i in elem.attrib['keys'].split()]
_source_item['keys'] = _keys
else:
_source_item['keys'] = []
if elem.get('keyserver', False):
_source_item['keyserver'] = elem.attrib['keyserver']
else:
_source_item['keyserver'] = None
_item = elem.text
_flags = elem.get('flags', '')
if _flags:
for f in _flags.split():
if f.strip().lower() == 'none':
continue
_source_item['flags'].append(f.strip().lower())
if _source_item['flags']:
if 'regex' in _source_item['flags']:
ptrn = _item.format(**self.xml_suppl.btags['regex'])
else:
ptrn = None
_source_item['fname'] = detect.remote_files(
'/'.join((_source['mirror'],
_source['rootpath'])),
ptrn = ptrn,
flags = _source_item['flags'])
else:
_source_item['fname'] = _item
return(_source_item)

def get_xsd(self):
path = os.path.join(os.path.dirname(__file__),
'bdisk.xsd')
with open(path, 'r') as f:
xsd = f.read()
if isinstance(self.xsd, lxml.etree.XMLSchema):
return(self.xsd)
if not self.xsd:
path = os.path.join(os.path.dirname(__file__), 'bdisk.xsd')
else:
path = os.path.abspath(os.path.expanduser(self.xsd))
with open(path, 'rb') as f:
xsd = lxml.etree.parse(f)
return(xsd)

def validate(self):
self.xsd = etree.XMLSchema(self.get_xsd())
return(self.xsd.validate(self.xml))
def parse_accounts(self):
## PROFILE/ACCOUNTS
self.cfg['users'] = []
# First we handle the root user, since it's a "special" case.
_root = self.profile.xpath('./accounts/rootpass')
self.cfg['root'] = transform.user(_root)
for user in self.profile.xpath('./accounts/user'):
_user = {'username': user.xpath('./username/text()')[0],
'sudo': transform.xml2py(user.attrib['sudo']),
'comment': None}
_comment = user.xpath('./comment/text()')
if len(_comment):
_user['comment'] = _comment[0]
_password = user.xpath('./password')
_user.update(transform.user(_password))
self.cfg['users'].append(_user)
return()

def get_profile(self):
"""Get a configuration profile.
def parse_all(self):
self.parse_profile()
self.parse_meta()
self.parse_accounts()
self.parse_sources()
self.parse_buildpaths()
self.parse_pki()
self.parse_gpg()
self.parse_sync()
return()

Get a configuration profile from the XML object and set that as a
profile object. If a profile is specified, attempt to find it. If not,
follow the default rules as specified in __init__.
"""
if self.profile:
# A profile identifier was provided
if isinstance(self.profile, str):
_profile_name = self.profile
self.profile = {}
for i in _profile_specifiers:
self.profile[i] = None
self.profile['name'] = _profile_name
elif isinstance(self.profile, dict):
for k in _profile_specifiers:
if k not in self.profile.keys():
self.profile[k] = None
def parse_buildpaths(self):
## PROFILE/BUILD(/PATHS)
self.cfg['build'] = {'paths': {}}
build = self.profile.xpath('./build')[0]
_optimize = build.get('its_full_of_stars', 'false')
self.cfg['build']['optimize'] = transform.xml2py(_optimize)
for path in build.xpath('./paths/*'):
self.cfg['build']['paths'][path.tag] = path.text
self.cfg['build']['basedistro'] = build.get('basedistro', 'archlinux')
# iso and ipxe are their own basic profile elements, but we group them
# in here because 1.) they're related, and 2.) they're simple to
# import. This may change in the future if they become more complex.
## PROFILE/ISO
self.cfg['iso'] = {'sign': None,
'multi_arch': None}
self.cfg['ipxe'] = {'sign': None,
'iso': None}
for x in ('iso', 'ipxe'):
# We enable all features by default.
elem = self.profile.xpath('./{0}'.format(x))[0]
for a in self.cfg[x]:
self.cfg[x][a] = transform.xml2py(elem.get(a, 'true'))
if x == 'ipxe':
self.cfg[x]['uri'] = elem.xpath('./uri/text()')[0]
return()

def parse_gpg(self):
## PROFILE/GPG
self.cfg['gpg'] = {'keyid': None,
'gnupghome': None,
'publish': None,
'prompt_passphrase': None,
'keys': []}
elem = self.profile.xpath('./gpg')[0]
for attr in elem.xpath('./@*'):
self.cfg['gpg'][attr.attrname] = transform.xml2py(attr)
for key in elem.xpath('./key'):
_keytpl = {'algo': 'rsa',
'keysize': '4096'}
_key = copy.deepcopy(_keytpl)
_key['name'] = None
_key['email'] = None
_key['comment'] = None
for attr in key.xpath('./@*'):
_key[attr.attrname] = transform.xml2py(attr)
for param in key.xpath('./*'):
if param.tag == 'subkey':
# We only support one subkey (for key generation).
if 'subkey' not in _key:
_key['subkey'] = copy.deepcopy(_keytpl)
for attr in param.xpath('./@*'):
_key['subkey'][attr.attrname] = transform.xml2py(attr)
print(_key)
else:
raise TypeError('profile must be a string (name of profile), '
'a dictionary, or None')
xpath = ('/bdisk/'
'profile{0}').format(_profile_xpath_gen(self.profile))
self.profile = self.xml.xpath(xpath)
if not self.profile:
raise RuntimeError('Could not find the profile specified in '
'the given configuration')
else:
# We need to find the default.
profiles = []
for p in self.xml.xpath('/bdisk/profile'):
profiles.append(p)
# Look for one named "default" or "DEFAULT" etc.
for idx, value in enumerate([e.attrib['name'].lower() \
for e in profiles]):
if value == 'default':
self.profile = copy.deepcopy(profiles[idx])
break
# We couldn't find a profile with a default name. Try to grab the
# first profile.
if self.profile is None:
# Grab the first profile.
if profiles:
self.profile = profile[0]
else:
# No profiles found.
raise RuntimeError('Could not find any usable '
'configuration profiles')
_key[param.tag] = transform.xml2py(param.text, attrib = False)
self.cfg['gpg']['keys'].append(_key)
return()

def parse_meta(self):
## PROFILE/META
# Get the various meta strings. We skip regexes (we handle those
# separately since they're unique'd per id attrib) and variables (they
# are already substituted by self.xml_suppl.substitute(x)).
_meta_iters = ('dev', 'names')
for t in _meta_iters:
self.cfg[t] = {}
_xpath = './meta/{0}'.format(t)
for e in self.profile.xpath(_xpath):
for se in e:
if not isinstance(se, lxml.etree._Comment):
self.cfg[t][se.tag] = transform.xml2py(se.text,
attrib = False)
for e in ('desc', 'uri', 'ver', 'max_recurse'):
_xpath = './meta/{0}/text()'.format(e)
self.cfg[e] = transform.xml2py(self.profile.xpath(_xpath)[0],
attrib = False)
# HERE is where we would handle regex patterns.
# But we don't, because they're in self.xml_suppl.btags['regex'].
#self.cfg['regexes'] = {}
#_regexes = self.profile.xpath('./meta/regexes/pattern')
#if len(_regexes):
# for ptrn in _regexes:
# self.cfg['regexes'][ptrn.attrib['id']] = re.compile(ptrn.text)
return()

def parse_pki(self):
## PROFILE/PKI
self.cfg['pki'] = {'clients': []}
elem = self.profile.xpath('./pki')[0]
self.cfg['pki']['overwrite'] = transform.xml2py(
elem.get('overwrite', 'false'))
ca = elem.xpath('./ca')[0]
clients = elem.xpath('./client')
self.cfg['pki']['ca'] = self.get_pki_obj(ca, 'ca')
for client in clients:
self.cfg['pki']['clients'].append(self.get_pki_obj(client,
'client'))
return()

def parse_profile(self):
## PROFILE
# The following are attributes of profiles that serve as identifiers.
self.cfg['profile'] = {'id': None,
'name': None,
'uuid': None}
for a in self.cfg['profile']:
if a in self.profile.attrib:
self.cfg['profile'][a] = transform.xml2py(
self.profile.attrib[a],
attrib = True)
# Small bug in transform.xml2py that we unfortunately can't fix, so we manually fix.
if 'id' in self.cfg['profile'] and isinstance(self.cfg['profile']['id'], bool):
self.cfg['profile']['id'] = int(self.cfg['profile']['id'])
return()

def parse_sources(self):
## PROFILE/SOURCES
self.cfg['sources'] = []
for source in self.profile.xpath('./sources/source'):
_source = {}
_source['arch'] = source.attrib['arch']
_source['mirror'] = source.xpath('./mirror/text()')[0]
_source['rootpath'] = source.xpath('./rootpath/text()')[0]
# The tarball, checksum, and sig components requires some...
# special care.
for e in ('tarball', 'checksum', 'sig'):
_source[e] = self.get_source(source, e, _source)
self.cfg['sources'].append(_source)
return()

def parse_sync(self):
## PROFILE/SYNC
self.cfg['sync'] = {}
elem = self.profile.xpath('./sync')[0]
# We populate defaults in case they weren't specified.
for e in ('gpg', 'ipxe', 'iso', 'tftp'):
self.cfg['sync'][e] = {'enabled': False,
'path': None}
sub = elem.xpath('./{0}'.format(e))[0]
for a in sub.xpath('./@*'):
self.cfg['sync'][e][a.attrname] = transform.xml2py(a)
self.cfg['sync'][e]['path'] = sub.text
rsync = elem.xpath('./rsync')[0]
self.cfg['sync']['rsync'] = {'enabled': False}
for a in rsync.xpath('./@*'):
self.cfg['sync']['rsync'][a.attrname] = transform.xml2py(a)
for sub in rsync.xpath('./*'):
self.cfg['sync']['rsync'][sub.tag] = transform.xml2py(
sub.text,
attrib = False)
return()

def validate(self, parsed = False):
xsd = self.get_xsd()
if not isinstance(xsd, lxml.etree.XMLSchema):
self.xsd = etree.XMLSchema(xsd)
else:
pass


# This would return a bool if it validates or not.
#self.xsd.validate(self.xml)
# We want to get a more detailed exception.
xml = etree.fromstring(self.xml_suppl.return_full())
self.xsd.assertValid(xml)
if parsed:
# We wait until after it's parsed to evaluate because otherwise we
# can't use utils.valid().
# We only bother with stuff that would hinder building, though -
# e.g. we don't check that profile's UUID is a valid UUID4.
# The XSD can catch a lot of stuff, but it's not so hot with things like URI validation,
# email validation, etc.
# URLs
for url in (self.cfg['uri'], self.cfg['dev']['website']):
if not valid.url(url):
raise ValueError('{0} is not a valid URL.'.format(url))
# Emails
for k in self.cfg['gpg']['keys']:
if not valid.email(k['email']):
raise ValueError('GPG key {0}: {1} is not a valid email address'.format(k['name'], k['email']))
if not valid.email(self.cfg['dev']['email']):
raise ValueError('{0} is not a valid email address'.format(self.cfg['dev']['email']))
if self.cfg['pki']:
if 'subject' in self.cfg['pki']['ca']:
if not valid.email(self.cfg['pki']['ca']['subject']['emailAddress']):
raise ValueError('{0} is not a valid email address'.format(
self.cfg['pki']['ca']['subject']['emailAddress']))
for cert in self.cfg['pki']['clients']:
if not cert['subject']:
continue
if not valid.email(cert['subject']['emailAddress']):
raise ValueError('{0} is not a valid email address'.format(cert['subject']['email']))
# Salts/hashes
if self.cfg['root']['salt']:
if not valid.salt_hash(self.cfg['root']['salt']):
raise ValueError('{0} is not a valid salt'.format(self.cfg['root']['salt']))
if self.cfg['root']['hashed']:
if not valid.salt_hash_full(self.cfg['root']['salt_hash'], self.cfg['root']['hash_algo']):
raise ValueError('{0} is not a valid hash of type {1}'.format(self.cfg['root']['salt_hash'],
self.cfg['root']['hash_algo']))
for u in self.cfg['users']:
if u['salt']:
if not valid.salt_hash(u['salt']):
raise ValueError('{0} is not a valid salt'.format(u['salt']))
if u['hashed']:
if not valid.salt_hash_full(u['salt_hash'], u['hash_algo']):
raise ValueError('{0} is not a valid hash of type {1}'.format(u['salt_hash'], u['hash_algo']))
# GPG Key IDs
if self.cfg['gpg']['keyid']:
if not valid.gpgkeyID(self.cfg['gpg']['keyid']):
raise ValueError('{0} is not a valid GPG Key ID/fingerprint'.format(self.cfg['gpg']['keyid']))
for s in self.cfg['sources']:
if 'sig' in s:
for k in s['sig']['keys']:
if not valid.gpgkeyID(k):
raise ValueError('{0} is not a valid GPG Key ID/fingerprint'.format(k))
return()

68
bdisk/env_prep.py
View File

@ -1,3 +1,67 @@
import copy
import importlib
import hashlib
import importlib # needed for the guest-os-specific stuff...
import os
from . import utils
from urllib.parse import urljoin


def hashsum_downloader(url, filename = None):
# TODO: support "latest" and "regex" flags? or remove from specs (since the tarball can be specified by these)?
# move that to the utils.DOwnload() class?
d = utils.Download(url, progress = False)
hashes = {os.path.basename(k):v for (v, k) in [line.split() for line in d.fetch().decode('utf-8').splitlines()]}
if filename:
if filename in hashes:
return(hashes[filename])
else:
raise KeyError('Filename {0} not in the list of hashes'.format(filename))
return(hashes)


class Prepper(object):
def __init__(self, dirs, sources, gpg = None):
# dirs is a ConfParse.cfg['build']['paths'] dict of dirs
self.CreateDirs(dirs)
# TODO: set up GPG env here so we can use it to import sig key and verify sources
for idx, s in enumerate(sources):
self._download(idx)

def CreateDirs(self, dirs):
for d in dirs:
os.makedirs(d, exist_ok = True)
return()


def _download(self, source_idx):
download = True
_source = self.cfg['sources'][source_idx]
_dest_dir = os.path.join(self.cfg['build']['paths']['cache'], source_idx)
_tarball = os.path.join(_dest_dir, _source['tarball']['fname'])
_remote_dir = urljoin(_source['mirror'], _source['rootpath'])
_remote_tarball = urljoin(_remote_dir + '/', _source['tarball']['fname'])
def _hash_verify(): # TODO: move to utils.valid()?
# Get a checksum.
if 'checksum' in _source:
if not _source['checksum']['explicit']:
_source['checksum']['value'] = hashsum_downloader(urljoin(_remote_dir + '/',
_source['checksum']['fname']))
if not _source['checksum']['hash_algo']:
_source['checksum']['hash_algo'] = utils.detect.any_hash(_source['checksum']['value'],
normalize = True)[0]
_hash = hashlib.new(_source['checksum']['hash_algo'])
with open(_tarball, 'rb') as f:
# It's potentially a large file, so we chunk it 64kb at a time.
_hashbuf = f.read(64000)
while len(_hashbuf) > 0:
_hash.update(_hashbuf)
_hashbuf = f.read(64000)
if _hash.hexdigest().lower() != _source['checksum']['value'].lower():
return(False)
return(True)
def _sig_verify(gpg_instance): # TODO: move to utils.valid()? or just use as part of the bdisk.GPG module?
pass
if os.path.isfile(_tarball):
download = _hash_verify()
download = _sig_verify()
if download:
d = utils.Download(_remote_tarball)

8
bdisk/main.py
View File

@ -1,4 +1,4 @@
#!/usr/bin/env python3.6
#!/usr/bin/env python3

import argparse
import confparse
@ -14,8 +14,10 @@ def parseArgs():
epilog = ('https://git.square-r00t.net'))
return(args)

def run():
pass
def run(cfg):
cfg = confparse.Conf(cfg, validate_cfg = True)
cfg.parse_all()


def run_interactive():
args = vars(parseArgs().parse_args())

396
bdisk/mtree.py Executable file
View File

@ -0,0 +1,396 @@
#!/usr/bin/env python3

import argparse
import copy
import datetime
import grp
import hashlib
import os
import pathlib
import platform
import pwd
import re
import stat
from collections import OrderedDict
try:
import pycksum
has_cksum = True
except ImportError:
has_cksum = False

# Parse BSD mtree spec files.
# On arch, BSD mtree is ported in the AUR as nmtree.
# TODO: add a generator class as well? (in process)
# TODO: add a checking function as well?

# The format used for headers
_header_strptime_fmt = '%a %b %d %H:%M:%S %Y'

# Supported hash types (for generation). These are globally available always.
_hashtypes = ['md5', 'sha1', 'sha256', 'sha384', 'sha512']
# If RIPEMD-160 is supported, we add it (after MD5).
if 'ripemd160' in hashlib.algorithms_available:
_hashtypes.insert(1, 'rmd160')

# Iterative to determine which type an item is.
_stype_map = {'block': stat.S_ISBLK,
'char': stat.S_ISCHR,
'dir': stat.S_ISDIR,
'fifo': stat.S_ISFIFO,
'file': stat.S_ISREG,
'link': stat.S_ISLNK,
'socket': stat.S_ISSOCK}

# Regex pattern for cleaning up an octal perm mode into a string representation.
_octre = re.compile('^0o')

class MTreeGen(object):
def __init__(self, path):
self.path = pathlib.PosixPath(os.path.abspath(os.path.expanduser(path)))
# These are used to keep a cached copy of the info.
self._sysinfo = {'uids': {}, 'gids': {}}
self._build_header()
# We use this to keep track of where we are exactly in the tree so we can generate a full absolute path at
# any moment relative to the tree.
self._path_pointer = copy.deepcopy(self.path)


def paths_iterator(self):
for root, dirs, files in os.walk(self.path):
for f in files:
_fname = self.path.joinpath(f)
_stats = self._get_stats(_fname)
if not _stats:
print(('WARNING: {0} either disappeared while we were trying to parse it or '
'it is a broken symlink.').format(_fname))
continue
# TODO: get /set line here?
item = ' {0} \\\n'.format(f)
_type = 'file' # TODO: stat this more accurately
_cksum = self._gen_cksum(_fname)
item += ' {0} {1} {2}\\\n'.format(_stats['size'],
_stats['time'],
('{0} '.format(_cksum) if _cksum else ''))
# TODO: here's where the hashes would get added
# TODO: here's where we parse dirs. maybe do that before files?
# remember: mtree specs use ..'s to traverse upwards when done with a dir
for d in dirs:
_dname = self.path.joinpath(d)
_stats = self._get_stats(_dname)
if not _stats:
print(('WARNING: {0} either disappeared while we were trying to parse it or '
'it is a broken symlink.').format(_dname))
continue
# TODO: get /set line here?
return()


def _gen_cksum(self, fpath):
if not has_cksum:
return(None)
if not os.path.isfile(fpath):
return(None)
# TODO: waiting on https://github.com/sobotklp/pycksum/issues/2 for byte iteration (because large files maybe?)
c = pycksum.Cksum()
with open(fpath, 'rb') as f:
c.add(f)
return(c.get_cksum())


def _get_stats(self, path):
stats = {}
try:
_st = os.stat(path, follow_symlinks = False)
except FileNotFoundError:
# Broken symlink? Shouldn't occur since follow_symlinks is False anyways, BUT...
return(None)
# Ownership
stats['uid'] = _st.st_uid
stats['gid'] = _st.st_gid
if _st.st_uid in self._sysinfo['uids']:
stats['uname'] = self._sysinfo['uids'][_st.st_uid]
else:
_pw = pwd.getpwuid(_st.st_uid).pw_name
stats['uname'] = _pw
self._sysinfo['uids'][_st.stuid] = _pw
if _st.st_gid in self._sysinfo['gids']:
stats['gname'] = self._sysinfo['gids'][_st.st_gid]
else:
_grp = grp.getgrgid(_st.st_gid).gr_name
stats['gname'] = _grp
self._sysinfo['gids'][_st.stgid] = _grp
# Type and Mode
for t in _stype_map:
if _stype_map[t](_st.st_mode):
stats['type'] = t
# TODO: need a reliable way of parsing this.
# for instance, for /dev/autofs, _st.st_dev = 6 (os.makedev(6) confirms major is 0, minor is 6)
# but netBSD mtree (ported) says it's "0xaeb" (2795? or, as str, "®b" apparently).
# I'm guessing the kernel determines this, but where is it pulling it from/how?
# We can probably do 'format,major,minor' (or, for above, 'linux,0,6').
# if t in ('block', 'char'):
# stats['device'] = None
# Handle symlinks.
if t == 'link':
_target = path
while os.path.islink(_target):
_target = os.path.realpath(_target)
stats['link'] = _target
break
stats['mode'] = '{0:0>4}'.format(_octre.sub('', str(oct(stat.S_IMODE(_st.st_mode)))))
stats['size'] = _st.st_size
stats['time'] = str(float(_st.st_mtime))
stats['nlink'] = _st.st_nlink
# TODO: "flags" keyword? is that meaningful on linux?
stats['flags'] = 'none'
return(stats)



def _gen_hashes(self, fpath):
hashes = OrderedDict({})
if not os.path.isfile(fpath):
return(hashes)
_hashnums = len(_hashtypes)
for idx, h in enumerate(_hashtypes):
# Stupid naming inconsistencies.
_hashname = (h if h is not 'rmd160' else 'ripemd160')
_hasher = hashlib.new(_hashname)
with open(fpath, 'rb') as f:
# Hash 64kb at a time in case it's a huge file. TODO: is this the most ideal chunk size?
_hashbuf = f.read(64000)
while len(_hashbuf) > 0:
_hasher.update(_hashbuf)
_hashbuf = f.read(64000)
hashes[h] = _hasher.hexdigest()
return(hashes)
# if idx + 1 < _hashnums:
# hashes += ' {0}={1} \\\n'.format(h, _hasher.hexdigest())
# else:
# hashes += ' {0}={1}\n'.format(h, _hasher.hexdigest())
# return(hashes)


def _build_header(self):
self.spec = ''
_header = OrderedDict({})
_header['user'] = pwd.getpwuid(os.geteuid()).pw_name
_header['machine'] = platform.node()
_header['tree'] = str(self.path)
_header['date'] = datetime.datetime.utcnow().strftime(_header_strptime_fmt)
for h in _header:
self.spec += '#\t{0:>7}: {1}\n'.format(h, _header[h])
self.spec += '\n'
return()



class MTreeParse(object):
def __init__(self, spec):
if not isinstance(spec, (str, bytes)):
raise ValueError('spec must be a raw string of the spec or a bytes object of the string')
if isinstance(spec, bytes):
try:
spec = spec.decode('utf-8')
except UnicodeDecodeError:
raise ValueError('spec must be a utf-8 encoded set of bytes if using byte mode')
self.orig_spec = copy.deepcopy(spec) # For referencing in case someone wanted to write it out.
# We NOW need to handle the escaped linebreaking it does.
self._specdata = re.sub('\\\\\s+', '', spec).splitlines()
self._get_header()
self.spec = {'header': self.header,
'paths': {}}
# Template for an item.
# Default keywords are:
# flags, gid, link, mode, nlink, size, time, type, uid
self._tplitem = {
'type': None, # ('block', 'char', 'dir', 'fifo', 'file', 'link', 'socket')
# checksum of file (if it's a file) (int)
# On all *nix platforms, the cksum(1) utility (which is what the mtree spec uses) follows
# the POSIX standard CRC (which is NOT CRC-1/CRC-16 nor CRC32!):
# http://pubs.opengroup.org/onlinepubs/009695299/utilities/cksum.html
# For a python implementation,
# https://stackoverflow.com/questions/6835381/python-equivalent-of-unix-cksum-function
# See also crcmod (in PyPi).
'cksum': None,
# "The device number to use for block or char file types." Should be converted to a tuple of one
# of the following:
# - (format(str), major(int), minor(int))
# - (format(str), major(int), unit(str?), subunit(str?)) (only used on bsdos formats)
# - (number(int?), ) ("opaque" number)
# Valid formats are, per man page of mtree:
# native, 386bsd, 4bsd, bsdos, freebsd, hpux, isc, linux, netbsd, osf1, sco, solaris, sunos,
# svr3, svr4, ultrix
'device': None,
# File flags as symbolic name. BSD-specific thing? TODO: testing on BSD system
'flags': [],
'ignore': False, # An mtree-internal flag to ignore hierarchy under this item
'gid': None, # The group ID (int)
'gname': None, # The group name (str)
'link': None, # The link target/source, if a link.
# The MD5 checksum digest (str? hex?). "md5digest" is a synonym for this, so it's consolidated in
# as the same keyword.
'md5': None,
# The mode (in octal) (we convert it to a python-native int for os.chmod/stat, etc.)
# May also be a symbolic value; TODO: map symbolic to octal/int.
'mode': None,
'nlink': None, # Number of hard links for this item.
'optional': False, # This item may or may not be present in the compared directory for checking.
'rmd160': None, # The RMD-160 checksum of the file. "rmd160digest" is a synonym.
'sha1': None, # The SHA-1 sum. "sha1digest" is a synonym.
'sha256': None, # SHA-2 256-bit checksum; "sha256digest" is a synonym.
'sha384': None, # SHA-2 384-bit checksum; "sha384digest" is a synonym.
'sha512': None, # SHA-2 512-bit checksum; "sha512digest" is a synonym.
'size': None, # Size of the file in bytes (int).
'tags': [], # mtree-internal tags (comma-separated in the mtree spec).
'time': None, # Time the file was last modified (in Epoch fmt as float).
'uid': None, # File owner UID (int)
'uname': None # File owner username (str)
# And lastly, "children" is where the children files/directories go. We don't include it in the template;
# it's added programmatically.
# 'children': {}
}
# Global aspects are handled by "/set" directives.
# They are restored by an "/unset". Since they're global and stateful, they're handled as a class attribute.
self.settings = copy.deepcopy(self._tplitem)
self._parse_items()
del(self.settings, self._tplitem)


def _get_header(self):
self.header = {}
_headre = re.compile('^#\s+(user|machine|tree|date):\s')
_cmtre = re.compile('^\s*#\s*')
_blklnre = re.compile('^\s*$')
for idx, line in enumerate(self._specdata):
if _headre.search(line): # We found a header item.
l = [i.lstrip() for i in _cmtre.sub('', line).split(':', 1)]
header = l[0]
val = (l[1] if l[1] is not '(null)' else None)
if header == 'date':
val = datetime.datetime.strptime(val, _header_strptime_fmt)
elif header == 'tree':
val = pathlib.PosixPath(val)
self.header[header] = val
elif _blklnre.search(line):
break # We've reached the end of the header. Otherwise...
else: # We definitely shouldn't be here, but this means the spec doesn't even have a header.
break
return()


def _parse_items(self):
# A pattern (compiled for performance) to match commands.
_stngsre = re.compile('^/(un)?set\s')
# Per the man page:
# "Empty lines and lines whose first non-whitespace character is a hash mark (‘#’) are ignored."
_ignre = re.compile('^(\s*(#.*)?)?$')
# The following regex is used to quickly and efficiently check for a synonymized hash name.
_hashre = re.compile('^(md5|rmd160|sha1|sha256|sha384|sha512)(digest)?$')
# The following regex is to test if we need to traverse upwards in the path.
_parentre = re.compile('^\.{,2}/?$')
# _curpath = self.header['tree']
_curpath = pathlib.PosixPath('/')
_types = ('block', 'char', 'dir', 'fifo', 'file', 'link', 'socket')
# This parses keywords. Used by both item specs and /set.
def _kwparse(kwline):
out = {}
for i in kwline:
l = i.split('=', 1)
if len(l) < 2:
l.append(None)
k, v = l
if v == 'none':
v = None
# These are represented as octals.
if k in ('mode', ):
# TODO: handle symbolic references too (e.g. rwxrwxrwx)
if v.isdigit():
v = int(v, 8) # Convert from the octal. This can then be used directly with os.chmod etc.
# These are represented as ints
elif k in ('uid', 'gid', 'cksum', 'nlink'):
if v.isdigit():
v = int(v)
# These are booleans (represented as True by their presence).
elif k in ('ignore', 'optional'):
v = True
# These are lists (comma-separated).
elif k in ('flags', 'tags'):
if v:
v = [i.strip() for i in v.split(',')]
# The following are synonyms.
elif _hashre.search(k):
k = _hashre.sub('\g<1>', k)
elif k == 'time':
v = datetime.datetime.fromtimestamp(float(v))
elif k == 'type':
if v not in _types:
raise ValueError('{0} not one of: {1}'.format(v, ', '.join(_types)))
out[k] = v
return(out)
def _unset_parse(unsetline):
out = {}
if unsetline[1] == 'all':
return(copy.deepcopy(self._tplitem))
for i in unsetline:
out[i] = self._tplitem[i]
return(out)
# The Business-End (TM)
for idx, line in enumerate(self._specdata):
_fname = copy.deepcopy(_curpath)
# Skip these lines
if _ignre.search(line):
continue
l = line.split()
if _parentre.search(line):
_curpath = _curpath.parent
elif not _stngsre.search(line):
# So it's an item, not a command.
_itemsettings = copy.deepcopy(self.settings)
_itemsettings.update(_kwparse(l[1:]))
if _itemsettings['type'] == 'dir':
# SOMEONE PLEASE let me know if there's a cleaner way to do this.
_curpath = pathlib.PosixPath(os.path.normpath(_curpath.joinpath(l[0])))
_fname = _curpath
else:
_fname = pathlib.PosixPath(os.path.normpath(_curpath.joinpath(l[0])))
self.spec['paths'][_fname] = _itemsettings
else:
# It's a command. We can safely split on whitespace since the man page specifies the
# values are not to contain whitespace.
# /set
if l[0] == '/set':
del(l[0])
self.settings.update(_kwparse(l))
# /unset
else:
self.settings.update(_unset_parse(l))
continue
return()


def parseArgs():
args = argparse.ArgumentParser(description = 'An mtree parser')
# TODO: support stdin piping
args.add_argument('specfile',
help = 'The path to the spec file to parse')
return(args)


# Allow to be run as a CLI utility as well.
def main():
args = vars(parseArgs().parse_args())
import os
with open(os.path.abspath(os.path.expanduser(args['specfile']))) as f:
mt = MTreeParse(f.read())
with open('/tmp/newspec', 'w') as f:
f.write('\n'.join(mt._specdata))
import pprint
import inspect
del(mt.orig_spec)
del(mt._specdata)
import shutil
pprint.pprint(inspect.getmembers(mt), width = shutil.get_terminal_size()[0])

if __name__ == '__main__':
main()

129
bdisk/prompt_strings.py Normal file
View File

@ -0,0 +1,129 @@
# These are *key* ciphers, for encrypting exported keys.
openssl_ciphers = ['aes128', 'aes192', 'aes256', 'bf', 'blowfish',
'camellia128', 'camellia192', 'camellia256', 'cast', 'des',
'des3', 'idea', 'rc2', 'seed']
# These are *hash algorithms* for cert digests.
openssl_digests = ['blake2b512', 'blake2s256', 'gost', 'md4', 'md5', 'mdc2',
'rmd160', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']

class PromptStrings(object):
gpg = {
'attribs': {
'algo': {
'text': 'the subkey\'s encryption type/algorithm',
# The following can ONLY be used for encryption, not signing: elg, cv
#'choices': ['rsa', 'dsa', 'elg', 'ed', 'cv', 'nistp', 'brainpool.1', 'secp.k1'],
'choices': ['rsa', 'dsa', 'ed', 'nist', 'brainpool.1', 'sec.k1'],
#'default': 'rsa'
'default': 'ed'
},
'keysize': {
'text': 'the subkey\'s key size (in bits)',
'choices': {
'rsa': ['1024', '2048', '4096'],
'dsa': ['768', '2048', '3072'],
#'elg': ['1024', '2048', '4096'], # Invalid for signing, etc.
'ed': ['25519'],
#'cv': ['25519'],
'nistp': ['256', '384', '521'],
'brainpool.1': ['256', '384', '512'],
'sec.k1': ['256']
},
'default': {
'rsa': '4096',
'dsa': '3072',
'ed': '25519',
'nistp': '521',
'brainpool.1': '512',
'sec.k1': '256'
}
}
},
'params': ['name', 'email', 'comment']
}
ssl = {
'attribs': {
'cert': {
'hash_algo': {
'text': ('What hashing algorithm do you want to use? '
'(Default is sha512.)'),
'prompt': 'Hashing algorithm: ',
'options': openssl_digests,
'default': 'aes256'
}
},
'key': {
'cipher': {
'text': ('What encryption algorithm/cipher do you want to '
'use? (Default is aes256.) Use "none" to specify '
'a key without a passphrase.'),
'prompt': 'Cipher: ',
'options': openssl_ciphers + ['none'],
'default': 'aes256'
},
'keysize': {
'text': ('What keysize/length (in bits) do you want the '
'key to be? (Default is 4096; much higher values '
'are possible but are untested and thus not '
'supported by this tool; feel free to edit the '
'generated configuration by hand.) (If the key '
'cipher is "none", this is ignored.)'),
'prompt': 'Keysize: ',
# TODO: do all openssl_ciphers support these sizes?
'options': ['1024', '2048', '4096'],
'default': 'aes256'
},
'passphrase': {
'text': ('What passphrase do you want to use for the key? '
'If you specified the cipher as "none", this is '
'ignored (you can just hit enter).'),
'prompt': 'Passphrase (will not echo back): ',
'options': None,
'default': ''
}
}
},
'paths': {
'cert': '(or read from) the certificate',
'key': '(or read from) the key',
'csr': ('(or read from) the certificate signing request (if '
'blank, we won\'t write to disk - the operation will '
'occur entirely in memory assuming we need to generate/'
'sign)')
},
'paths_ca': {
'index': ('(or read from) the CA (Certificate Authority) Database '
'index file (if left blank, one will not be used)'),
'serial': ('(or read from) the CA (Certificate Authority) '
'Database serial file (if left blank, one will not be '
'used)'),
},
'subject': {
'countryName': {
'text': ('the 2-letter country abbreviation (must conform to '
'ISO3166 ALPHA-2)?\n'
'Country code: ')
},
'localityName': {
'text': ('the city/town/borough/locality name?\n'
'Locality: ')
},
'stateOrProvinceName': {
'text': ('the state/region name (full string)?\n'
'Region: ')
},
'organization': {
'text': ('your organization\'s name?\n'
'Organization: ')
},
'organizationalUnitName': {
'text': ('your department/role/team/department name?\n'
'Organizational Unit: ')
},
'emailAddress': {
'text': ('the email address to be associated with this '
'certificate/PKI object?\n'
'Email: ')
}
}
}

599
bdisk/utils.py
View File

@ -1,17 +1,25 @@
# Yes, this is messy. They doesn't belong anywhere else, leave me alone.

import _io
import copy
import crypt
import GPG
import getpass
import hashid
import hashlib
import iso3166
import os
import pprint
import prompt_strings
import re
import string
import uuid
import validators
import zlib
import requests
import lxml.etree
import lxml.objectify
from bs4 import BeautifulSoup
from collections import OrderedDict
from dns import resolver
from email.utils import parseaddr as emailparse
@ -25,6 +33,7 @@ passlib_schemes = ['des_crypt', 'md5_crypt', 'sha256_crypt', 'sha512_crypt']
# Build various hash digest name lists
digest_schemes = list(hashlib.algorithms_available)
# Provided by zlib
# TODO?
digest_schemes.append('adler32')
digest_schemes.append('crc32')

@ -33,12 +42,53 @@ crypt_map = {'sha512': crypt.METHOD_SHA512,
'md5': crypt.METHOD_MD5,
'des': crypt.METHOD_CRYPT}

# These are *key* ciphers, for encrypting exported keys.
openssl_ciphers = ['aes128', 'aes192', 'aes256', 'bf', 'blowfish',
'camellia128', 'camellia192', 'camellia256', 'cast', 'des',
'des3', 'idea', 'rc2', 'seed']
openssl_digests = ['blake2b512', 'blake2s256', 'gost', 'md4', 'md5', 'mdc2',
'rmd160', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']

class Download(object):
def __init__(self, url, progress = True, offset = None, chunksize = 1024):
self.cnt_len = None
self.head = requests.head(url, allow_redirects = True).headers
self.req_headers = {}
self.range = False
self.url = url
self.offset = offset
self.chunksize = chunksize
self.progress = progress
if 'accept-ranges' in self.head:
if self.head['accept-ranmges'].lower() != 'none':
self.range = True
if 'content-length' in self.head:
try:
self.cnt_len = int(self.head['content-length'])
except TypeError:
pass
if self.cnt_len and self.offset and self.range:
if not self.offset <= self.cnt_len:
raise ValueError(('The offset requested ({0}) is greater than '
'the content-length value').format(self.offset, self.cnt_len))
self.req_headers['range'] = 'bytes={0}-'.format(self.offset)

def fetch(self):
if not self.progress:
self.req = requests.get(self.url, allow_redirects = True, headers = self.req_headers)
self.bytes_obj = self.req.content
else:
self.req = requests.get(self.url, allow_redirects = True, stream = True, headers = self.req_headers)
self.bytes_obj = bytes()
_bytelen = 0
# TODO: better handling for logging instead of print()s?
for chunk in self.req.iter_content(chunk_size = self.chunksize):
self.bytes_obj += chunk
if self.cnt_len:
print('\033[F')
print('{0:.2f}'.format((_bytelen / float(self.head['content-length'])) * 100),
end = '%',
flush = True)
_bytelen += self.chunksize
else:
print('.', end = '')
print()
return(self.bytes_obj)


class XPathFmt(string.Formatter):
def get_field(self, field_name, args, kwargs):
@ -51,18 +101,19 @@ class detect(object):
def __init__(self):
pass

def any_hash(self, hash_str):
def any_hash(self, hash_str, normalize = False):
h = hashid.HashID()
hashes = []
for i in h.identifyHash(hash_str):
if i.extended:
continue
x = i.name
if x.lower() in ('crc-32', 'ripemd-160', 'sha-1', 'sha-224',
'sha-256', 'sha-384', 'sha-512'):
if x.lower() in ('crc-32', 'ripemd-160', 'sha-1', 'sha-224', 'sha-256', 'sha-384', 'sha-512'):
# Gorram you, c0re.
x = re.sub('-', '', x.lower())
_hashes = [h.lower() for h in digest_schemes]
_hashes = [h.lower() for h in digest_schemes] # TODO: move this outside so we don't define it every invoke
if normalize:
x = re.sub('(-|crypt|\s+)', '', x.lower())
if x.lower() in sorted(list(set(_hashes))):
hashes.append(x)
return(hashes)
@ -76,9 +127,45 @@ class detect(object):
return(None)
return()

def password_hash_salt(self, salted_hash):
_hash_list = salted_hash.split('$')
if len(_hash_list) < 3:
return(None)
salt = _hash_list[2]
return(salt)

def remote_files(self, url_base, ptrn = None, flags = []):
soup = BeautifulSoup(Download(url_base, progress = False).fetch().decode('utf-8'),
'lxml')
urls = []
if 'regex' in flags:
if not isinstance(ptrn, str):
raise ValueError('"ptrn" must be a regex pattern to match '
'against')
else:
ptrn = re.compile(ptrn)
for u in soup.find_all('a'):
if not u.has_attr('href'):
continue
if 'regex' in flags:
if not ptrn.search(u.attrs['href']):
continue
if u.has_attr('href'):
urls.append(u.attrs['href'])
if not urls:
return(None)
# We certainly can't intelligently parse the printed timestamp since it
# varies so much and that'd be a nightmare to get consistent...
# But we CAN sort by filename.
if 'latest' in flags:
urls = sorted(list(set(urls)))
urls = urls[-1]
else:
urls = urls[0]
return(urls)

def gpgkeyID_from_url(self, url):
with urlopen(url) as u:
data = u.read()
data = Download(url, progress = False).bytes_obj
g = GPG.GPGHandler()
key_ids = g.get_sigs(data)
del(g)
@ -130,7 +217,7 @@ class detect(object):
# Get any easy ones out of the way first.
if name in digest_schemes:
return(name)
# Otherwise grab the first one that matches, in order from the .
# Otherwise grab the first one that matches
_digest_re = re.compile('^{0}$'.format(name.strip()), re.IGNORECASE)
for h in digest_schemes:
if _digest_re.search(h):
@ -146,6 +233,9 @@ class generate(object):
_salt = crypt.mksalt(algo)
else:
_salt = salt
if not password:
# Intentionally empty password.
return('')
return(crypt.crypt(password, _salt))

def hashlib_names(self):
@ -156,13 +246,18 @@ class generate(object):
hashes.append(h)
return(hashes)

def salt(self, algo = 'sha512'):
def salt(self, algo = None):
if not algo:
algo = 'sha512'
algo = crypt_map[algo]
return(crypt.mksalt(algo))

class prompts(object):
def __init__(self):
pass
self.promptstr = prompt_strings.PromptStrings()

# TODO: these strings should be indexed in a separate module and
# sourced/imported. we should generally just find a cleaner way to do this.

def confirm_or_no(self, prompt = '', invert = False,
usage = '{0} to confirm, otherwise {1}...\n'):
@ -194,21 +289,18 @@ class prompts(object):
return(True)

def gpg_keygen_attribs(self):
_attribs = {'algo': {'text': 'the subkey\'s encryption type/algorithm',
'choices': ['rsa', 'dsa'],
'default': 'rsa'},
'keysize': {'text': 'the subkey\'s key size (in bits)',
'choices': {'rsa': ['1024', '2048', '4096'],
'dsa': ['768', '2048', '3072']},
'default': {'rsa': '4096',
'dsa': '3072'}}}
_params = {'name': None,
'email': None,
#'email': valid().email, # Use this to force valid email.
'comment': None}
_strs = self.promptstr.gpg
gpg_vals = {'attribs': {},
'params': {}}
for a in _attribs:
_checks = {
'params': {
'name': {'error': 'name cannot be empty',
'check': valid().nonempty_str},
'email': {'error': 'not a valid email address',
'check': valid().email}
}
}
for a in _strs['attribs']:
_a = None
while not _a:
if 'algo' in gpg_vals['attribs'] and a == 'keysize':
@ -261,6 +353,45 @@ class prompts(object):
continue
else:
gpg_vals['params'][p] = _p
=======
_choices = _strs['attribs']['keysize']['choices'][_algo]
_dflt = _strs['attribs']['keysize']['default'][_algo]
else:
_choices = _strs['attribs'][a]['choices']
_dflt = _strs['attribs'][a]['default']
_a = (input(
('\nWhat should be {0}? (Default is {1}.)\nChoices:\n'
'\n\t{2}\n\n{3}: ').format(
_strs['attribs'][a]['text'],
_dflt,
'\n\t'.join(_choices),
a.title()
)
)).strip().lower()
if _a == '':
_a = _dflt
elif _a not in _choices:
print(
('Invalid selection; choosing default '
'({0})').format(_dflt)
)
_a = _dflt
gpg_vals['attribs'][a] = _a
for p in _strs['params']:
_p = input(
('\nWhat is the {0} for the subkey?\n'
'{1}: ').format(p, p.title())
)
if p in _checks['params']:
while not _checks['params'][p]['check'](_p):
print(
('Invalid entry ({0}). Please retry.').format(
_checks['params'][p]['error']
)
)
_p = input('{0}: '.format(_p.title()))
gpg_vals['params'][p] = _p
>>>>>>> 69b6ec60d05d64a9e23e9a0707a0323f960a2936
return(gpg_vals)

def hash_select(self, prompt = '',
@ -310,116 +441,81 @@ class prompts(object):
ssl_vals = {'paths': {},
'attribs': {},
'subject': {}}
_checks = {
'subject': {
'countryName': valid().country_abbrev,
'emailAddress': valid().email
}
}
_strs = copy.deepcopy(self.promptstr.ssl)
# pki_role should be 'ca' or 'client'
if pki_role not in ('ca', 'client'):
raise ValueError('pki_role must be either "ca" or "client"')
_attribs = {'cert': {'hash_algo': {'text': ('What hashing algorithm '
'do you want to use? (Default is sha512.)'),
'prompt': 'Hashing algorithm: ',
'options': openssl_digests,
'default': 'sha512'}},
'key': {'cipher': {'text': ('What encryption algorithm/'
'cipher do you want to use? (Default is '
'aes256.)'),
'prompt': 'Cipher: ',
'options': openssl_ciphers,
'default': 'aes256'},
# This can actually theoretically be anywhere from
# 512 to... who knows how high. I couldn't find the
# upper bound. So we just set it to sensible
# defaults. If they want something higher, they can
# edit the XML when they're done.
'keysize': {'text': ('What keysize/length (in '
'bits) do you want the key to be? (Default is '
'4096; much higher values are possible but '
'are untested and thus not supported by this '
'tool; feel free to edit the generated '
'configuration by hand.)'),
'prompt': 'Keysize: ',
'options': ['1024', '2048', '4096'],
'default': '4096'}}}
_paths = {'cert': '(or read from) the certificate',
'key': '(or read from) the key',
'csr': ('(or read from) the certificate signing request (if '
'blank, we won\'t write to disk - the operation '
'will occur entirely in memory assuming we need to '
'generate/sign)')}
# NOTE: need to validate US and email
if pki_role == 'ca':
_paths['index'] = ('(or read from) the CA DB index file (if left '
'blank, one will not be used)')
_paths['serial'] = ('(or read from) the CA DB serial file (if '
'left blank, one will not be used)')
for a in _attribs:
# this is getting triggered for clients?
_strs['paths'].update(_strs['paths_ca'])
for a in _strs['attribs']:
ssl_vals['attribs'][a] = {}
for x in _attribs[a]:
for x in _strs['attribs'][a]:
ssl_vals['attribs'][a][x] = None
for p in _paths:
for p in _strs['paths']:
if p == 'csr':
_allow_empty = True
else:
_allow_empty = False
ssl_vals['paths'][p] = self.path(_paths[p],
ssl_vals['paths'][p] = self.path(_strs['paths'][p],
empty_passthru = _allow_empty)
print()
if ssl_vals['paths'][p] == '':
ssl_vals['paths'][p] = None
if p in _attribs:
for x in _attribs[p]:
if p in _strs['attribs']:
for x in _strs['attribs'][p]:
while not ssl_vals['attribs'][p][x]:
# cipher attrib is prompted for before this.
if p == 'key' and x == 'passphrase':
if ssl_vals['attribs']['key']['cipher'] == 'none':
ssl_vals['attribs'][p][x] = 'none'
continue
ssl_vals['attribs'][p][x] = getpass.getpass(
('{0}\n{1}').format(
_strs['attribs'][p][x]['text'],
_strs['attribs'][p][x]['prompt'])
)
if ssl_vals['attribs'][p][x] == '':
ssl_vals['attribs'][p][x] = 'none'
else:
ssl_vals['attribs'][p][x] = (input(
('\n{0}\n\n\t{1}\n\n{2}').format(
_attribs[p][x]['text'],
'\n\t'.join(_attribs[p][x]['options']),
_attribs[p][x]['prompt'])
)).strip().lower()
_strs['attribs'][p][x]['text'],
'\n\t'.join(
_strs['attribs'][p][x]['options']),
_strs['attribs'][p][x]['prompt']))
).strip().lower()
if ssl_vals['attribs'][p][x] not in \
_attribs[p][x]['options']:
print(('\nInvalid selection; setting default '
'({0}).').format(_attribs[p][x]['default']))
_strs['attribs'][p][x]['options']:
print(
('\nInvalid selection; setting default '
'({0}).').format(
_strs['attribs'][p][x]['default']
)
)
ssl_vals['attribs'][p][x] = \
_attribs[p][x]['default']
_subject = {'countryName': {'text': ('the 2-letter country '
'abbreviation (must conform to '
'ISO3166 ALPHA-2)?\nCountry '
'code: '),
'check': 'func',
'func': valid().country_abbrev},
'localityName': {'text': ('the city/town/borough/locality '
'name?\nLocality: '),
'check': None},
'stateOrProvinceName': {'text': ('the state/region '
'name (full string)?'
'\nRegion: '),
'check': None},
'organization': {'text': ('your organization\'s name?'
'\nOrganization: '),
'check': None},
'organizationalUnitName': {'text': ('your department/role/'
'team/department name?'
'\nOrganizational '
'Unit: '),
'check': None},
'emailAddress': {'text': ('the email address to be '
'associated with this '
'certificate/PKI object?\n'
'Email: '),
'check': 'func',
'func': valid().email}}
for s in _subject:
_strs['attribs'][p][x]['default']
for s in _strs['subject']:
ssl_vals['subject'][s] = None
for s in _subject:
for s in _strs['subject']:
while not ssl_vals['subject'][s]:
_input = (input(
('\nWhat is {0}').format(_subject[s]['text'])
('\nWhat is {0}').format(
_strs['subject'][s]['text'])
)).strip()
_chk = _subject[s]['check']
if _chk:
if _chk == 'func':
_chk = _subject[s]['func'](_input)
if not _chk:
print('Invalid value; retrying.')
continue
print()
if s in _checks['subject']:
if not _checks['subject'][s](_input):
print('Invalid entry; try again.')
ssl_vals['subject'][s] = None
continue
ssl_vals['subject'][s] = _input
_url = transform().url_to_dict(cn_url, no_None = True)
ssl_vals['subject']['commonName'] = _url['host']
@ -454,12 +550,12 @@ class transform(object):
def py2xml(self, value, attrib = True):
if value in (False, ''):
if attrib:
return("no")
return("false")
else:
return(None)
elif isinstance(value, bool):
# We handle the False case above.
return("yes")
return("true")
elif isinstance(value, str):
return(value)
else:
@ -477,7 +573,6 @@ class transform(object):
text_out = re.sub('[^\w]', '', text_out)
return(text_out)

# noinspection PyDictCreation
def url_to_dict(self, orig_url, no_None = False):
def _getuserinfo(uinfo_str):
if len(uinfo_str) == 0:
@ -622,6 +717,88 @@ class transform(object):
url['full_url'] += '#{0}'.format('#'.join(_f))
return(url)

def user(self, user_elem):
_attribs = ('hashed', 'hash_algo', 'salt')
acct = {}
for a in _attribs:
acct[a] = None
if len(user_elem):
elem = user_elem[0]
for a in _attribs:
if a in elem.attrib:
acct[a] = self.xml2py(elem.attrib[a], attrib = True)
if acct['hashed']:
if not acct['hash_algo']:
_hash = detect().password_hash(elem.text)
if _hash:
acct['hash_algo'] = _hash
else:
acct['hash_algo'] = None
# We no longer raise ValueError. Per shadow(5):
#######################################################
# If the password field contains some string that is
# not a valid result of crypt(3), for instance ! or *,
# the user will not be able to use a unix password to
# log in (but the user may log in the system by other
# means).
# This field may be empty, in which case no passwords
# are required to authenticate as the specified login
# name. However, some applications which read the
# /etc/shadow file may decide not to permit any access
# at all if the password field is empty.
# A password field which starts with an exclamation
# mark means that the password is locked. The remaining
# characters on the line represent the password field
# before the password was locked.
#######################################################
# raise ValueError(
# 'Invalid salted password hash: {0}'.format(
# elem.text)
# )
acct['salt_hash'] = elem.text
acct['passphrase'] = None
else:
if not acct['hash_algo']:
acct['hash_algo'] = 'sha512'
acct['passphrase'] = elem.text
_saltre = re.compile('^\s*(auto|none|)\s*$', re.IGNORECASE)
if acct['salt']:
if _saltre.search(acct['salt']):
_salt = generate.salt(acct['hash_algo'])
acct['salt'] = _salt
else:
if not acct['hashed']:
acct['salt_hash'] = generate().hash_password(
acct['passphrase'],
algo = crypt_map[acct['hash_algo']])
acct['salt'] = detect().password_hash_salt(acct['salt_hash'])
if 'salt_hash' not in acct:
acct['salt_hash'] = generate().hash_password(
acct['passphrase'],
salt = acct['salt'],
algo = crypt_map[acct['hash_algo']])
return(acct)

def xml2py(self, value, attrib = True):
yes = re.compile('^\s*(true|1)\s*$', re.IGNORECASE)
no = re.compile('^\s*(false|0)\s*$', re.IGNORECASE)
none = re.compile('^\s*(none|)\s*$', re.IGNORECASE)
if no.search(value):
if attrib:
return(False)
else:
return(None)
elif yes.search(value):
# We handle the False case above.
return(True)
elif value.strip() == '' or none.search(value):
return(None)
elif valid().integer(value):
return(int(value))
else:
return(value)
return()

class valid(object):
def __init__(self):
pass
@ -664,6 +841,11 @@ class valid(object):
return(False)
return()

def nonempty_str(self, str_in):
if str_in.strip() == '':
return(False)
return(True)

def password(self, passwd):
# https://en.wikipedia.org/wiki/ASCII#Printable_characters
# https://serverfault.com/a/513243/103116
@ -693,14 +875,19 @@ class valid(object):
return(True)

def salt_hash(self, salthash):
_idents = ''.join([i.ident for i in crypt_map if i.ident])
_idents = ''.join([i.ident for i in crypt_map.values() if i.ident])
# noinspection PyStringFormat
_regex = re.compile('^(\$[{0}]\$)?[./0-9A-Za-z]{{0,16}}\$?'.format(
_idents))
_regex = re.compile('^(\$[{0}]\$)?[./0-9A-Za-z]{{0,16}}\$?'.format(_idents))
if not _regex.search(salthash):
return(False)
return(True)

def salt_hash_full(self, salthash, hash_type):
h = [re.sub('-', '', i.lower()).split()[0] for i in detect.any_hash(self, salthash, normalize = True)]
if hash_type.lower() not in h:
return(False)
return(True)

def plugin_name(self, name):
if len(name) == 0:
return(False)
@ -748,22 +935,38 @@ class valid(object):

class xml_supplicant(object):
def __init__(self, cfg, profile = None, max_recurse = 5):
raw = self._detect_cfg(cfg)
xmlroot = lxml.etree.fromstring(raw)
self.selector_ids = ('id', 'name', 'uuid')
self.btags = {'xpath': {},
'regex': {},
'variable': {}}
raw = self._detect_cfg(cfg)
# This is changed in just a moment.
self.profile = profile
# This is retained so we can "refresh" the profile if needed.
self.orig_profile = profile
try:
self.orig_xml = lxml.etree.fromstring(raw)
# We need to strip the naked namespace for XPath to work.
self.xml = copy.deepcopy(self.orig_xml)
self.roottree = self.xml.getroottree()
self.tree = self.roottree.getroot()
self.strip_naked_ns()
except lxml.etree.XMLSyntaxError:
raise ValueError('The configuration provided does not seem to be '
'valid')
self.get_profile(profile = profile)
# This is disabled; we set it above.
#self.xml = lxml.etree.fromstring(raw)
self.fmt = XPathFmt()
self.max_recurse = max_recurse
self.max_recurse = int(self.profile.xpath(
'//meta/max_recurse/text()')[0])
# I don't have permission to credit them, but to the person who helped
# me with this regex - thank you. You know who you are.
# Originally this pattern was the one from:
# https://stackoverflow.com/a/12728199/733214
self.ptrn = re.compile(('(?<=(?<!\{)\{)(?:[^{}]+'
'|{{[^{}]*}})*(?=\}(?!\}))'))
self.root = lxml.etree.ElementTree(xmlroot)
if not profile:
self.profile = xmlroot.xpath('/bdisk/profile[1]')[0]
else:
self.profile = xmlroot.xpath(profile)[0]
self.root = lxml.etree.ElementTree(self.xml)
self._parse_regexes()
self._parse_variables()
@ -796,17 +999,97 @@ class xml_supplicant(object):
return(cfg)

def _parse_regexes(self):
for regex in self.profile.xpath('//meta/regexes/pattern'):
self.btags['regex'][regex.attrib['id']] = re.compile(regex.text)
for regex in self.profile.xpath('./meta/regexes/pattern'):
_key = 'regex%{0}'.format(regex.attrib['id'])
self.btags['regex'][_key] = regex.text
return()

def _parse_variables(self):
for variable in self.profile.xpath('//meta/variables/variable'):
for variable in self.profile.xpath('./meta/variables/variable'):
self.btags['variable'][
'variable%{0}'.format(variable.attrib['id'])
] = variable.text
return()

def btags_to_dict(self, text_in):
d = {}
ptrn_id = self.ptrn.findall(text_in)
if len(ptrn_id) >= 1:
for item in ptrn_id:
try:
btag, expr = item.split('%', 1)
if btag not in self.btags:
continue
if item not in self.btags[btag]:
self.btags[btag][item] = None
#self.btags[btag][item] = expr # remove me?
if btag == 'xpath':
d[item] = (btag, expr)
elif btag == 'variable':
d[item] = (btag, self.btags['variable'][item])
except ValueError:
return(d)
return(d)

def get_profile(self, profile = None):
"""Get a configuration profile.

Get a configuration profile from the XML object and set that as a
profile object. If a profile is specified, attempt to find it. If not,
follow the default rules as specified in __init__.
"""
if profile:
# A profile identifier was provided
if isinstance(profile, str):
_profile_name = profile
profile = {}
for i in self.selector_ids:
profile[i] = None
profile['name'] = _profile_name
elif isinstance(profile, dict):
for k in self.selector_ids:
if k not in profile.keys():
profile[k] = None
else:
raise TypeError('profile must be a string (name of profile), '
'a dictionary, or None')
xpath = '/bdisk/profile{0}'.format(self.xpath_selector(profile))
self.profile = self.xml.xpath(xpath)
if len(self.profile) != 1:
raise ValueError('Could not determine a valid, unique '
'profile; please check your profile '
'specifier(s)')
else:
# We need the actual *profile*, not a list of matching
# profile(s).
self.profile = self.profile[0]
if not len(self.profile):
raise RuntimeError('Could not find the profile specified in '
'the given configuration')
else:
# We need to find the default.
profiles = []
for p in self.xml.xpath('/bdisk/profile'):
profiles.append(p)
# Look for one named "default" or "DEFAULT" etc.
for idx, value in enumerate([e.attrib['name'].lower() \
for e in profiles]):
if value == 'default':
#self.profile = copy.deepcopy(profiles[idx])
self.profile = profiles[idx]
break
# We couldn't find a profile with a default name. Try to grab the
# first profile.
if self.profile is None:
# Grab the first profile.
if profiles:
self.profile = profiles[0]
else:
# No profiles found.
raise RuntimeError('Could not find any usable '
'configuration profiles')
return()

def get_path(self, element):
path = element
try:
@ -818,6 +1101,33 @@ class xml_supplicant(object):
).format(element.text))
return(path)

def return_full(self):
# https://stackoverflow.com/a/22553145/733214
local_xml = lxml.etree.Element('bdisk',
nsmap = self.orig_xml.nsmap,
attrib = self.orig_xml.attrib)
local_xml.text = '\n '
for elem in self.xml.xpath('/bdisk/profile'):
local_xml.append(copy.deepcopy(elem))
return(lxml.etree.tostring(local_xml))

def return_naked_ns(self):
# It's so stupid I have to do this.
return(self.orig_xml.nsmap)

def strip_naked_ns(self):
# I cannot *believe* that LXML doesn't have this built-in, considering
# how common naked namespaces are.
# https://stackoverflow.com/a/18160164/733214
for elem in self.roottree.getiterator():
if not hasattr(elem.tag, 'find'):
continue
i = elem.tag.find('}')
if i >= 0:
elem.tag = elem.tag[i + 1:]
lxml.objectify.deannotate(self.roottree, cleanup_namespaces = True)
return()

def substitute(self, element, recurse_count = 0):
if recurse_count >= self.max_recurse:
return(element)
@ -858,33 +1168,10 @@ class xml_supplicant(object):
_dictmap = self.btags_to_dict(element.text)
return(element)

def xpath_selector(self, selectors,
selector_ids = ('id', 'name', 'uuid')):
def xpath_selector(self, selectors):
# selectors is a dict of {attrib:value}
xpath = ''
for i in selectors.items():
if i[1] and i[0] in selector_ids:
if i[1] and i[0] in self.selector_ids:
xpath += '[@{0}="{1}"]'.format(*i)
return(xpath)

def btags_to_dict(self, text_in):
d = {}
ptrn_id = self.ptrn.findall(text_in)
if len(ptrn_id) >= 1:
for item in ptrn_id:
try:
btag, expr = item.split('%', 1)
if btag not in self.btags:
continue
if item not in self.btags[btag]:
self.btags[btag][item] = None
#self.btags[btag][item] = expr # remove me?
if btag == 'xpath':
d[item] = (btag, expr)
elif btag == 'variable':
d[item] = (btag, self.btags['variable'][item])
except ValueError:
return(d)
return(d)



3
bin/xmllint.sh Executable file
View File

@ -0,0 +1,3 @@
#!/bin/bash

xmllint -schema /opt/dev/bdisk/bdisk/bdisk.xsd /opt/dev/bdisk/docs/examples/multi_profile.xml --noout

123
docs/examples/multi_profile.xml
View File

@ -1,15 +1,16 @@
<?xml version='1.0' encoding='UTF-8'?>
<bdisk>
<bdisk xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://bdisk.square-r00t.net/" xsi:schemaLocation="http://bdisk.square-r00t.net bdisk.xsd">
<profile name="default" id="1" uuid="8cdd6bcb-c147-4a63-9779-b5433c510dbc">
<meta>
<names>
<name>BDisk</name>
<name>BDISK</name>
<!--<name>{xpath%../uxname/text()}</name>-->
<uxname>bdisk</uxname>
<!-- Just like with previous versions of BDisk, you can reference other values...
but now with the neat benefits of XPath! Everything you could do in build.ini's and more.
See https://www.w3schools.com/xml/xpath_syntax.asp
If you need a literal curly brace, double them (e.g. for "{foo}", use "{{foo}}"),
UNLESS it's in a {regex%...} placeholder/filter (as part of the expression). -->
UNLESS it's in a <regexes><pattern> as part of the expression. Those are taken as literal strings. -->
<pname>{xpath%../name/text()}</pname>
</names>
<desc>A rescue/restore live environment.</desc>
@ -24,7 +25,7 @@
<!-- If the maximum level is reached, the substitution will evaluate as blank. -->
<max_recurse>5</max_recurse>
<!-- You need to store regex patterns here and reference them in a special way later, and it's only valid for certain
items. See the manual for more information. -->
items. See the manual for more information. NO btags within the patterns is allowed. -->
<regexes>
<pattern id="tarball_x86_64">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-x86_64\.tar\.gz$</pattern>
<pattern id="sig_x86_64">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-x86_64\.tar\.gz\.sig$</pattern>
@ -38,37 +39,37 @@
</meta>
<accounts>
<!-- Salted/hashed password is "test" -->
<rootpass hashed="yes">$6$7KfIdtHTcXwVrZAC$LZGNeMNz7v5o/cYuA48FAxtZynpIwO5B1CPGXnOW5kCTVpXVt4SypRqfM.AoKkFt/O7MZZ8ySXJmxpELKmdlF1</rootpass>
<user sudo="yes">
<rootpass hashed="true">$6$7KfIdtHTcXwVrZAC$LZGNeMNz7v5o/cYuA48FAxtZynpIwO5B1CPGXnOW5kCTVpXVt4SypRqfM.AoKkFt/O7MZZ8ySXJmxpELKmdlF1</rootpass>
<user sudo="true">
<username>{xpath%//meta/names/uxname/text()}</username>
<!-- You can also use substitution from different profiles in this same configuration: -->
<!-- <username>{xpath%//profile[@name='another_profile']/meta/names/uxname"}</username> -->
<comment>{xpath%//meta/dev/author/text()}</comment>
<password hashed="no" hash_algo="sha512" salt="auto">testpassword</password>
<password hashed="false" hash_algo="sha512" salt="auto">testpassword</password>
</user>
<user sudo="no">
<user sudo="false">
<username>testuser</username>
<name>Test User</name>
<password hashed="no" hash_algo="sha512" salt="auto">anothertestpassword</password>
<comment>Test User</comment>
<password hashed="false" hash_algo="sha512" salt="auto">anothertestpassword</password>
</user>
</accounts>
<sources>
<source arch="x86_64">
<mirror>http://archlinux.mirror.domain.tld</mirror>
<rootpath>/iso/latest</rootpath>
<tarball flags="regex,latest">{regex%tarball_x86_64}</tarball>
<checksum hash_algo="sha1" flags="none">sha1sums.txt</checksum>
<sig keys="7F2D434B9741E8AC" keyserver="hkp://pool.sks-keyservers.net" flags="regex,latest">{regex%sig_x86_64}</sig>
<tarball flags="regex latest">{regex%tarball_x86_64}</tarball>
<checksum hash_algo="sha1" explicit="false" flags="latest">sha1sums.txt</checksum>
<sig keys="7F2D434B9741E8AC" keyserver="hkp://pool.sks-keyservers.net" flags="regex latest">{regex%sig_x86_64}</sig>
</source>
<source arch="i686">
<mirror>http://archlinux32.mirror.domain.tld</mirror>
<rootpath>/iso/latest</rootpath>
<tarball flags="regex,latest">{regex%tarball_i686}</tarball>
<checksum hash_algo="sha512" explicit="yes">cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e</checksum>
<sig keys="248BF41F9BDD61D41D060AE774EDA3C6B06D0506" keyserver="hkp://pool.sks-keyservers.net" flags="regex,latest">{regex%sig_i686}</sig>
<tarball flags="regex latest">{regex%tarball_i686}</tarball>
<checksum hash_algo="sha512" explicit="true">cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e</checksum>
<sig keys="248BF41F9BDD61D41D060AE774EDA3C6B06D0506" keyserver="hkp://pool.sks-keyservers.net" flags="regex latest">{regex%sig_i686}</sig>
</source>
</sources>
<build its_full_of_stars="yes">
<build its_full_of_stars="true">
<paths>
<base>{variable%bdisk_root}/base</base>
<cache>{variable%bdisk_root}/cache</cache>
@ -85,11 +86,11 @@
</paths>
<basedistro>archlinux</basedistro>
</build>
<iso sign="yes" multi_arch="yes"/>
<ipxe sign="yes" iso="yes">
<iso sign="true" multi_arch="true"/>
<ipxe sign="true" iso="true">
<uri>{xpath%//meta/dev/website/text()}/ipxe</uri>
</ipxe>
<pki overwrite="no">
<pki overwrite="false">
<!-- http://ipxe.org/crypto -->
<ca>
<cert hash_algo="sha512">{xpath%../../../build/paths/pki/text()}/ca.crt</cert>
@ -125,7 +126,7 @@
<csr/>
<key cipher="none" passphrase="none" keysize="4096">{xpath%//build/paths/pki/text()}/{xpath%../../../meta/names/uxname/text()}.key</key>
<subject>
<commonName>some client name</commonName>
<commonName>website.tld</commonName>
<countryName>XX</countryName>
<localityName>Some City</localityName>
<stateOrProvinceName>Some State</stateOrProvinceName>
@ -135,22 +136,25 @@
</subject>
</client>
</pki>
<!-- If prompt_passphrase is "no" and passphrase attribute is not given for a gpg element, we will try to use a
<!-- If prompt_passphrase is false and passphrase attribute is not given for a gpg element, we will try to use a
blank passphrase for all operations. -->
<gpg keyid="none" gnupghome="none" publish="no" prompt_passphrase="no">
<gpg keyid="none" gnupghome="none" publish="false" prompt_passphrase="false">
<!-- The below is only used if we are generating a key (i.e. keyid="none"). -->
<key type="rsa" keysize="4096" expire="0">
<name>{xpath%../../../../meta/dev/author/text()}</name>
<email>{xpath%../../../../meta/dev/email/text()}</email>
<comment>for {xpath%../../../../meta/names/pname/text()} [autogenerated] | {xpath%../../../../meta/uri/text()} | {xpath%../../../../meta/desc/text()}</comment>
<key algo="rsa" keysize="4096" expire="0">
<name>{xpath%../../../meta/dev/author/text()}</name>
<email>{xpath%../../../meta/dev/email/text()}</email>
<!-- If present, the subkey element will create a secondary key used *only* for signing. This is good security practice. Obviously, this is only used if we are creating a new (master) key. -->
<subkey algo="ed" keysize="25519" expire="0"/>
<comment>for {xpath%../../../meta/names/pname/text()} [autogenerated] | {xpath%../../../meta/uri/text()} | {xpath%../../../meta/desc/text()}</comment>
</key>
</gpg>
<sync>
<ipxe enabled="yes">/srv/http/{xpath%../../meta/names/uxname/text()}</ipxe>
<tftp enabled="yes">/tftproot/{xpath%../../meta/names/uxname/text()}</tftp>
<iso enabled="yes">/srv/http/isos/{xpath%../../meta/names/uxname/text()}</iso>
<gpg enabled="yes" format="asc">/srv/http/{xpath%../../meta/names/uxname/text()}/pubkey.asc</gpg>
<rsync enabled="yes">
<!-- ipxe includes the http directory. or should, anyways. -->
<ipxe enabled="true">/srv/http/{xpath%../../meta/names/uxname/text()}</ipxe>
<tftp enabled="true">/tftproot/{xpath%../../meta/names/uxname/text()}</tftp>
<iso enabled="true">/srv/http/isos/{xpath%../../meta/names/uxname/text()}</iso>
<gpg enabled="true" format="asc">/srv/http/{xpath%../../meta/names/uxname/text()}/pubkey.asc</gpg>
<rsync enabled="true">
<user>root</user>
<host>mirror.domain.tld</host>
<port>22</port>
@ -158,17 +162,16 @@
</rsync>
</sync>
</profile>
<profile name="alternate" id="2" uuid="2ed07c19-2071-4d66-8569-da40475ba716">
<profile name="alternate" id="2" uuid="2ed07c19-2071-4d66-8569-da40475ba716">
<meta>
<names>
<name>AnotherCD</name>
<name>ALTCD</name>
<uxname>bdisk_alt</uxname>
<pname>{xpath%../name/text()}</pname>
</names>
<desc>Another rescue/restore live environment.</desc>
<dev>
<author>Another Dev Eloper</author>
<!-- You can reference other profiles within the same configuration. -->
<author>Another Dev Eloper</author><!-- You can reference other profiles within the same configuration. -->
<email>{xpath%//profile[@name="default"]/meta/dev/email/text()}</email>
<website>{xpath%//profile[@name="default"]/meta/dev/website/text()}</website>
</dev>
@ -186,30 +189,30 @@
</variables>
</meta>
<accounts>
<rootpass hashed="no">atotallyinsecurepassword</rootpass>
<user sudo="no">
<rootpass hashed="false">atotallyinsecurepassword</rootpass>
<user sudo="false">
<username>testuser</username>
<comment>Test User</comment>
<password hashed="no" hash_algo="sha512" salt="auto">atestpassword</password>
<password hashed="false" hash_algo="sha512" salt="auto">atestpassword</password>
</user>
</accounts>
<sources>
<source arch="x86_64">
<mirror>http://archlinux.mirror.domain.tld</mirror>
<rootpath>/iso/latest</rootpath>
<tarball flags="regex,latest">{regex%tarball_x86_64}</tarball>
<checksum hash_algo="sha1" flags="none">sha1sums.txt</checksum>
<sig keys="7F2D434B9741E8AC" keyserver="hkp://pool.sks-keyservers.net" flags="regex,latest">{regex%sig_x86_64}</sig>
<tarball flags="regex latest">{regex%tarball_x86_64}</tarball>
<checksum hash_algo="sha1" explicit="false" flags="latest">sha1sums.txt</checksum>
<sig keys="7F2D434B9741E8AC" keyserver="hkp://pool.sks-keyservers.net" flags="regex latest">{regex%sig_x86_64}</sig>
</source>
<source arch="i686">
<mirror>http://archlinux32.mirror.domain.tld</mirror>
<rootpath>/iso/latest</rootpath>
<tarball flags="regex,latest">{regex%tarball_i686}</tarball>
<checksum hash_algo="sha512" explicit="yes">cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e</checksum>
<sig keys="248BF41F9BDD61D41D060AE774EDA3C6B06D0506" keyserver="hkp://pool.sks-keyservers.net" flags="regex,latest">{regex%sig_i686}</sig>
<tarball flags="regex latest">{regex%tarball_i686}</tarball>
<checksum hash_algo="sha512" explicit="true">cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e</checksum>
<sig keys="248BF41F9BDD61D41D060AE774EDA3C6B06D0506" keyserver="hkp://pool.sks-keyservers.net" flags="regex latest">{regex%sig_i686}</sig>
</source>
</sources>
<build its_full_of_stars="yes">
<build its_full_of_stars="true">
<paths>
<base>{variable%bdisk_root}/base</base>
<cache>{variable%bdisk_root}/cache</cache>
@ -226,11 +229,11 @@
</paths>
<basedistro>archlinux</basedistro>
</build>
<iso sign="yes" multi_arch="yes"/>
<ipxe sign="yes" iso="yes">
<iso sign="true" multi_arch="true"/>
<ipxe sign="true" iso="true">
<uri>{xpath%//meta/dev/website/text()}/ipxe</uri>
</ipxe>
<pki overwrite="no">
<pki overwrite="false">
<ca>
<cert hash_algo="sha512">{xpath%../../../build/paths/pki/text()}/ca.crt</cert>
<csr/>
@ -252,7 +255,7 @@
<csr/>
<key cipher="none" passphrase="none" keysize="4096">{xpath%//build/paths/pki/text()}/{xpath%../../../meta/names/uxname/text()}.key</key>
<subject>
<commonName>some client name</commonName>
<commonName>website.tld</commonName>
<countryName>XX</countryName>
<localityName>Some City</localityName>
<stateOrProvinceName>Some State</stateOrProvinceName>
@ -262,19 +265,19 @@
</subject>
</client>
</pki>
<gpg keyid="none" gnupghome="none" publish="no" prompt_passphrase="no">
<key type="rsa" keysize="4096" expire="0">
<name>{xpath%../../../../meta/dev/author/text()}</name>
<email>{xpath%../../../../meta/dev/email/text()}</email>
<comment>for {xpath%../../../../meta/names/pname/text()} [autogenerated] | {xpath%../../../../meta/uri/text()} | {xpath%../../../../meta/desc/text()}</comment>
<gpg keyid="none" gnupghome="none" publish="false" prompt_passphrase="false">
<key algo="rsa" keysize="4096" expire="0">
<name>{xpath%../../../meta/dev/author/text()}</name>
<email>{xpath%../../../meta/dev/email/text()}</email>
<comment>for {xpath%../../../meta/names/pname/text()} [autogenerated] | {xpath%../../../meta/uri/text()} | {xpath%../../../meta/desc/text()}</comment>
</key>
</gpg>
<sync>
<ipxe enabled="yes">/srv/http/{xpath%../../meta/names/uxname/text()}</ipxe>
<tftp enabled="yes">/tftproot/{xpath%../../meta/names/uxname/text()}</tftp>
<iso enabled="yes">/srv/http/isos/{xpath%../../meta/names/uxname/text()}</iso>
<gpg enabled="yes" format="asc">/srv/http/{xpath%../../meta/names/uxname/text()}/pubkey.asc</gpg>
<rsync enabled="yes">
<ipxe enabled="true">/srv/http/{xpath%../../meta/names/uxname/text()}</ipxe>
<tftp enabled="true">/tftproot/{xpath%../../meta/names/uxname/text()}</tftp>
<iso enabled="true">/srv/http/isos/{xpath%../../meta/names/uxname/text()}</iso>
<gpg enabled="true" format="asc">/srv/http/{xpath%../../meta/names/uxname/text()}/pubkey.asc</gpg>
<rsync enabled="true">
<user>root</user>
<host>mirror.domain.tld</host>
<port>22</port>

39
docs/examples/regen_multi.py
View File

@ -1,13 +1,34 @@
#!/usr/bin/env python3.6

import copy
from lxml import etree
from lxml import etree, objectify

parser = etree.XMLParser(remove_blank_text = True)
#parser = etree.XMLParser(remove_blank_text = True)
parser = etree.XMLParser(remove_blank_text = False)

# We need to append to a new root because you can't edit nsmap, and you can't
# xpath on an element with a naked namespace (e.g. 'xlmns="..."').
ns = {None: 'http://bdisk.square-r00t.net/',
'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}
xsi = {'{http://www.w3.org/2001/XMLSchema-instance}schemaLocation':
'http://bdisk.square-r00t.net bdisk.xsd'}
new_cfg = etree.Element('bdisk', nsmap = ns, attrib = xsi)
new_cfg.text = '\n '

with open('single_profile.xml', 'rb') as f:
xml = etree.fromstring(f.read(), parser)


roottree = xml.getroottree()
for elem in roottree.getiterator():
if not hasattr(elem.tag, 'find'):
continue
i = elem.tag.find('}')
if i >= 0:
elem.tag = elem.tag[i + 1:]
objectify.deannotate(roottree, cleanup_namespaces = True)


single_profile = xml.xpath('/bdisk/profile[1]')[0]
alt_profile = copy.deepcopy(single_profile)
for c in alt_profile.xpath('//comment()'):
@ -19,7 +40,7 @@ alt_profile.attrib['name'] = 'alternate'
alt_profile.attrib['id'] = '2'
alt_profile.attrib['uuid'] = '2ed07c19-2071-4d66-8569-da40475ba716'

meta_tags = {'name': 'AnotherCD',
meta_tags = {'name': 'ALTCD',
'uxname': 'bdisk_alt',
'pname': '{xpath%../name/text()}',
'desc': 'Another rescue/restore live environment.',
@ -42,18 +63,22 @@ for e in accounts.iter():
if e.tag in accounts_tags:
e.text = accounts_tags[e.tag]
if e.tag == 'rootpass':
e.attrib['hashed'] = 'no'
e.attrib['hashed'] = 'false'
elif e.tag == 'user':
e.attrib['sudo'] = 'no'
e.attrib['sudo'] = 'false'
# Delete the second user
accounts.remove(accounts[2])
author = alt_profile.xpath('/profile/meta/dev/author')[0]
author.addnext(etree.Comment(
' You can reference other profiles within the same configuration. '))
xml.append(alt_profile)
#xml.append(alt_profile)

for child in xml.xpath('/bdisk/profile'):
new_cfg.append(copy.deepcopy(child))
new_cfg.append(alt_profile)

with open('multi_profile.xml', 'wb') as f:
f.write(etree.tostring(xml,
f.write(etree.tostring(new_cfg,
pretty_print = True,
encoding = 'UTF-8',
xml_declaration = True))

83
docs/examples/single_profile.xml
View File

@ -1,15 +1,18 @@
<?xml version="1.0" encoding="UTF-8" ?>
<bdisk>
<bdisk xmlns="http://bdisk.square-r00t.net/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://bdisk.square-r00t.net bdisk.xsd">
<profile name="default" id="1" uuid="8cdd6bcb-c147-4a63-9779-b5433c510dbc">
<meta>
<names>
<name>BDisk</name>
<name>BDISK</name>
<!--<name>{xpath%../uxname/text()}</name>-->
<uxname>bdisk</uxname>
<!-- Just like with previous versions of BDisk, you can reference other values...
but now with the neat benefits of XPath! Everything you could do in build.ini's and more.
See https://www.w3schools.com/xml/xpath_syntax.asp
If you need a literal curly brace, double them (e.g. for "{foo}", use "{{foo}}"),
UNLESS it's in a {regex%...} placeholder/filter (as part of the expression). -->
UNLESS it's in a <regexes><pattern> as part of the expression. Those are taken as literal strings. -->
<pname>{xpath%../name/text()}</pname>
</names>
<desc>A rescue/restore live environment.</desc>
@ -24,10 +27,11 @@
<!-- If the maximum level is reached, the substitution will evaluate as blank. -->
<max_recurse>5</max_recurse>
<!-- You need to store regex patterns here and reference them in a special way later, and it's only valid for certain
items. See the manual for more information. -->
items. See the manual for more information. NO btags within the patterns is allowed. -->
<regexes>
<pattern id="tarball_x86_64">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-x86_64\.tar\.gz$</pattern>
<pattern id="sig_x86_64">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-x86_64\.tar\.gz\.sig$</pattern>
<pattern id="sig_x86_64">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-x86_64\.tar\.gz\.sig$
</pattern>
<pattern id="tarball_i686">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-i686\.tar\.gz$</pattern>
<pattern id="sig_i686">archlinux-bootstrap-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-i686\.tar\.gz\.sig$</pattern>
</regexes>
@ -38,20 +42,20 @@
</meta>
<accounts>
<!-- Salted/hashed password is "test" -->
<rootpass hashed="yes">$6$7KfIdtHTcXwVrZAC$LZGNeMNz7v5o/cYuA48FAxtZynpIwO5B1CPGXnOW5kCTVpXVt4SypRqfM.AoKkFt/O7MZZ8ySXJmxpELKmdlF1</rootpass>
<user sudo="yes">
<username>{xpath%//meta/names/uxname/text()}</username>
<rootpass hashed="true">$6$7KfIdtHTcXwVrZAC$LZGNeMNz7v5o/cYuA48FAxtZynpIwO5B1CPGXnOW5kCTVpXVt4SypRqfM.AoKkFt/O7MZZ8ySXJmxpELKmdlF1</rootpass>
<user sudo="true">
<username>{xpath%../../../meta/names/uxname/text()}</username>
<!-- You can also use substitution from different profiles in this same configuration: -->
<!-- <username>{xpath%//profile[@name='another_profile']/meta/names/uxname"}</username> -->
<comment>{xpath%//meta/dev/author/text()}</comment>
<password hashed="no"
<comment>{xpath%../../../meta/dev/author/text()}</comment>
<password hashed="false"
hash_algo="sha512"
salt="auto">testpassword</password>
</user>
<user sudo="no">
<user sudo="false">
<username>testuser</username>
<name>Test User</name>
<password hashed="no"
<comment>Test User</comment>
<password hashed="false"
hash_algo="sha512"
salt="auto">anothertestpassword</password>
</user>
@ -60,25 +64,29 @@
<source arch="x86_64">
<mirror>http://archlinux.mirror.domain.tld</mirror>
<rootpath>/iso/latest</rootpath>
<tarball flags="regex,latest">{regex%tarball_x86_64}</tarball>
<tarball flags="regex latest">{regex%tarball_x86_64}</tarball>
<checksum hash_algo="sha1"
flags="none">sha1sums.txt</checksum>
explicit="false"
flags="latest">sha1sums.txt</checksum>
<sig keys="7F2D434B9741E8AC"
keyserver="hkp://pool.sks-keyservers.net"
flags="regex,latest">{regex%sig_x86_64}</sig>
flags="regex latest">{regex%sig_x86_64}</sig>
</source>
<source arch="i686">
<mirror>http://archlinux32.mirror.domain.tld</mirror>
<rootpath>/iso/latest</rootpath>
<tarball flags="regex,latest">{regex%tarball_i686}</tarball>
<tarball flags="regex latest">{regex%tarball_i686}</tarball>
<checksum hash_algo="sha512"
explicit="yes">cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e</checksum>
explicit="true">cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e </checksum>
<sig keys="248BF41F9BDD61D41D060AE774EDA3C6B06D0506"
keyserver="hkp://pool.sks-keyservers.net"
flags="regex,latest">{regex%sig_i686}</sig>
flags="regex latest">{regex%sig_i686}</sig>
</source>
</sources>
<build its_full_of_stars="yes">
<packages>
<package repo="core">openssh</package>
</packages>
<build its_full_of_stars="true">
<paths>
<base>{variable%bdisk_root}/base</base>
<cache>{variable%bdisk_root}/cache</cache>
@ -95,11 +103,11 @@
</paths>
<basedistro>archlinux</basedistro>
</build>
<iso sign="yes" multi_arch="yes" />
<ipxe sign="yes" iso="yes">
<iso sign="true" multi_arch="true"/>
<ipxe sign="true" iso="true">
<uri>{xpath%//meta/dev/website/text()}/ipxe</uri>
</ipxe>
<pki overwrite="no">
<pki overwrite="false">
<!-- http://ipxe.org/crypto -->
<ca>
<cert hash_algo="sha512">{xpath%../../../build/paths/pki/text()}/ca.crt</cert>
@ -109,7 +117,7 @@
then provide a path.
e.g.:
<csr>{xpath%build/paths/ssl/text()}/ca.csr</csr> -->
<csr />
<csr/>
<!-- If you use an index file (or want to) to serialize client certificates, specify it here. -->
<!-- It must conform to CADB spec (https://pki-tutorial.readthedocs.io/en/latest/cadb.html). -->
<!-- You should probably also specify a serial file if so. -->
@ -134,12 +142,12 @@
</ca>
<client>
<cert hash_algo="sha512">{xpath%../../../build/paths/pki/text()}/{xpath%../../../meta/names/uxname/text()}.crt</cert>
<csr />
<csr/>
<key cipher="none"
passphrase="none"
keysize="4096">{xpath%//build/paths/pki/text()}/{xpath%../../../meta/names/uxname/text()}.key</key>
<subject>
<commonName>some client name</commonName>
<commonName>website.tld</commonName>
<countryName>XX</countryName>
<localityName>Some City</localityName>
<stateOrProvinceName>Some State</stateOrProvinceName>
@ -149,26 +157,27 @@
</subject>
</client>
</pki>
<!-- If prompt_passphrase is "no" and passphrase attribute is not given for a gpg element, we will try to use a
<!-- If prompt_passphrase is "false" and passphrase attribute is not given for a gpg element, we will try to use a
blank passphrase for all operations. -->
<gpg keyid="none"
gnupghome="none"
publish="no"
prompt_passphrase="no">
publish="false"
prompt_passphrase="false">
<!-- The below is only used if we are generating a key (i.e. keyid="none"). -->
<key algo="rsa" keysize="4096" expire="0">
<name>{xpath%../../../../meta/dev/author/text()}</name>
<email>{xpath%../../../../meta/dev/email/text()}</email>
<comment>for {xpath%../../../../meta/names/pname/text()} [autogenerated] | {xpath%../../../../meta/uri/text()} | {xpath%../../../../meta/desc/text()}</comment>
<name>{xpath%../../../meta/dev/author/text()}</name>
<email>{xpath%../../../meta/dev/email/text()}</email>
<comment>for {xpath%../../../meta/names/pname/text()} [autogenerated] | {xpath%../../../meta/uri/text()} | {xpath%../../../meta/desc/text()}</comment>
</key>
</gpg>
<sync>
<ipxe enabled="yes">/srv/http/{xpath%../../meta/names/uxname/text()}</ipxe>
<tftp enabled="yes">/tftproot/{xpath%../../meta/names/uxname/text()}</tftp>
<iso enabled="yes">/srv/http/isos/{xpath%../../meta/names/uxname/text()}</iso>
<gpg enabled="yes"
<!-- ipxe includes the http directory. or should, anyways. -->
<ipxe enabled="true">/srv/http/{xpath%../../meta/names/uxname/text()}</ipxe>
<tftp enabled="true">/tftproot/{xpath%../../meta/names/uxname/text()}</tftp>
<iso enabled="true">/srv/http/isos/{xpath%../../meta/names/uxname/text()}</iso>
<gpg enabled="true"
format="asc">/srv/http/{xpath%../../meta/names/uxname/text()}/pubkey.asc</gpg>
<rsync enabled="yes">
<rsync enabled="true">
<user>root</user>
<host>mirror.domain.tld</host>
<port>22</port>

3
docs/manual/TODO
View File

@ -8,3 +8,6 @@
- in faq/ISOBIG.adoc and the doc section it references, make sure we reference that the package lists are now in the environment plugin!

- change all references to build.ini to something like "BDisk configuration file"

- reminder: users can specify a local file source for <sources><source> items by using "file:///absolute/path/to/file"
-- todo: add http auth, ftp, ftps

8
examples/README Normal file
View File

@ -0,0 +1,8 @@
This directory contains example files/data that you may see referenced in documentation/code.

- mtree.spec
This file is an example mtree spec sheet that one may use for an overlay. It was generated by the command "mtree -c -K all -p /home/bts".
If you're on Arch, a port of mtree can be found in the AUR under the package name "nmtree" (it's maintained by the same author as BDisk!).
If you're on Debian or Ubuntu (or forks thereof), you can find it in the "freebsd-buildutils" package. (The executable is called "fmtree").
If you're on Gentoo, it's in sys-apps/mtree.
If you're on RHEL/CentOS, the "extras" repository has gomtree, which (although written in Go) should be able to produce mtree spec files (but this is unknown for certain).

1191
examples/mtree.spec Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
external/apacman-current.pkg.tar.xz vendored Normal file
View File

Binary file not shown.

BIN
external/aurman-current.pkg.tar.xz vendored Normal file
View File

Binary file not shown.