add collapsing toc to ascii.html

dns.resolver.resolve > dns.resolve.query

.gitignore

@@ -22,4 +22,6 @@ __pycache__/
 *.run
 *.7z
 *.rar
+*.sqlite3
 *.deb
+.idea/

LICENSE

@@ -0,0 +1,674 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<http://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<http://www.gnu.org/philosophy/why-not-lgpl.html>.

TODO

@@ -0,0 +1,39 @@
+- sshsecure is being re-written in golang
+
+-vault, schema dumper (dump mounts, paths (otional w/switch or toggle), and meta information)
+--ability to recreate from xml dump
+
+-git
+
+-net/addr needs DNS/PTR/allocation stuff etc.
+
+-net/mirroring
+
+-storage, see if we can access lvm and cryptsetup functions via https://github.com/storaged-project/libblockdev/issues/41
+--http://storaged.org/doc/udisks2-api/latest/gdbus-org.freedesktop.UDisks2.MDRaid.html
+--http://storaged.org/doc/udisks2-api/latest/gdbus-org.freedesktop.UDisks2.Encrypted.html
+--http://mindbending.org/en/python-and-udisks-part-2
+--http://storaged.org/doc/udisks2-api/2.6.5/gdbus-org.freedesktop.UDisks2.Block.html
+--https://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html
+
+
+sshkeys:
+-need to verify keys via GPG signature. we also need to have a more robust way of updating pubkeys - categorization, role
+-write API to get pubkeys, hostkeys? really wish DBs supported nesting
+-separate by algo, but this is easy to do (split on space, [0])
+
+snippet: create mtree with libarchive, bsdtar -cf /tmp/win.mtree --one-file-system --format=mtree --options='mtree:sha512,mtree:indent'  /path/*
+probably need to package https://packages.debian.org/source/stretch/freebsd-buildutils to get fmtree for reading
+
+-net, add ipxe - write flask app that determines path based on MAC addr
+
+-net, add shorewall templater
+
+-port in sslchk
+
+-script that uses uconv(?) and pymysql to export database to .ods
+
+-IRC
+-- i should use the python IRC module on pypi to join an irc network (freenode, probably, for my personal interests) and
+   run an iteration over all nicks in a channel with /ctcp <nick> version. handy when i'm trying to find someone running
+   a certain platform/client i have some questions about.

aif/cfgs/base.xml

@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<aif xmlns:aif="https://aif.square-r00t.net"
+     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+     xsi:schemaLocation="https://aif.square-r00t.net aif.xsd">
+    <storage>
+        <disk device="/dev/sda" diskfmt="gpt">
+            <part num="1" start="0%" size="10%" fstype="ef00" />
+            <part num="2" start="10%" size="100%" fstype="8300" />
+        </disk>
+        <mount source="/dev/sda2" target="/mnt/aif" order="1" />
+        <mount source="/dev/sda1" target="/mnt/aif/boot" order="2" />
+    </storage>
+    <network hostname="aiftest.square-r00t.net">
+        <iface device="auto" address="auto" netproto="ipv4" />
+    </network>
+    <system timezone="EST5EDT" locale="en_US.UTF-8" chrootpath="/mnt/aif" reboot="1">
+        <users rootpass="!" />
+        <service name="sshd" status="1" />
+        <service name="cronie" status="1" />
+        <service name="haveged" status="1" />
+    </system>
+    <pacman command="apacman -S">
+        <repos>
+            <repo name="core" enabled="true" siglevel="default" mirror="file:///etc/pacman.d/mirrorlist" />
+            <repo name="extra" enabled="true" siglevel="default" mirror="file:///etc/pacman.d/mirrorlist" />
+            <repo name="community" enabled="true" siglevel="default" mirror="file:///etc/pacman.d/mirrorlist" />
+            <repo name="multilib" enabled="true" siglevel="default" mirror="file:///etc/pacman.d/mirrorlist" />
+            <repo name="testing" enabled="false" siglevel="default" mirror="file:///etc/pacman.d/mirrorlist" />
+            <repo name="multilib-testing" enabled="false" siglevel="default" mirror="file:///etc/pacman.d/mirrorlist" />
+            <repo name="archlinuxfr" enabled="false" siglevel="Optional TrustedOnly" mirror="http://repo.archlinux.fr/$arch" />
+        </repos>
+        <mirrorlist>
+            <mirror>http://mirror.us.leaseweb.net/archlinux/$repo/os/$arch</mirror>
+            <mirror>http://mirrors.advancedhosters.com/archlinux/$repo/os/$arch</mirror>
+            <mirror>http://ftp.osuosl.org/pub/archlinux/$repo/os/$arch</mirror>
+            <mirror>http://arch.mirrors.ionfish.org/$repo/os/$arch</mirror>
+            <mirror>http://mirrors.gigenet.com/archlinux/$repo/os/$arch</mirror>
+            <mirror>http://mirror.jmu.edu/pub/archlinux/$repo/os/$arch</mirror>
+        </mirrorlist>
+        <software>
+            <package name="sed" repo="core" />
+            <package name="python" />
+            <package name="openssh" />
+            <package name="vim" />
+            <package name="vim-plugins" />
+            <package name="haveged" />
+            <package name="byobu" />
+            <package name="etc-update" />
+            <package name="cronie" />
+            <package name="mlocate" />
+            <package name="mtree-git" />
+        </software>
+    </pacman>
+    <bootloader type="grub" target="/boot" efi="true" />
+    <scripts>
+        <script uri="https://aif.square-r00t.net/cfgs/scripts/pkg/python.sh" order="1" execution="pkg" />
+        <script uri="https://aif.square-r00t.net/cfgs/scripts/pkg/apacman.py" order="2" execution="pkg" />
+        <script uri="https://aif.square-r00t.net/cfgs/scripts/post/sshsecure.py" order="1" execution="post" />
+        <script uri="https://aif.square-r00t.net/cfgs/scripts/post/sshkeys.py" order="2" execution="post" />
+        <script uri="https://aif.square-r00t.net/cfgs/scripts/post/configs.py" order="3" execution="post" />
+    </scripts>
+</aif>

aif/scripts/pkg/apacman.py

@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+
+import datetime
+import os
+import re
+import shutil
+import subprocess
+from urllib.request import urlopen
+
+pkg_base = 'apacman'
+pkgs = ('', '-deps', '-utils')
+url_base = 'https://aif.square-r00t.net/cfgs/files'
+local_dir = '/tmp'
+
+conf_options = {}
+conf_options['apacman'] = {'enabled': ['needed', 'noconfirm', 'noedit', 'progress', 'purgebuild', 'skipcache', 'keepkeys'],
+                           'disabled': [],
+                           'values': {'tmpdir': '"/var/tmp/apacmantmp-$UID"'}}
+conf_options['pacman'] = {'enabled': [],
+                          'disabled': [],
+                          'values': {'UseSyslog': None, 'Color': None, 'TotalDownload': None, 'CheckSpace': None, 'VerbosePkgLists': None}}
+
+def downloadPkg(pkgfile, dlfile):
+    url = os.path.join(url_base, pkgfile)
+    # Prep the destination
+    os.makedirs(os.path.dirname(dlfile), exist_ok = True)
+    # Download the pacman package
+    with urlopen(url) as u:
+        with open(dlfile, 'wb') as f:
+            f.write(u.read())
+    return()
+
+def installPkg(pkgfile):
+    # Install it
+    subprocess.run(['pacman', '-Syyu'])  # Installing from an inconsistent state is bad, mmkay?
+    subprocess.run(['pacman', '--noconfirm', '--needed', '-S', 'base-devel'])
+    subprocess.run(['pacman', '--noconfirm', '--needed', '-S', 'multilib-devel'])
+    subprocess.run(['pacman', '--noconfirm', '--needed', '-U', pkgfile])
+    return()
+
+def configurePkg(opts, pkgr):
+    cf = '/etc/{0}.conf'.format(pkgr)
+    # Configure it
+    shutil.copy2(cf, '{0}.bak.{1}'.format(cf, int(datetime.datetime.utcnow().timestamp())))
+    with open(cf, 'r') as f:
+        conf = f.readlines()
+    for idx, line in enumerate(conf):
+        l = line.split('=')
+        opt = l[0].strip('\n').strip()
+        if len(l) > 1:
+            val = l[1].strip('\n').strip()
+        # enabled options
+        for o in opts['enabled']:
+            if re.sub('^#?', '', opt).strip() == o:
+                if pkgr == 'apacman':
+                    conf[idx] = '{0}=1\n'.format(o)
+                elif pkgr == 'pacman':
+                    conf[idx] = '{0}\n'.format(o)
+        # disabled options
+        for o in opts['disabled']:
+            if re.sub('^#?', '', opt).strip() == o:
+                if pkgr == 'apacman':
+                    conf[idx] = '{0}=0\n'.format(o)
+                elif pkgr == 'pacman':
+                    conf[idx] = '#{0}\n'.format(o)
+        # values
+        for o in opts['values']:
+            if opts['values'][o] is not None:
+                if re.sub('^#?', '', opt).strip() == o:
+                    if pkgr == 'apacman':
+                        conf[idx] = '{0}={1}\n'.format(o, opts['values'][o])
+                    elif pkgr == 'pacman':
+                        conf[idx] = '{0} = {1}\n'.format(o, opts['values'][o])
+            else:
+                if re.sub('^#?', '', opt).strip() == o:
+                    conf[idx] = '{0}\n'.format(o)
+    with open(cf, 'w') as f:
+        f.write(''.join(conf))
+
+def finishPkg():
+    # Finish installing (optional deps)
+    for p in ('git', 'customizepkg-scripting', 'pkgfile', 'rsync'):
+        subprocess.run(['apacman', '--noconfirm', '--needed', '-S', p])
+
+def main():
+    for p in pkgs:
+        pkg = pkg_base + p
+        fname = '{0}.tar.xz'.format(pkg)
+        local_pkg = os.path.join(local_dir, fname)
+        downloadPkg(fname, local_pkg)
+        installPkg(local_pkg)
+    for tool in ('pacman', 'apacman'):
+        configurePkg(conf_options[tool], tool)
+    finishPkg()
+    return()
+
+if __name__ == '__main__':
+    main()

aif/scripts/pkg/python.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+pacman --needed --noconfirm -S python python-pip python-setuptools

aif/scripts/post/configs.py

@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+
+import os
+import pathlib
+import pwd
+import subprocess
+
+def byobu(user = 'root'):
+    homedir = os.path.expanduser('~{0}'.format(user))
+    subprocess.run(['byobu-enable'])
+    b = '{0}/.byobu'.format(homedir)
+    # The keybindings, and general enabling
+    confs = {'backend': 'BYOBU_BACKEND=tmux\n',
+             'color': 'BACKGROUND=k\nFOREGROUND=w\nMONOCHROME=0',  # NOT a typo; the original source I got this from had no end newline.
+             'color.tmux': 'BYOBU_DARK="\#333333"\nBYOBU_LIGHT="\#EEEEEE"\nBYOBU_ACCENT="\#75507B"\nBYOBU_HIGHLIGHT="\#DD4814"\n',
+             'datetime.tmux': 'BYOBU_DATE="%Y-%m-%d "\nBYOBU_TIME="%H:%M:%S"\n',
+             'keybindings': 'source $BYOBU_PREFIX/share/byobu/keybindings/common\n',
+             'keybindings.tmux': 'unbind-key -n C-a\nset -g prefix ^A\nset -g prefix2 ^A\nbind a send-prefix\n',
+             'profile': 'source $BYOBU_PREFIX/share/byobu/profiles/common\n',
+             'profile.tmux': 'source $BYOBU_PREFIX/share/byobu/profiles/tmux\n',
+             'prompt': '[ -r /usr/share/byobu/profiles/bashrc ] && . /usr/share/byobu/profiles/bashrc  #byobu-prompt#\n',
+             '.screenrc': None,
+             '.tmux.conf': None,
+             '.welcome-displayed': None,
+             'windows': None,
+             'windows.tmux': None}
+    for c in confs.keys():
+        with open('{0}/{1}'.format(b, c), 'w') as f:
+            if confs[c] is not None:
+                f.write(confs[c])
+            else:
+                f.write('')
+    # The status file- add some extras, and remove the session string which is broken apparently.
+    # Holy shit I wish there was a way of storing compressed text in plaintext besides base64.
+    statusconf = ["#    status - Byobu's default status enabled/disabled settings\n", '#\n', '#      Override these in $BYOBU_CONFIG_DIR/status\n',
+                  '#      where BYOBU_CONFIG_DIR is XDG_CONFIG_HOME if defined,\n', '#      and $HOME/.byobu otherwise.\n', '#\n',
+                  '#    Copyright (C) 2009-2011 Canonical Ltd.\n', '#\n', '#    Authors: Dustin Kirkland <kirkland@byobu.org>\n', '#\n',
+                  '#    This program is free software: you can redistribute it and/or modify\n', '#    it under the terms of the GNU ' +
+                  'General Public License as published by\n', '#    the Free Software Foundation, version 3 of the License.\n', '#\n',
+                  '#    This program is distributed in the hope that it will be useful,\n', '#    but WITHOUT ANY WARRANTY; without even the ' +
+                  'implied warranty of\n', '#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n', '#    GNU General Public License ' +
+                  'for more details.\n', '#\n', '#    You should have received a copy of the GNU General Public License\n', '#    along with this ' +
+                  'program.  If not, see <http://www.gnu.org/licenses/>.\n', '\n', "# Status beginning with '#' are disabled.\n", '\n', '# Screen has ' +
+                  'two status lines, with 4 quadrants for status\n', 'screen_upper_left="color"\n', 'screen_upper_right="color whoami hostname ' +
+                  'ip_address menu"\n', 'screen_lower_left="color logo distro release #arch session"\n', 'screen_lower_right="color network #disk_io ' +
+                  'custom #entropy raid reboot_required updates_available #apport #services #mail users uptime #ec2_cost #rcs_cost #fan_speed #cpu_temp ' +
+                  'battery wifi_quality #processes load_average cpu_count cpu_freq memory #swap disk #time_utc date time"\n', '\n', '# Tmux has one ' +
+                  'status line, with 2 halves for status\n', 'tmux_left=" logo #distro release arch #session"\n', '# You can have as many tmux right ' +
+                  'lines below here, and cycle through them using Shift-F5\n', 'tmux_right=" network disk_io #custom #entropy raid reboot_required ' +
+                  '#updates_available #apport services #mail #users uptime #ec2_cost #rcs_cost #fan_speed #cpu_temp #battery #wifi_quality processes ' +
+                  'load_average cpu_count cpu_freq memory #swap disk whoami hostname ip_address time_utc date time"\n', '#tmux_right="network ' +
+                  '#disk_io #custom entropy raid reboot_required updates_available #apport #services #mail users uptime #ec2_cost #rcs_cost fan_speed ' +
+                  'cpu_temp battery wifi_quality #processes load_average cpu_count cpu_freq memory #swap #disk whoami hostname ip_address #time_utc ' +
+                  'date time"\n', '#tmux_right="network #disk_io custom #entropy raid reboot_required updates_available #apport #services #mail users ' +
+                  'uptime #ec2_cost #rcs_cost #fan_speed #cpu_temp battery wifi_quality #processes load_average cpu_count cpu_freq memory #swap #disk ' +
+                  '#whoami #hostname ip_address #time_utc date time"\n', '#tmux_right="#network disk_io #custom entropy #raid #reboot_required ' +
+                  '#updates_available #apport #services #mail #users #uptime #ec2_cost #rcs_cost fan_speed cpu_temp #battery #wifi_quality #processes ' +
+                  '#load_average #cpu_count #cpu_freq #memory #swap whoami hostname ip_address #time_utc disk date time"\n']
+    with open('{0}/status'.format(b), 'w') as f:
+        f.write(''.join(statusconf))
+    # The statusrc file is another lengthy one.
+    statusrc = ["#    statusrc - Byobu's default status configurations\n", '#\n', '#      Override these in $BYOBU_CONFIG_DIR/statusrc\n',
+                '#      where BYOBU_CONFIG_DIR is XDG_CONFIG_HOME if defined,\n', '#      and $HOME/.byobu otherwise.\n', '#\n', '#    Copyright (C) ' +
+                '2009-2011 Canonical Ltd.\n', '#\n', '#    Authors: Dustin Kirkland <kirkland@byobu.org>\n', '#\n', '#    This program is free software: ' +
+                'you can redistribute it and/or modify\n', '#    it under the terms of the GNU General Public License as published by\n',
+                '#    the Free Software Foundation, version 3 of the License.\n', '#\n', '#    This program is distributed in the hope that it will be ' +
+                'useful,\n', '#    but WITHOUT ANY WARRANTY; without even the implied warranty of\n', '#    MERCHANTABILITY or FITNESS FOR A PARTICULAR ' +
+                'PURPOSE.  See the\n', '#    GNU General Public License for more details.\n', '#\n', '#    You should have received a copy of the GNU ' +
+                'General Public License\n', '#    along with this program.  If not, see <http://www.gnu.org/licenses/>.\n', '\n', '# Configurations that ' +
+                'you can override; if you leave these commented out,\n', '# Byobu will try to auto-detect them.\n', '\n', '# This should be auto-detected ' +
+                'for most distro, but setting it here will save\n', '# some call to lsb_release and the like.\n', '#BYOBU_DISTRO=Ubuntu\n', '\n',
+                '# Default: depends on the distro (which is either auto-detected, either set\n', '# via $DISTRO)\n', '#LOGO="\\o/"\n', '\n', '# Abbreviate ' +
+                'the release to N characters\n', '# By default, this is disabled.  But if you set RELEASE_ABBREVIATED=1\n', '# and your lsb_release is ' +
+                '"precise", only "p" will be displayed\n', '#RELEASE_ABBREVIATED=1\n', '\n', '# Default: /\n', '#MONITORED_DISK=/\n', '\n', '# Minimum ' +
+                'disk throughput that triggers the notification (in kB/s)\n', '# Default: 50\n', '#DISK_IO_THRESHOLD=50\n', '\n', '# Default: eth0\n',
+                '#MONITORED_NETWORK=eth0\n', '\n', '# Unit used for network throughput (either bits per second or bytes per second)\n', '# Default: ' +
+                'bits\n', '#NETWORK_UNITS=bytes\n', '\n', '# Minimum network throughput that triggers the notification (in kbit/s)\n', '# Default: 20\n',
+                '#NETWORK_THRESHOLD=20\n', '\n', '# You can add an additional source of temperature here\n', '#MONITORED_TEMP=/proc/acpi/thermal_zone/' +
+                'THM0/temperature\n', '\n', '# Default: C\n', '#TEMP=F\n', '\n', '#SERVICES="eucalyptus-nc|NC eucalyptus-cloud|CLC eucalyptus-walrus ' +
+                'eucalyptus-cc|CC eucalyptus-sc|SC"\n', '\n', '#FAN=$(find /sys -type f -name fan1_input | head -n1)\n', '\n', '# You can set this to 1 ' +
+                'to report your external/public ip address\n', '# Default: 0\n', '#IP_EXTERNAL=0\n', '\n', '# The users notification normally counts ssh ' +
+                "sessions; set this configuration to '1'\n", '# to instead count number of distinct users logged onto the system\n', '# Default: 0\n',
+                '#USERS_DISTINCT=0\n', '\n', '# Set this to zero to hide seconds int the time display\n', '# Default 1\n', '#TIME_SECONDS=0\n']
+    with open('{0}/statusrc'.format(b), 'w') as f:
+        f.write(''.join(statusrc))
+    setPerms(user, b)
+    return()
+
+def vim():
+    vimc = ['\n', 'set nocompatible\n', 'set number\n', 'syntax on\n', 'set paste\n', 'set ruler\n', 'if has("autocmd")\n','  au BufReadPost * if ' +
+            'line("\'\\"") > 1 && line("\'\\"") <= line("$") | exe "normal! g\'\\"" | endif\n', 'endif\n', '\n', '" bind F3 to insert a timestamp.\n', '" In ' +
+            'normal mode, insert.\n', 'nmap <F3> i<C-R>=strftime("%c")<CR><Esc>\n', '\n', 'set pastetoggle=<F2>\n', '\n', '" https://stackoverflow.com/' +
+            'questions/27771616/turn-off-all-automatic-code-complete-in-jedi-vim\n', 'let g:jedi#completions_enabled = 0\n', 'let g:jedi#show_call_' +
+            'signatures = "0"\n']
+    with open('/etc/vimrc', 'a') as f:
+        f.write(''.join(vimc))
+    setPerms('root', '/etc/vimrc')
+    return()
+
+def bash():
+    bashc = ['\n', 'alias vi=/usr/bin/vim\n', 'export EDITOR=vim\n', '\n', 'if [ -f ~/.bashrc ];\n', 'then\n', ' source ~/.bashrc\n', 'fi \n',
+             'if [ -d ~/bin ];\n', 'then\n', ' export PATH="$PATH:~/bin"\n', 'fi\n', '\n', 'alias grep="grep --color"\n',
+             'alias egrep="egrep --color"\n', '\n', 'alias ls="ls --color=auto"\n', 'alias vi="/usr/bin/vim"\n', '\n', 'export HISTTIMEFORMAT="%F %T "\n',
+             'export PATH="${PATH}:/sbin:/bin:/usr/sbin"\n']
+    with open('/etc/bash.bashrc', 'a') as f:
+        f.write(''.join(bashc))
+    setPerms('root', '/etc/bash.bashrc')
+    return()
+
+def mlocate():
+    subprocess.run(['updatedb'])
+    return()
+
+def setPerms(user, path):
+    uid = pwd.getpwnam(user).pw_uid
+    gid = pwd.getpwnam(user).pw_gid
+    pl = pathlib.PurePath(path).parts
+    for basedir, dirs, files in os.walk(path):
+        os.chown(basedir, uid, gid)
+        if os.path.isdir(basedir):
+            os.chmod(basedir, 0o755)
+        elif os.path.isfile(basedir):
+            os.chmod(basedir, 0o644)
+        for f in files:
+            os.chown(os.path.join(basedir, f), uid, gid)
+            os.chmod(os.path.join(basedir, f), 0o644)
+    return()
+
+def main():
+    byobu()
+    vim()
+    bash()
+    mlocate()
+
+if __name__ == '__main__':
+    main()

aif/scripts/post/hostscan.py

@@ -0,0 +1,206 @@
+#!/usr/bin/env python3
+
+# Note: for hashed known-hosts, https://gist.github.com/maxtaco/5080023
+
+import argparse
+import grp
+import os
+import pwd
+import re
+import subprocess
+import sys
+
+# Defaults
+#def_supported_keys = subprocess.run(['ssh',
+#                                     '-Q',
+#                                     'key'], stdout = subprocess.PIPE).stdout.decode('utf-8').splitlines()
+def_supported_keys = ['dsa', 'ecdsa', 'ed25519', 'rsa']
+def_mode = 'append'
+def_syshostkeys = '/etc/ssh/ssh_known_hosts'
+def_user = pwd.getpwuid(os.geteuid())[0]
+def_grp = grp.getgrgid(os.getegid())[0]
+
+
+class hostscanner(object):
+    def __init__(self, args):
+        self.args = args
+        if self.args['keytypes'] == ['all']:
+            self.args['keytypes'] = def_supported_keys
+        if self.args['system']:
+            if os.geteuid() != 0:
+                exit(('You have specified system-wide modification but ' +
+                      'are not running with root privileges! Exiting.'))
+            self.args['output'] = def_syshostkeys
+        if self.args['output'] != sys.stdout:
+            _pardir = os.path.dirname(os.path.abspath(os.path.expanduser(self.args['output'])))
+            if _pardir.startswith('/home'):
+                _octmode = 0o700
+            else:
+                _octmode = 0o755
+            os.makedirs(_pardir, mode = _octmode, exist_ok = True)
+            os.chown(_pardir,
+                     pwd.getpwnam(self.args['chown_user'])[2],
+                     grp.getgrnam(self.args['chown_grp'])[2])
+
+    def getHosts(self):
+        self.keys = {}
+        _hosts = os.path.abspath(os.path.expanduser(self.args['infile']))
+        with open(_hosts, 'r') as f:
+            for l in f.readlines():
+                l = l.strip()
+                if re.search('^\s*(#.*)?$', l, re.MULTILINE):
+                    continue  # Skip commented and blank lines
+                k = re.sub('^([0-9a-z-\.]+)\s*#.*$',
+                           '\g<1>',
+                           l.strip().lower(),
+                           re.MULTILINE)
+                self.keys[k] = []
+        return()
+
+    def getKeys(self):
+        def parseType(k):
+            _newkey = re.sub('^ssh-', '', k).split('-')[0]
+            if _newkey == 'dss':
+                _newkey = 'dsa'
+            return(_newkey)
+        for h in list(self.keys.keys()):
+            _h = h.split(':')
+            if len(_h) == 1:
+                _host = _h[0]
+                _port = 22
+            elif len(_h) == 2:
+                _host = _h[0]
+                _port = int(_h[1])
+            _cmdline = ['ssh-keyscan',
+                        '-t', ','.join(self.args['keytypes']),
+                        '-p', str(_port),
+                        _host]
+            if self.args['hash']:
+                #https://security.stackexchange.com/a/56283
+                # verify via:
+                # SAMPLE ENTRY: |1|F1E1KeoE/eEWhi10WpGv4OdiO6Y=|3988QV0VE8wmZL7suNrYQLITLCg= ssh-rsa ...
+                #key=$(echo F1E1KeoE/eEWhi10WpGv4OdiO6Y= | base64 -d | xxd -p)
+                #echo -n "192.168.1.61" | openssl sha1 -mac HMAC -macopt hexkey:${key} | awk '{print $2}' | xxd -r -p | base64
+                _cmdline.insert(1, '-H')
+            _cmd = subprocess.run(_cmdline,
+                                  stdout = subprocess.PIPE,
+                                  stderr = subprocess.PIPE)
+            if not re.match('\s*#.*', _cmd.stderr.decode('utf-8')):
+                _printerr = []
+                for i in _cmd.stderr.decode('utf-8').splitlines():
+                    if i.strip() not in _printerr:
+                        _printerr.append(i.strip())
+                print('{0}: errors detected; skipping ({1})'.format(h, '\n'.join(_printerr)))
+                del(self.keys[h])
+                continue
+            for l in _cmd.stdout.decode('utf-8').splitlines():
+                _l = l.split()
+                _key = {'type': _l[1],
+                        'host': _l[0],
+                        'key': _l[2]}
+                if parseType(_key['type']) in self.args['keytypes']:
+                    self.keys[h].append(_key)
+        return()
+
+    def write(self):
+        if self.args['writemode'] == 'replace':
+            if os.path.isfile(self.args['output']) and self.args['output'] != sys.stdout:
+                os.move(self.args['output'], os.path.join(self.args['output'], '.bak'))
+        for h in self.keys.keys():
+            for i in self.keys[h]:
+                _s = '# Automatically added via hostscan.py\n{0} {1} {2}\n'.format(i['host'],
+                                                                                   i['type'],
+                                                                                   i['key'])
+                if self.args['output'] == sys.stdout:
+                    print(_s, end = '')
+                else:
+                    with open(self.args['output'], 'a') as f:
+                                f.write(_s)
+                    os.chmod(self.args['output'], 0o644)
+                    os.chown(self.args['output'],
+                             pwd.getpwnam(self.args['chown_user'])[2],
+                             grp.getgrnam(self.args['chown_grp'])[2])
+        return()
+
+def parseArgs():
+    def getTypes(t):
+        keytypes = t.split(',')
+        keytypes = [k.strip() for k in keytypes]
+        for k in keytypes:
+            if k not in ('all', *def_supported_keys):
+                raise argparse.ArgumentError('Must be one or more of the following: all, {0}'.format(', '.join(def_supported_keys)))
+        return(keytypes)
+    args = argparse.ArgumentParser(description = ('Scan a list of hosts and present their hostkeys in ' +
+                                                  'a format suitable for an SSH known_hosts file.'))
+    args.add_argument('-u',
+                      '--user',
+                      dest = 'chown_user',
+                      default = def_user,
+                      help = ('The username to chown the file to (if \033[1m{0}\033[0m is specified). ' +
+                              'Default: \033[1m{1}\033[0m').format('-o/--output', def_user))
+    args.add_argument('-g',
+                      '--group',
+                      dest = 'chown_grp',
+                      default = def_grp,
+                      help = ('The group to chown the file to (if \033[1m{0}\033[0m is specified). ' +
+                              'Default: \033[1m{1}\033[0m').format('-o/--output', def_grp))
+    args.add_argument('-H',
+                      '--hash',
+                      dest = 'hash',
+                      action = 'store_true',
+                      help = ('If specified, hash the hostkeys (see ssh-keyscan(1)\'s -H option for more info)'))
+    args.add_argument('-m',
+                      '--mode',
+                      dest = 'writemode',
+                      default = def_mode,
+                      choices = ['append', 'replace'],
+                      help = ('If \033[1m{0}\033[0m is specified, the mode to use for the ' +
+                              'destination file. The default is \033[1m{1}\033[0m').format('-o/--output', def_mode))
+    args.add_argument('-k',
+                      '--keytypes',
+                      dest = 'keytypes',
+                      type = getTypes,
+                      default = 'all',
+                      help = ('A comma-separated list of key types to add (if supported by the target host). ' +
+                              'The default is to add all keys found. Must be one (or more) of: \033[1m{0}\033[0m').format(', '.join(def_supported_keys)))
+    args.add_argument('-o',
+                      '--output',
+                      default = sys.stdout,
+                      metavar = 'OUTFILE',
+                      dest = 'output',
+                      help = ('If specified, write the hostkeys to \033[1m{0}\033[0m instead of ' +
+                              '\033[1m{1}\033[0m (the default). ' +
+                              'Overrides \033[1m{2}\033[0m').format('OUTFILE',
+                                                                    'stdout',
+                                                                    '-S/--system-wide'))
+    args.add_argument('-S',
+                      '--system-wide',
+                      dest = 'system',
+                      action = 'store_true',
+                      help = ('If specified, apply to the entire system (not just the ' +
+                              'specified/running user) via {0}. ' +
+                              'Requires \033[1m{1}\033[0m in /etc/ssh/ssh_config (usually ' +
+                              'enabled silently by default) and running with root ' +
+                              'privileges').format(def_syshostkeys,
+                                                   'GlobalKnownHostsFile {0}'.format(def_syshostkeys)))
+    args.add_argument(metavar = 'HOSTLIST_FILE',
+                      dest = 'infile',
+                      help = ('The path to the list of hosts. Can contain blank lines and/or comments. ' +
+                              'One host per line. Can be \033[1m{0}\033[0m (as long as it\'s resolvable), ' +
+                              '\033[1m{1}\033[0m, or \033[1m{2}\033[0m. To specify an alternate port, ' +
+                              'add \033[1m{3}\033[0m to the end (e.g. ' +
+                              '"some.host.tld:22")').format('hostname',
+                                                            'IP address',
+                                                            'FQDN',
+                                                            ':<PORTNUM>'))
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    scan = hostscanner(args)
+    scan.getHosts()
+    scan.getKeys()
+    scan.write()
+
+if __name__ == '__main__':
+    main()

aif/scripts/post/sshkeys.py

@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+
+import os
+import pwd
+from urllib.request import urlopen
+
+keysfile = 'https://square-r00t.net/ssh/all'
+
+def copyKeys(keystring, user = 'root'):
+    uid = pwd.getpwnam(user).pw_uid
+    gid = pwd.getpwnam(user).pw_gid
+    homedir = os.path.expanduser('~{0}'.format(user))
+    sshdir = '{0}/.ssh'.format(homedir)
+    authfile = '{0}/authorized_keys'.format(sshdir)
+    os.makedirs(sshdir, mode = 0o700, exist_ok = True)
+    with open(authfile, 'a') as f:
+        f.write(keystring)
+    for basedir, dirs, files in os.walk(sshdir):
+        os.chown(basedir, uid, gid)
+        os.chmod(basedir, 0o700)
+        for f in files:
+            os.chown(os.path.join(basedir, f), uid, gid)
+            os.chmod(os.path.join(basedir, f), 0o600)
+    return()
+
+def main():
+    with urlopen(keysfile) as keys:
+        copyKeys(keys.read().decode('utf-8'))
+
+if __name__ == '__main__':
+    main()

aif/scripts/post/sshsecure.py

@@ -0,0 +1,428 @@
+#!/usr/bin/env python3
+
+# Pythonized automated way of running https://sysadministrivia.com/news/hardening-ssh-security
+# TODO: check for cryptography module. if it exists, we can do this entirely pythonically
+#       without ever needing to use subprocess/ssh-keygen, i think!
+
+# Thanks to https://stackoverflow.com/a/39126754.
+
+# Also, I need to re-write this. It's getting uglier.
+
+# stdlib
+import datetime
+import glob
+import os
+import pwd
+import re
+import signal
+import shutil
+import subprocess  # REMOVE WHEN SWITCHING TO PURE PYTHON
+#### PREP FOR PURE PYTHON IMPLEMENTATION ####
+# # non-stdlib - testing and automatic install if necessary.
+# # TODO #
+# - cryptography module won't generate new-format "openssh-key-v1" keys.
+# - See https://github.com/pts/py_ssh_keygen_ed25519 for possible conversion to python 3
+# - https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.key
+# - https://github.com/pyca/cryptography/issues/3509 and https://github.com/paramiko/paramiko/issues/1136
+# has_crypto = False
+# pure_py = False
+# has_pip = False
+# pipver = None
+# try:
+#     import cryptography
+#     has_crypto = True
+# except ImportError:
+#     # We'll try to install it. We set up the logic below.
+#     try:
+#         import pip
+#         has_pip = True
+#         # We'll use these to create a temporary lib path and remove it when done.
+#         import sys
+#         import tempfile
+#     except ImportError:
+#         # ABSOLUTE LAST fallback, if we got to THIS case, is to use subprocess.
+#         has_pip = False
+#         import subprocess
+#
+# # Try installing it then!
+# if not all((has_crypto, )):
+#     # venv only included after python 3.3.x. We fallback to subprocess if we can't do dis.
+#     if sys.hexversion >= 0x30300f0:
+#         has_ensurepip = False
+#         import venv
+#         if not has_pip and sys.hexversion >= 0x30400f0:
+#             import ensurepip
+#             has_ensurepip = True
+#         temppath = tempfile.mkdtemp('_VENV')
+#         v = venv.create(temppath)
+#         if has_ensurepip and not has_pip:
+#             # This SHOULD be unnecessary, but we want to try really hard.
+#             ensurepip.bootstrap(root = temppath)
+#             import pip
+#             has_pip = True
+#         if has_pip:
+#             pipver = pip.__version__.split('.')
+#             # A thousand people are yelling at me for this.
+#             if int(pipver[0]) >= 10:
+#                 from pip._internal import main as pipinstall
+#             else:
+#                 pipinstall = pip.main
+#             if int(pipver[0]) >= 8:
+#                 pipcmd = ['install',
+#                           '--prefix={0}'.format(temppath),
+#                           '--ignore-installed']
+#             else:
+#                 pipcmd = ['install',
+#                           '--install-option="--prefix={0}"'.format(temppath),
+#                           '--ignore-installed']
+#             # Get the lib path.
+#             libpath = os.path.join(temppath, 'lib')
+#             if os.path.exists('{0}64'.format(libpath)) and not os.path.islink('{0}64'.format(libpath)):
+#                 libpath += '64'
+#             for i in os.listdir(libpath):  # TODO: make this more sane. We cheat a bit here by making assumptions.
+#                 if re.search('python([0-9]+(\.[0-9]+)?)?$', i):
+#                     libpath = os.path.join(libpath, i)
+#                     break
+#             libpath = os.path.join(libpath, 'site-packages')
+#             sys.prefix = temppath
+#             for m in ('cryptography', 'ed25519'):
+#                 pipinstall(['install', 'cryptography'])
+#             sys.path.append(libpath)
+#             try:
+#                 import cryptography
+#                 has_crypto = True
+#             except ImportError:  # All that trouble for nothin'. Shucks.
+#                 pass
+#
+# if all((has_crypto, )):
+#     pure_py = True
+#
+# if pure_py:
+#     from cryptography.hazmat.primitives import serialization as crypto_serialization
+#     from cryptography.hazmat.primitives.asymmetric import rsa
+#     from cryptography.hazmat.backends import default_backend as crypto_default_backend
+#
+
+# We need static backup suffixes.
+tstamp = int(datetime.datetime.utcnow().timestamp())
+
+# TODO: associate various config directives with version, too.
+# For now, we use this for primarily CentOS 6.x, which doesn't support ED25519 and probably some of the MACs.
+# Bastards.
+# https://ssh-comparison.quendi.de/comparison/cipher.html at some point in the future...
+# TODO: maybe implement some parsing of the ssh -Q stuff? https://superuser.com/a/869005/984616
+# If you encounter a version incompatibility, please let me know!
+# nmap --script ssh2-enum-algos -PN -sV -p22 <host>
+magic_ver = 6.5
+ssh_ver = subprocess.run(['ssh', '-V'], stderr = subprocess.PIPE).stderr.decode('utf-8').strip().split()[0]
+# FUCK YOU, DEBIAN. FUCK YOU  AND ALL OF YOUR DERIVATIVES. YOU'RE FUCKING TRASH.
+# YOU BELONG NOWHERE NEAR A DATACENTER.
+ssh_ver = float(re.sub('^(?:Open|Sun_)SSH_([0-9\.]+)(?:p[0-9]+)?(?:,.*)?.*$', '\g<1>', ssh_ver))
+if ssh_ver >= magic_ver:
+    has_ed25519 = True
+    supported_keys = ('ed25519', 'rsa')
+    new_moduli = False
+else:
+    has_ed25519 = False
+    supported_keys = ('rsa', )
+    new_moduli = False
+# https://github.com/openssh/openssh-portable/commit/3e60d18fba1b502c21d64fc7e81d80bcd08a2092
+if ssh_ver >= 8.1:
+    new_moduli = True
+
+
+conf_options = {}
+conf_options['sshd'] = {'KexAlgorithms': 'diffie-hellman-group-exchange-sha256',
+                        'Protocol': '2',
+                        'HostKey': ['/etc/ssh/ssh_host_rsa_key'],
+                        #'PermitRootLogin': 'prohibit-password',  # older daemons don't like "prohibit-..."
+                        'PermitRootLogin': 'without-password',
+                        'PasswordAuthentication': 'no',
+                        'ChallengeResponseAuthentication': 'no',
+                        'PubkeyAuthentication': 'yes',
+                        'Ciphers': 'aes256-ctr,aes192-ctr,aes128-ctr',
+                        'MACs': 'hmac-sha2-512,hmac-sha2-256'}
+if has_ed25519:
+    conf_options['sshd']['HostKey'].append('/etc/ssh/ssh_host_ed25519_key')
+    conf_options['sshd']['KexAlgorithms'] = ','.join(('curve25519-sha256@libssh.org',
+                                                      conf_options['sshd']['KexAlgorithms']))
+    conf_options['sshd']['Ciphers'] = ','.join((('chacha20-poly1305@openssh.com,'
+                                                 'aes256-gcm@openssh.com,'
+                                                 'aes128-gcm@openssh.com'),
+                                                conf_options['sshd']['Ciphers']))
+    conf_options['sshd']['MACs'] = ','.join((('hmac-sha2-512-etm@openssh.com,'
+                                              'hmac-sha2-256-etm@openssh.com,'
+                                              'umac-128-etm@openssh.com'),
+                                             conf_options['sshd']['MACs'],
+                                             'umac-128@openssh.com'))
+# Uncomment if this is further configured
+#conf_options['sshd']['AllowGroups'] = 'ssh-user'
+
+conf_options['ssh'] = {'Host': {'*': {'KexAlgorithms': 'diffie-hellman-group-exchange-sha256',
+                                      'PubkeyAuthentication': 'yes',
+                                      'HostKeyAlgorithms': 'ssh-rsa'}}}
+if has_ed25519:
+    conf_options['ssh']['Host']['*']['KexAlgorithms'] = ','.join(('curve25519-sha256@libssh.org',
+                                                                  conf_options['ssh']['Host']['*']['KexAlgorithms']))
+    conf_options['ssh']['Host']['*']['HostKeyAlgorithms'] = ','.join(
+                                                            (('ssh-ed25519-cert-v01@openssh.com,'
+                                                              'ssh-rsa-cert-v01@openssh.com,'
+                                                              'ssh-ed25519'),
+                                                             conf_options['ssh']['Host']['*']['HostKeyAlgorithms']))
+
+
+def hostKeys(buildmoduli):
+    # Starting haveged should help lessen the time load a non-negligible amount, especially on virtual platforms.
+    if os.path.lexists('/usr/bin/haveged'):
+        # We could use psutil here, but then that's a python dependency we don't need.
+        # We could parse the /proc directory, but that's quite unnecessary. pgrep's installed by default on
+        # most distros.
+        with open(os.devnull, 'wb') as devnull:
+            if subprocess.run(['pgrep', 'haveged'], stdout = devnull).returncode != 0:
+                subprocess.run(['haveged'], stdout = devnull)
+    #Warning: The moduli stuff takes a LONG time to run. Hours.
+    if buildmoduli:
+        if not new_moduli:
+            subprocess.run(['ssh-keygen',
+                            '-G', '/etc/ssh/moduli.all',
+                            '-b', '4096',
+                            '-q'])
+            subprocess.run(['ssh-keygen',
+                            '-T', '/etc/ssh/moduli.safe',
+                            '-f', '/etc/ssh/moduli.all',
+                            '-q'])
+        else:
+            subprocess.run(['ssh-keygen',
+                            '-q',
+                            '-M', 'generate',
+                            '-O', 'bits=4096',
+                            '/etc/ssh/moduli.all'])
+            subprocess.run(['ssh-keygen',
+                            '-q',
+                            '-M', 'screen',
+                            '-f', '/etc/ssh/moduli.all',
+                            '/etc/ssh/moduli.safe'])
+        if os.path.lexists('/etc/ssh/moduli'):
+            os.rename('/etc/ssh/moduli', '/etc/ssh/moduli.old')
+        os.rename('/etc/ssh/moduli.safe', '/etc/ssh/moduli')
+        os.remove('/etc/ssh/moduli.all')
+    for suffix in ('', '.pub'):
+        for k in glob.glob('/etc/ssh/ssh_host_*key{0}'.format(suffix)):
+            os.rename(k, '{0}.old.{1}'.format(k, tstamp))
+    if has_ed25519:
+        subprocess.run(['ssh-keygen',
+                        '-t', 'ed25519',
+                        '-f', '/etc/ssh/ssh_host_ed25519_key',
+                        '-q',
+                        '-N', ''])
+    subprocess.run(['ssh-keygen',
+                    '-t', 'rsa',
+                    '-b', '4096',
+                    '-f', '/etc/ssh/ssh_host_rsa_key',
+                    '-q',
+                    '-N', ''])
+    # We currently don't use this, but for simplicity's sake let's return the host keys.
+    hostkeys = {}
+    for k in supported_keys:
+        with open('/etc/ssh/ssh_host_{0}_key.pub'.format(k), 'r') as f:
+            hostkeys[k] = f.read()
+    return(hostkeys)
+
+def config(opts, t):
+    special = {'sshd': {}, 'ssh': {}}
+    # We need to handle these directives a little differently...
+    special['sshd']['opts'] = ['Match']
+    special['sshd']['filters'] = ['User', 'Group', 'Host', 'LocalAddress', 'LocalPort', 'Address']
+    # These are arguments supported by each of the special options. We'll use this to verify entries.
+    special['sshd']['args'] = ['AcceptEnv', 'AllowAgentForwarding', 'AllowGroups', 'AllowStreamLocalForwarding',
+                               'AllowTcpForwarding', 'AllowUsers', 'AuthenticationMethods', 'AuthorizedKeysCommand',
+                               'AuthorizedKeysCommandUser', 'AuthorizedKeysFile', 'AuthorizedPrincipalsCommand',
+                               'AuthorizedPrincipalsCommandUser', 'AuthorizedPrincipalsFile', 'Banner',
+                               'ChrootDirectory', 'ClientAliveCountMax', 'ClientAliveInterval', 'DenyGroups',
+                               'DenyUsers', 'ForceCommand', 'GatewayPorts', 'GSSAPIAuthentication',
+                               'HostbasedAcceptedKeyTypes', 'HostbasedAuthentication',
+                               'HostbasedUsesNameFromPacketOnly', 'IPQoS', 'KbdInteractiveAuthentication',
+                               'KerberosAuthentication', 'MaxAuthTries', 'MaxSessions', 'PasswordAuthentication',
+                               'PermitEmptyPasswords', 'PermitOpen', 'PermitRootLogin', 'PermitTTY', 'PermitTunnel',
+                               'PermitUserRC', 'PubkeyAcceptedKeyTypes', 'PubkeyAuthentication', 'RekeyLimit',
+                               'RevokedKeys', 'StreamLocalBindMask', 'StreamLocalBindUnlink', 'TrustedUserCAKeys',
+                               'X11DisplayOffset', 'X11Forwarding', 'X11UseLocalHost']
+    special['ssh']['opts'] = ['Host', 'Match']
+    special['ssh']['args'] = ['canonical', 'exec', 'host', 'originalhost', 'user', 'localuser']
+    cf = '/etc/ssh/{0}_config'.format(t)
+    shutil.copy2(cf, '{0}.bak.{1}'.format(cf, tstamp))
+    with open(cf, 'r') as f:
+        conf = f.readlines()
+    conf.append('\n\n# Added per https://sysadministrivia.com/news/hardening-ssh-security\n\n')
+    confopts = []
+    # Get an index of directives pre-existing in the config file.
+    for line in conf[:]:
+        opt = line.split()
+        if opt:
+            if not re.match('^(#.*|\s+.*)$', opt[0]):
+                confopts.append(opt[0])
+    # We also need to modify the config file- comment out starting with the first occurrence of the 
+    # specopts, if it exists. This is why we make a backup.
+    commentidx = None
+    for idx, i in enumerate(conf):
+        if re.match('^({0})\s+.*$'.format('|'.join(special[t]['opts'])), i):
+            commentidx = idx
+            break
+    if commentidx is not None:
+        idx = commentidx
+        while idx <= (len(conf) - 1):
+            conf[idx] = '#{0}'.format(conf[idx])
+            idx += 1
+    # Now we actually start replacing/adding some major configuration.
+    for o in opts.keys():
+        if o in special[t]['opts'] or isinstance(opts[o], dict):
+            # We need to put these at the bottom of the file due to how they're handled by sshd's config parsing.
+            continue
+        # We handle these a little specially too- they're for multiple lines sharing the same directive.
+        # Since the config should be explicit, we remove any existing entries specified that we find.
+        else:
+            if o in confopts:
+                # If I was more worried about recursion, or if I was appending here, I should use conf[:].
+                # But I'm not. So I won't.
+                for idx, opt in enumerate(conf):
+                    if re.match('^{0}(\s.*)?\n$'.format(o), opt):
+                        conf[idx] = '#{0}'.format(opt)
+            # Here we handle the "multiple-specifying" options- notably, HostKey.
+            if isinstance(opts[o], list):
+                for l in opts[o]:
+                    if l is not None:
+                        conf.append('{0} {1}\n'.format(o, l))
+                    else:
+                        conf.append('{0}\n'.format(o))
+            else:
+                # So it isn't something we explicitly save until the end (such as a Match or Host),
+                # and it isn't something that's specified multiple times.
+                if opts[o] is not None:
+                    conf.append('{0} {1}\n'.format(o, opts[o]))
+                else:
+                    conf.append('{0}\n'.format(o))
+    # NOW we can add the Host/Match/etc. directives.
+    for o in opts.keys():
+        if isinstance(opts[o], dict):
+            for k in opts[o].keys():
+                conf.append('{0} {1}\n'.format(o, k))
+                for l in opts[o][k].keys():
+                    if opts[o][k][l] is not None:
+                        conf.append('\t{0} {1}\n'.format(l, opts[o][k][l]))
+                    else:
+                        conf.append('\t{0}\n'.format(l))
+    with open(cf, 'w') as f:
+        f.write(''.join(conf))
+    return()
+
+def clientKeys(user = 'root'):
+    uid = pwd.getpwnam(user).pw_uid
+    gid = pwd.getpwnam(user).pw_gid
+    homedir = os.path.expanduser('~{0}'.format(user))
+    sshdir = '{0}/.ssh'.format(homedir)
+    os.makedirs(sshdir, mode = 0o700, exist_ok = True)
+    if has_ed25519:
+        if not os.path.lexists('{0}/id_ed25519'.format(sshdir)) \
+                and not os.path.lexists('{0}/id_ed25519.pub'.format(sshdir)):
+            subprocess.run(['ssh-keygen',
+                            '-t', 'ed25519',
+                            '-o',
+                            '-a', '100',
+                            '-f', '{0}/id_ed25519'.format(sshdir),
+                            '-q',
+                            '-N', ''])
+    if not os.path.lexists('{0}/id_rsa'.format(sshdir)) and not os.path.lexists('{0}/id_rsa.pub'.format(sshdir)):
+        if has_ed25519:
+            subprocess.run(['ssh-keygen',
+                            '-t', 'rsa',
+                            '-b', '4096',
+                            '-o',
+                            '-a', '100',
+                            '-f', '{0}/id_rsa'.format(sshdir),
+                            '-q',
+                            '-N', ''])
+        else:
+            subprocess.run(['ssh-keygen',
+                            '-t', 'rsa',
+                            '-b', '4096',
+                            '-a', '100',
+                            '-f', '{0}/id_rsa'.format(sshdir),
+                            '-q',
+                            '-N', ''])
+    for basedir, dirs, files in os.walk(sshdir):
+        os.chown(basedir, uid, gid)
+        os.chmod(basedir, 0o700)
+        for f in files:
+            os.chown(os.path.join(basedir, f), uid, gid)
+            os.chmod(os.path.join(basedir, f), 0o600)
+    if 'pubkeys' not in globals():
+        pubkeys = {}
+    pubkeys[user] = {}
+    for k in supported_keys:
+        with open('{0}/id_{1}.pub'.format(sshdir, k), 'r') as f:
+            pubkeys[user][k] = f.read()
+    return(pubkeys)
+
+def daemonMgr():
+    # In case the script is running without sshd running.
+    pidfile = '/var/run/sshd.pid'
+    if not os.path.isfile(pidfile):
+        return()
+    # We're about to do somethin' stupid. Let's make it a teeny bit less stupid.
+    with open(os.devnull, 'w') as devnull:
+        confchk = subprocess.run(['sshd', '-T'], stdout = devnull)
+    if confchk.returncode != 0:
+        for suffix in ('', '.pub'):
+            for k in glob.glob('/etc/ssh/ssh_host_*key{0}'.format(suffix)):
+                os.rename('{0}.old.{1}'.format(k, tstamp), k)
+            for conf in ('', 'd'):
+                cf = '/etc/ssh/ssh{0}_config'.format(conf)
+                os.rename('{0}.{1}'.format(cf, tstamp),
+                          cf)
+        exit('OOPS. We goofed. Backup restored and bailing out.')
+    # We need to restart sshd once we're done. I feel dirty doing this, but this is the most cross-platform way I can
+    # do it. First, we need the path to the PID file.
+    # TODO: do some kind of better way of doing this.
+    with open('/etc/ssh/sshd_config', 'r') as f:
+        for line in f.readlines():
+            if re.search('^\s*PidFile\s+.*', line):
+                pidfile = re.sub('^\s*PidFile\s+(.*)(#.*)?$', '\g<1>', line)
+                break
+    with open(pidfile, 'r') as f:
+        pid = int(f.read().strip())
+    os.kill(pid, signal.SIGHUP)
+    return()
+
+def main():
+    self_pidfile = '/tmp/sshsecure.pid'
+    is_running = False
+    # First, check to see if we're already running.
+    # This is where I'd put a psutil call... IF I HAD ONE.
+    if os.path.isfile(self_pidfile):
+        is_running = subprocess.run(['pgrep', '-F', self_pidfile], stdout = subprocess.PIPE)
+        if is_running.stdout.decode('utf-8').strip() != '':
+            # We're still running. Exit gracefully.
+            print('We seem to still be running from a past execution; exiting')
+            exit(0)
+        else:
+            # It's a stale PID file.
+            os.remove(self_pidfile)
+    with open(self_pidfile, 'w') as f:
+        f.write(str(os.getpid()) + '\n')
+    _chkfile = '/etc/ssh/.aif-generated'
+    if not os.path.isfile(_chkfile):
+        # Warning: The moduli stuff can take a LONG time to run. Hours.
+        buildmoduli = True
+        hostKeys(buildmoduli)
+    for t in ('sshd', 'ssh'):
+        config(conf_options[t], t)
+    clientKeys()
+    with open(_chkfile, 'w') as f:
+        f.write(('ssh, sshd, and hostkey configurations/keys have been modified by sshsecure.py from OpTools.\n'
+                 'https://git.square-r00t.net/OpTools/\n'))
+    daemonMgr()
+    os.remove(self_pidfile)
+    return()
+
+if __name__ == '__main__':
+    main()

arch/arch_mirror_ranking.py

@@ -0,0 +1,147 @@
+#!/usr/bin/env python3
+
+import argparse
+import datetime
+# import dns  # TODO: replace server['ipv4'] with IPv4 address(es)? etc.
+import json
+import re
+import sys
+from urllib.request import urlopen
+##
+import iso3166
+
+
+servers_json_url = 'https://www.archlinux.org/mirrors/status/json/'
+protos = ('http', 'https', 'rsync')
+
+
+class MirrorIdx(object):
+    def __init__(self, country = None, proto = None, is_active = None, json_url = servers_json_url,
+                 name_re = None, ipv4 = None, ipv6 = None, isos = None, statuses = False, *args, **kwargs):
+        _tmpargs = locals()
+        del (_tmpargs['self'])
+        for k, v in _tmpargs.items():
+            setattr(self, k, v)
+        self.validateParams()
+        self.servers_json = {}
+        self.servers = []
+        self.servers_with_scores = []
+        self.ranked_servers = []
+        self.fetchJSON()
+        self.buildServers()
+        self.rankServers()
+
+    def fetchJSON(self):
+        if self.statuses:
+            sys.stderr.write('Fetching servers from {0}...\n'.format(self.json_url))
+        with urlopen(self.json_url) as u:
+            self.servers_json = json.load(u)
+        return()
+
+    def buildServers(self):
+        _limiters = (self.proto, self.ipv4, self.ipv6, self.isos)
+        _filters = list(_limiters)
+        _filters.extend([self.name_re, self.country])
+        _filters = tuple(_filters)
+        if self.statuses:
+            sys.stderr.write('Applying filters (if any)...\n')
+        for s in self.servers_json['urls']:
+            # We handle these as "tri-value" (None, True, False)
+            if self.is_active is not None:
+                if s['active'] != self.is_active:
+                    continue
+            if not any(_filters):
+                self.servers.append(s.copy())
+                if s['score']:
+                    self.servers_with_scores.append(s)
+                continue
+            # These are based on string values.
+            if self.name_re:
+                if not self.name_re.search(s['url']):
+                    continue
+            if self.country:
+                if self.country != s['country_code']:
+                    continue
+            # These are regular True/False switches
+            match = False
+            # We want to be *very* explicit about the ordering and inclusion/exclusion of these.
+            # They MUST match the order of _limiters.
+            values = []
+            for k in ('protocol', 'ipv4', 'ipv6', 'isos'):
+                values.append(s[k])
+            valid = all([v for k, v in zip(_limiters, values) if k])
+            if valid:
+                self.servers.append(s)
+                if s['score']:
+                    self.servers_with_scores.append(s)
+        return()
+
+    def rankServers(self):
+        if self.statuses:
+            sys.stderr.write('Ranking mirrors...\n')
+        self.ranked_servers = sorted(self.servers_with_scores, key = lambda i: i['score'])
+        return()
+
+    def validateParams(self):
+        if self.proto and self.proto.lower() not in protos:
+            err = '{0} must be one of: {1}'.format(self.proto, ', '.join([i.upper() for i in protos]))
+            raise ValueError(err)
+        elif self.proto:
+            self.proto = self.proto.upper()
+        if self.country and self.country.upper() not in iso3166.countries:
+            err = ('{0} must be a valid ISO-3166-1 ALPHA-2 country code. '
+                   'See https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes'
+                   '#Current_ISO_3166_country_codes').format(self.country)
+            raise ValueError()
+        elif self.country:
+            self.country = self.country.upper()
+        if self.name_re:
+            self.name_re = re.compile(self.name_re)
+        return()
+
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = 'Fetch and rank Arch Linux mirrors')
+    args.add_argument('-c', '--country',
+                      dest = 'country',
+                      help = ('If specified, limit results to this country (in ISO-3166-1 ALPHA-2 format)'))
+    args.add_argument('-p', '--protocol',
+                      choices = protos,
+                      dest = 'proto',
+                      help = ('If specified, limit results to this protocol'))
+    args.add_argument('-r', '--name-regex',
+                      dest = 'name_re',
+                      help = ('If specified, limit results to URLs that match this regex pattern (Python re syntax)'))
+    args.add_argument('-4', '--ipv4',
+                      dest = 'ipv4',
+                      action = 'store_true',
+                      help = ('If specified, limit results to servers that support IPv4'))
+    args.add_argument('-6', '--ipv6',
+                      dest = 'ipv6',
+                      action = 'store_true',
+                      help = ('If specified, limit results to servers that support IPv6'))
+    args.add_argument('-i', '--iso',
+                      dest = 'isos',
+                      action = 'store_true',
+                      help = ('If specified, limit results to servers that have ISO images'))
+    is_active = args.add_mutually_exclusive_group()
+    is_active.add_argument('-a', '--active-only',
+                           default = None,
+                           const = True,
+                           action = 'store_const',
+                           dest = 'is_active',
+                           help = ('If specified, only include active servers (default is active + inactive)'))
+    is_active.add_argument('-n', '--inactive-only',
+                           default = None,
+                           const = False,
+                           action = 'store_const',
+                           dest = 'is_active',
+                           help = ('If specified, only include inactive servers (default is active + inactive)'))
+    return(args)
+
+
+if __name__ == '__main__':
+    args = vars(parseArgs().parse_args())
+    m = MirrorIdx(**args, statuses = True)
+    for s in m.ranked_servers:
+        print('Server = {0}$repo/os/$arch'.format(s['url']))

arch/autopkg/maintain.py

@@ -0,0 +1,165 @@
+#!/usr/bin/env python
+
+import argparse
+import json
+import os
+import sqlite3
+import run
+from urllib.request import urlopen
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = ('Modify (add/remove) packages for use with Autopkg'),
+                                   epilog = ('Operation-specific help; try e.g. "add --help"'))
+    commonargs = argparse.ArgumentParser(add_help = False)
+    commonargs.add_argument('-n', '--name',
+                            dest = 'pkgnm',
+                            required = True,
+                            help = ('The name of the PACKAGE to operate on.'))
+    commonargs.add_argument('-d', '--db',
+                            dest = 'dbfile',
+                            default = '~/.optools/autopkg.sqlite3',
+                            help = ('The location of the package database. THIS SHOULD NOT BE ANY FILE USED BY '
+                                    'ANYTHING ELSE! A default one will be created if it doesn\'t exist'))
+    subparsers = args.add_subparsers(help = ('Operation to perform'),
+                                     metavar = 'OPERATION',
+                                     dest = 'oper')
+    addargs = subparsers.add_parser('add',
+                                    parents = [commonargs],
+                                    help = ('Add a package. If a matching package NAME exists (-n/--name), '
+                                            'we\'ll replace it'))
+    addargs.add_argument('-b', '--base',
+                         dest = 'pkgbase',
+                         default = None,
+                         help = ('The pkgbase; only really needed for split-packages and we will automatically '
+                                 'fetch if it\'s left blank anyways'))
+    addargs.add_argument('-v', '--version',
+                         dest = 'pkgver',
+                         default = None,
+                         help = ('The current version; we will automatically fetch it if it\'s left blank'))
+    addargs.add_argument('-l', '--lock',
+                         dest = 'active',
+                         action = 'store_false',
+                         help = ('If specified, the package will still exist in the DB but it will be marked inactive'))
+    rmargs = subparsers.add_parser('rm',
+                                   parents = [commonargs],
+                                   help = ('Remove a package from the DB'))
+    buildargs = subparsers.add_parser('build',
+                                      help = ('Build all packages; same effect as running run.py'))
+    buildargs.add_argument('-d', '--db',
+                           dest = 'dbfile',
+                           default = '~/.optools/autopkg.sqlite3',
+                           help = ('The location of the package database. THIS SHOULD NOT BE ANY FILE USED BY '
+                                   'ANYTHING ELSE! A default one will be created if it doesn\'t exist'))
+    listargs = subparsers.add_parser('ls',
+                                     help = ('List packages (and information about them) only'))
+    listargs.add_argument('-d', '--db',
+                          dest = 'dbfile',
+                          default = '~/.optools/autopkg.sqlite3',
+                          help = ('The location of the package database. THIS SHOULD NOT BE ANY FILE USED BY '
+                                  'ANYTHING ELSE! A default one will be created if it doesn\'t exist'))
+    return(args)
+
+def add(args):
+    db = sqlite3.connect(args['dbfile'])
+    db.row_factory = sqlite3.Row
+    cur = db.cursor()
+    if not all((args['pkgbase'], args['pkgver'])):
+        # We need some additional info from the AUR API...
+        aur_url = 'https://aur.archlinux.org/rpc/?v=5&type=info&by=name&arg%5B%5D={0}'.format(args['pkgnm'])
+        with urlopen(aur_url) as url:
+            aur = json.loads(url.read().decode('utf-8'))['results']
+        if not aur:
+            raise ValueError(('Either something is screwy with our network access '
+                              'or the package {0} doesn\'t exist').format(args['pkgnm']))
+        if ((aur['PackageBase'] != aur['Name']) and (not args['pkgbase'])):
+            args['pkgbase'] = aur['PackageBase']
+        if not args['pkgver']:
+            args['pkgver'] = aur['Version']
+    cur.execute("SELECT id, pkgname, pkgbase, pkgver, active FROM packages WHERE pkgname = ?",
+                (args['pkgnm'], ))
+    row = cur.fetchone()
+    if row:
+        if args['pkgbase']:
+            q = ("UPDATE packages SET pkgbase = ? AND pkgver = ? AND ACTIVE = ? WHERE id = ?",
+                 (args['pkgbase'], args['pkgver'], ('0' if args['lock'] else '1'), row['id']))
+        else:
+            q = ("UPDATE packages SET pkgver = ? AND ACTIVE = ? WHERE id = ?",
+                 (args['pkgver'], ('0' if args['lock'] else '1'), row['id']))
+    else:
+        if args['pkgbase']:
+            q = (("INSERT INTO "
+                    "packages (pkgname, pkgbase, pkgver, active) "
+                    "VALUES (?, ?, ?, ?)"),
+                 (args['pkgnm'], args['pkgbase'], args['pkgver'], ('0' if args['lock'] else '1')))
+        else:
+            q = (("INSERT INTO "
+                    "packages (pkgname, pkgver, active) "
+                    "VALUES (?, ?, ?)"),
+                 (args['pkgnm'], args['pkgver'], ('0' if args['lock'] else '1')))
+    cur.execute(*q)
+    db.commit()
+    cur.close()
+    db.close()
+    return()
+
+def rm(args):
+    db = sqlite3.connect(args['dbfile'])
+    cur = db.cursor()
+    cur.execute("DELETE FROM packages WHERE pkgname = ?",
+                (args['pkgnm'], ))
+    db.commit()
+    cur.close()
+    db.close()
+    return()
+
+def build(args):
+    pm = run.PkgMake(db = args['dbfile'])
+    pm.main()
+    return()
+
+def ls(args):
+    db = sqlite3.connect(args['dbfile'])
+    db.row_factory = sqlite3.Row
+    cur = db.cursor()
+    rows = []
+    cur.execute("SELECT * FROM packages ORDER BY pkgname")
+    for r in cur.fetchall():
+        pkgnm = r['pkgname']
+        rows.append({'name': r['pkgname'],
+                     'row_id': r['id'],
+                     'pkgbase': ('' if not r['pkgbase'] else r['pkgbase']),
+                     'ver': r['pkgver'],
+                     'enabled': ('Yes' if r['active'] else 'No')})
+    header = '|      NAME      |  PACKAGE BASE  | VERSION | ENABLED | ROW ID |'
+    sep = '=' * len(header)
+    fmt = '|{name:<16}|{pkgbase:<16}|{ver:^9}|{enabled:^9}|{row_id:<8}|'
+    out = []
+    for row in rows:
+        out.append(fmt.format(**row))
+    header = '\n'.join((sep, header, sep))
+    out.insert(0, header)
+    out.append(sep)
+    print('\n'.join(out))
+    cur.close()
+    db.close()
+    return()
+
+def main():
+    rawargs = parseArgs()
+    args = vars(rawargs.parse_args())
+    if not args['oper']:
+        rawargs.print_help()
+        exit()
+    args['dbfile'] = os.path.abspath(os.path.expanduser(args['dbfile']))
+    if args['oper'] == 'add':
+        add(args)
+    elif args['oper'] == 'rm':
+        rm(args)
+    elif args['oper'] == 'build':
+        build(args)
+    elif args['oper'] == 'ls':
+        ls(args)
+    return()
+
+if __name__ == '__main__':
+    main()

arch/autopkg/run.py

@@ -0,0 +1,278 @@
+#!/usr/bin/env python
+
+import grp
+import json
+import os
+import pwd
+import re
+import shutil
+import sqlite3
+import subprocess
+import tarfile
+import urllib.request as reqs
+import urllib.parse as urlparse
+import setup
+# I *HATE* relying on non-stlib, and I hate even MORE that this is JUST TO COMPARE VERSION STRINGS.
+# WHY IS THIS FUNCTIONALITY NOT STDLIB YET.
+try:
+    from distutils.version import LooseVersion
+    has_lv = True
+except ImportError:
+    has_lv = False
+
+# The base API URL (https://wiki.archlinux.org/index.php/Aurweb_RPC_interface)
+aur_base = 'https://aur.archlinux.org/rpc/?v=5&type=info&by=name'
+# The length of the above. Important because of uri_limit.
+base_len = len(aur_base)
+# Maximum length of the URI.
+uri_limit = 4443
+
+class PkgMake(object):
+    def __init__(self, db = '~/.optools/autopkg.sqlite3'):
+        db = os.path.abspath(os.path.expanduser(db))
+        if not os.path.isfile(db):
+            setup.firstrun(db)
+        self.conn = sqlite3.connect(db)
+        self.conn.row_factory = sqlite3.Row
+        self.cur = self.conn.cursor()
+        self.cfg = setup.main(self.conn, self.cur)
+        if self.cfg['sign']:
+            _cmt_mode = self.conn.isolation_level  # autocommit
+            self.conn.isolation_level = None
+            self.fpr, self.gpg = setup.GPG(self.cur, homedir = self.cfg['gpg_homedir'], keyid = self.cfg['gpg_keyid'])
+            self.conn.isolation_level = _cmt_mode
+            # don't need this anymore; it should be duplicated or populated into self.fpr.
+            del(self.cfg['gpg_keyid'])
+            self.my_key = self.gpg.get_key(self.fpr, secret = True)
+            self.gpg.signers = [self.my_key]
+        else:
+            self.fpr = self.gpg = self.my_key = None
+            del(self.cfg['gpg_keyid'])
+        self.pkgs = {}
+        self._populatePkgs()
+
+    def main(self):
+        self.getPkg()
+        self.buildPkg()
+        return()
+
+    def _chkver(self, pkgbase):
+        new_ver = self.pkgs[pkgbase]['meta']['new_ver']
+        old_ver = self.pkgs[pkgbase]['meta']['pkgver']
+        is_diff = (new_ver != old_ver)  # A super-stupid fallback
+        if is_diff:
+            if has_lv:
+                is_diff = LooseVersion(new_ver) > LooseVersion(old_ver)
+            else:
+                # like, 90% of the time, this would work.
+                new_tuple = tuple(map(int, (re.split('\.|-', new_ver))))
+                old_tuple = tuple(map(int, (re.split('\.|-', old_ver))))
+                # But people at https://stackoverflow.com/a/11887825/733214 are very angry about it, hence the above.
+                is_diff = new_tuple > old_tuple
+        return(is_diff)
+
+    def _populatePkgs(self):
+        # These columns/keys are inferred by structure or unneeded. Applies to both DB and AUR API.
+        _notrack = ('pkgbase', 'pkgname', 'active', 'id', 'packagebaseid', 'numvotes', 'popularity', 'outofdate',
+                    'maintainer', 'firstsubmitted', 'lastmodified', 'depends', 'optdepends', 'conflicts', 'license',
+                    'keywords')
+        _attr_map = {'version': 'new_ver'}
+        # These are tracked per-package; all others are pkgbase and applied to all split pkgs underneath.
+        _pkg_specific = ('pkgdesc', 'arch', 'url', 'license', 'groups', 'depends', 'optdepends', 'provides',
+                         'conflicts', 'replaces', 'backup', 'options', 'install', 'changelog')
+        _aur_results = []
+        _urls = []
+        _params = {'arg[]': []}
+        _tmp_params = {'arg[]': []}
+        self.cur.execute("SELECT * FROM packages WHERE active = '1'")
+        for row in self.cur.fetchall():
+            pkgbase = (row['pkgbase'] if row['pkgbase'] else row['pkgname'])
+            pkgnm = row['pkgname']
+            if pkgbase not in self.pkgs:
+                self.pkgs[pkgbase] = {'packages': {pkgnm: {}},
+                                      'meta': {}}
+            for k in dict(row):
+                if not k:
+                    continue
+                if k in _notrack:
+                    continue
+                if k in _pkg_specific:
+                    self.pkgs[pkgbase]['packages'][pkgnm][k] = row[k]
+                else:
+                    if k not in self.pkgs[pkgbase]['meta']:
+                        self.pkgs[pkgbase]['meta'][k] = row[k]
+            # TODO: change this?
+            pkgstr = urlparse.quote(pkgnm)  # We perform against a non-pkgbased name for the AUR search.
+            _tmp_params['arg[]'].append(pkgstr)
+            l = base_len + (len(urlparse.urlencode(_tmp_params, doseq = True)) + 1)
+            if l >= uri_limit:
+                # We need to split into multiple URIs based on URI size because of:
+                # https://wiki.archlinux.org/index.php/Aurweb_RPC_interface#Limitations
+                _urls.append('&'.join((aur_base, urlparse.urlencode(_params, doseq = True))))
+                _params = {'arg[]': []}
+                _tmp_params = {'arg[]': []}
+            _params['arg[]'].append(pkgstr)
+        _urls.append('&'.join((aur_base, urlparse.urlencode(_params, doseq = True))))
+        for url in _urls:
+            with reqs.urlopen(url) as u:
+                _aur_results.extend(json.loads(u.read().decode('utf-8'))['results'])
+        for pkg in _aur_results:
+            pkg = {k.lower(): v for (k, v) in pkg.items()}
+            pkgnm = pkg['name']
+            pkgbase = pkg['packagebase']
+            for (k, v) in pkg.items():
+                if k in _notrack:
+                    continue
+                if k in _attr_map:
+                    k = _attr_map[k]
+                if k in _pkg_specific:
+                    self.pkgs[pkgbase]['packages'][pkgnm][k] = v
+                else:
+                    self.pkgs[pkgbase]['meta'][k] = v
+            self.pkgs[pkgbase]['meta']['snapshot'] = 'https://aur.archlinux.org{0}'.format(pkg['urlpath'])
+            self.pkgs[pkgbase]['meta']['filename'] = os.path.basename(pkg['urlpath'])
+            self.pkgs[pkgbase]['meta']['build'] = self._chkver(pkgbase)
+        return()
+
+    def _drop_privs(self):
+        # First get the list of groups to assign.
+        # This *should* generate a list *exactly* like as if that user ran os.getgroups(),
+        # with the addition of self.cfg['build_user']['gid'] (if it isn't included already).
+        newgroups = list(sorted([g.gr_gid
+                                 for g in grp.getgrall()
+                                 if pwd.getpwuid(self.cfg['build_user']['uid'])
+                                 in g.gr_mem]))
+        if self.cfg['build_user']['gid'] not in newgroups:
+            newgroups.append(self.cfg['build_user']['gid'])
+            newgroups.sort()
+        # This is the user's "primary group"
+        user_gid = pwd.getpwuid(self.cfg['build_user']['uid']).pw_gid
+        if user_gid not in newgroups:
+            newgroups.append(user_gid)
+        os.setgroups(newgroups)
+        # If we used os.setgid and os.setuid, we would PERMANENTLY/IRREVOCABLY drop privs.
+        # Being that that doesn't suit the meta of the rest of the script (chmodding, etc.) - probably not a good idea.
+        os.setresgid(self.cfg['build_user']['gid'], self.cfg['build_user']['gid'], -1)
+        os.setresuid(self.cfg['build_user']['uid'], self.cfg['build_user']['uid'], -1)
+        # Default on most linux systems. reasonable enough for building? (equal to chmod 755/644)
+        os.umask(0o0022)
+        # TODO: we need a full env construction here, I think, as well. PATH, HOME, GNUPGHOME at the very least?
+        return()
+
+    def _restore_privs(self):
+        os.setresuid(self.cfg['orig_user']['uid'], self.cfg['orig_user']['uid'], self.cfg['orig_user']['uid'])
+        os.setresgid(self.cfg['orig_user']['gid'], self.cfg['orig_user']['gid'], self.cfg['orig_user']['gid'])
+        os.setgroups(self.cfg['orig_user']['groups'])
+        os.umask(self.cfg['orig_user']['umask'])
+        # TODO: if we change the env, we need to change it back here. I capture it in self.cfg['orig_user']['env'].
+        return()
+
+    def getPkg(self):
+        self._drop_privs()
+        for pkgbase in self.pkgs:
+            if not self.pkgs[pkgbase]['meta']['build']:
+                continue
+            _pkgre = re.compile('^(/?.*/)*({0})/?'.format(pkgbase))
+            builddir = os.path.join(self.cfg['cache'], pkgbase)
+            try:
+                shutil.rmtree(builddir)
+            except FileNotFoundError:
+                # We *could* use ignore_errors or onerrors params, but we only want FileNotFoundError.
+                pass
+            os.makedirs(builddir, mode = self.cfg['chmod']['dirs'], exist_ok = True)
+            tarball = os.path.join(builddir, self.pkgs[pkgbase]['meta']['filename'])
+            with reqs.urlopen(self.pkgs[pkgbase]['meta']['snapshot']) as url:
+                # We have to write out to disk first because the tarfile module HATES trying to perform seeks on
+                # a tarfile stream. It HATES it.
+                with open(tarball, 'wb') as f:
+                    f.write(url.read())
+            tarnames = {}
+            with tarfile.open(tarball, mode = 'r:*') as tar:
+                for i in tar.getmembers():
+                    if any((i.isdir(), i.ischr(), i.isblk(), i.isfifo(), i.isdev())):
+                        continue
+                    if i.name.endswith('.gitignore'):
+                        continue
+                    # We want to strip leading dirs out.
+                    tarnames[i.name] = _pkgre.sub('', i.name)
+                    # Small bugfix.
+                    if tarnames[i.name] == '':
+                        tarnames[i.name] = os.path.basename(i.name)
+                    tarnames[i.name] = os.path.join(builddir, tarnames[i.name])
+                for i in tar.getmembers():
+                    if i.name in tarnames:
+                        # GOLLY I WISH TARFILE WOULD LET US JUST CHANGE THE ARCNAME DURING EXTRACTION ON THE FLY.
+                        with open(tarnames[i.name], 'wb') as f:
+                            f.write(tar.extractfile(i.name).read())
+            # No longer needed, so clean it up behind us.
+            os.remove(tarball)
+        self._restore_privs()
+        return()
+
+    def buildPkg(self):
+        self._drop_privs()
+        for pkgbase in self.pkgs:
+            if not self.pkgs[pkgbase]['meta']['build']:
+                continue
+            builddir = os.path.join(self.cfg['cache'], pkgbase)
+            os.chdir(builddir)
+            # subprocess.run(['makepkg'])  # TODO: figure out gpg sig checking?
+            subprocess.run(['makepkg', '--clean', '--force', '--skippgpcheck'])
+        self._restore_privs()
+        for pkgbase in self.pkgs:
+            if not self.pkgs[pkgbase]['meta']['build']:
+                continue
+            builddir = os.path.join(self.cfg['cache'], pkgbase)
+            # The i686 isn't even supported anymore, but let's keep this friendly for Archlinux32 folks.
+            _pkgre = re.compile(('^({0})-{1}-'
+                                     '(x86_64|i686|any)'
+                                     '\.pkg\.tar\.xz$').format('|'.join(self.pkgs[pkgbase]['packages'].keys()),
+                                                               self.pkgs[pkgbase]['meta']['new_ver']))
+            fname = None
+            # PROBABLY in the first root dir, and could be done with fnmatch, but...
+            for root, dirs, files in os.walk(builddir):
+                for f in files:
+                    if _pkgre.search(f):
+                        fname = os.path.join(root, f)
+                        break
+            if not fname:
+                raise RuntimeError('Could not find proper package build filename for {0}'.format(pkgbase))
+            destfile = os.path.join(self.cfg['dest'], os.path.basename(fname))
+            os.rename(fname, destfile)
+            # TODO: HERE IS WHERE WE SIGN THE PACKAGE?
+            # We also need to update the package info in the DB.
+            for p in self.pkgs[pkgbase]['packages']:
+                self.cur.execute("UPDATE packages SET pkgver = ? WHERE pkgname = ?",
+                                 (self.pkgs[pkgbase]['meta']['new_ver'], p))
+            self.cfg['pkgpaths'].append(destfile)
+            # No longer needed, so we can clear out the build directory.
+            shutil.rmtree(builddir)
+        os.chdir(self.cfg['dest'])
+        dbfile = os.path.join(self.cfg['dest'], 'autopkg.db.tar.gz')  # TODO: Custom repo name?
+        cmd = ['repo-add', '--nocolor', '--delta', dbfile]  # -s/--sign?
+        cmd.extend(self.cfg['pkgpaths'])
+        subprocess.run(cmd)
+        for root, dirs, files in os.walk(self.cfg['dest']):
+            for f in files:
+                fpath = os.path.join(root, f)
+                os.chmod(fpath, self.cfg['chmod']['files'])
+                os.chown(fpath, self.cfg['chown']['uid'], self.cfg['chown']['gid'])
+            for d in dirs:
+                dpath = os.path.join(root, d)
+                os.chmod(dpath, self.cfg['chmod']['dirs'])
+                os.chown(dpath, self.cfg['chown']['uid'], self.cfg['chown']['gid'])
+        return()
+
+    def close(self):
+        if self.cur:
+            self.cur.close()
+        if self.conn:
+            self.conn.close()
+        return()
+
+def main():
+    pm = PkgMake()
+    pm.main()
+
+if __name__ == '__main__':
+    main()

arch/autopkg/setup.py

@@ -0,0 +1,127 @@
+#!/usr/bin/env python
+
+import base64
+import copy
+import gpg
+import grp
+import json
+import lzma
+import os
+import pwd
+import re
+from socket import gethostname
+import sqlite3
+
+# NOTE: The gpg homedir should be owned by the user *running autopkg*.
+# Likely priv-dropping will only work for root.
+#
+
+dirs = ('cache', 'dest', 'gpg_homedir')
+u_g_pairs = ('chown', 'build_user')
+json_vals = ('chmod', )
+
+blank_db = """
+/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4H//AxNdACmURZ1gyBn4JmSIjib+MZX9x4eABpe77H+o
+CX2bysoKzO/OaDh2QGbNjiU75tmhPrWMvTFue4XOq+6NPls33xRRL8eZoITBdAaLqbwYY2XW/V/X
+Gx8vpjcBnpACjVno40FoJ1qWxJlBZ0PI/8gMoBr3Sgdqnf+Bqi+E6dOl66ktJMRr3bdZ5C9vOXAf
+42BtRfwJlwN8NItaWtfRYVfXl+40D05dugcxDLY/3uUe9MSgt46Z9+Q9tGjjrUA8kb5K2fqWSlQ2
+6KyF3KV1zsJSDLuaRkP42JNsBTgg6mU5rEk/3egdJiLn+7AupvWQ3YlKkeALZvgEKy75wdObf6QI
+jY4qjXjxOTwOG4oou7lNZ3fPI5qLCQL48M8ZbOQoTAQCuArdYqJmBwT2rF86SdQRP4EY6TlExa4o
++E+v26hKhYXO7o188jlmGFbuzqtoyMB1y3UG+Hi2SjPDilD5o6f9fEjiHZm2FY6rkPb9Km4UFlH1
+d2A4Wt4iGlciZBs0lFRPKkgHR4s7KHTMKuZyC08qE1B7FwvyBTBBYveA2UoZlKY7d22IbiiSQ3tP
+JKhj8nf8EWcgHPt46Juo80l7vqqn6AviY7b1JZXICdiJMbuWJEyzTLWuk4qlUBfimP7k9IjhDFpJ
+gEXdNgrnx/wr5CIbr1T5lI9vZz35EacgNA2bGxLA8VI0W9eYDts3BSfhiJOHWwLQPiNzJwd4aeM1
+IhqgTEpk+BD0nIgSB3AAB+NfJJavoQjpv0QBA6dH52utA5Nw5L//Ufw/YKaA7ui8YQyDJ7y2n9L3
+ugn6VJFFrYSgIe1oRkJBGRGuBgGNTS3aJmdFqEz1vjZBMkFdF+rryXzub4dst2Qh01E6/elowIUh
+2whMRVDO28QjyS9tLtLLzfTmBk2NSxs4+znE0ePKKw3n/p6YlbPRAw24QR8MTCOpQ2lH1UZNWBM2
+epxfmWtgO5b/wGYopRDEvDDdbPAq6+4zxTOT5RmdWZyc46gdizf9+dQW3wZ9iBDjh4MtuYPvLlqr
+0GRmsyrxgFxkwvVoXASNndS0NPcAADkAhYCxn+W2AAGvBoCAAgB/TQWascRn+wIAAAAABFla
+"""
+
+def firstrun(dbfile):
+    dbdata = lzma.decompress(base64.b64decode(blank_db))
+    with open(dbfile, 'wb') as f:
+        f.write(dbdata)
+    return()
+
+def main(connection, cursor):
+    cfg = {'orig_cwd': os.getcwd(),
+           'pkgpaths': []}
+    cursor.execute("SELECT directive, value FROM config")
+    for r in cursor.fetchall():
+        cfg[r['directive']] = r['value'].strip()
+    for k in cfg:
+        for x in (True, False, None):
+            if cfg[k] == str(x):
+                cfg[k] = x
+                break
+        if k in json_vals:
+            cfg[k] = json.loads(cfg[k])
+        if k == 'path':
+            paths = []
+            for i in cfg[k].split(':'):
+                p = os.path.abspath(os.path.expanduser(i))
+                paths.append(p)
+            cfg[k] = paths
+        if k in dirs:
+            if cfg[k]:
+                cfg[k] = os.path.abspath(os.path.expanduser(cfg[k]))
+                os.makedirs(cfg[k], exist_ok = True)
+        if k in u_g_pairs:
+            dflt = [pwd.getpwuid(os.geteuid()).pw_name, grp.getgrgid(os.getegid()).gr_name]
+            l = re.split(':|\.', cfg[k])
+            if len(l) == 1:
+                l.append(None)
+            for idx, i in enumerate(l[:]):
+                if i in ('', None):
+                    l[idx] = dflt[idx]
+            cfg[k] = {}
+            cfg[k]['uid'] = (int(l[0]) if l[0].isnumeric() else pwd.getpwnam(l[0]).pw_uid)
+            cfg[k]['gid'] = (int(l[1]) if l[1].isnumeric() else grp.getgrnam(l[1]).gr_gid)
+    cfg['orig_user'] = {'uid': os.geteuid(),
+                        'gid': os.getegid()}
+    # Ugh. https://orkus.wordpress.com/2011/04/17/python-getting-umask-without-change/
+    cfg['orig_user']['umask'] = os.umask(0)
+    os.umask(cfg['orig_user']['umask'])
+    cfg['orig_user']['groups'] = os.getgroups()
+    for i in cfg['chmod']:
+        cfg['chmod'][i] = int(cfg['chmod'][i], 8)
+    cfg['orig_user']['env'] = copy.deepcopy(dict(os.environ))
+    os.chown(cfg['cache'], uid = cfg['build_user']['uid'], gid = cfg['build_user']['gid'])
+    os.chown(cfg['dest'], uid = cfg['chown']['uid'], gid = cfg['chown']['gid'])
+    return(cfg)
+
+def GPG(cur, homedir = None, keyid = None):
+    g = gpg.Context(home_dir = homedir)
+    if not keyid:
+        # We don't have a key specified, so we need to generate one and update the config.
+        s = ('This signature and signing key were automatically generated using Autopkg from OpTools: '
+             'https://git.square-r00t.net/OpTools/')
+        g.sig_notation_add('automatically-generated@git.square-r00t.net', s, gpg.constants.sig.notation.HUMAN_READABLE)
+        userid = 'Autopkg Signing Key ({0}@{1})'.format(os.getenv('SUDO_USER', os.environ['USER']), gethostname())
+        params = {
+            #'algorithm': 'ed25519',
+            'algorithm': 'rsa4096',
+            'expires': False,
+            'expires_in': 0,
+            'sign': True,
+            'passphrase': None
+            }
+        keyid = g.create_key(userid, **params).fpr
+        # https://stackoverflow.com/a/50718957
+        q = {}
+        for col in ('keyid', 'homedir'):
+            if sqlite3.sqlite_version_info > (3, 24, 0):
+                q[col] = ("INSERT INTO config (directive, value) "
+                          "VALUES ('gpg_{0}', ?) "
+                          "ON CONFLICT (directive) "
+                          "DO UPDATE SET value = excluded.value").format(col)
+            else:
+                cur.execute("SELECT id FROM config WHERE directive = 'gpg_{0}'".format(col))
+                row = cur.fetchone()
+                if row:
+                    q[col] = ("UPDATE config SET value = ? WHERE id = '{0}'").format(row['id'])
+                else:
+                    q[col] = ("INSERT INTO config (directive, value) VALUES ('gpg_{0}', ?)").format(col)
+            cur.execute(q[col], (locals()[col], ))
+    return(keyid, g)

arch/buildup/pkgchk.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+
+import argparse
+import configparser
+import hashlib
+import os
+import re
+import shlex
+import subprocess
+import tarfile  # for verifying built PKGBUILDs. We just need to grab <tar>/.PKGINFO, and check: pkgver = <version>
+import tempfile
+from collections import OrderedDict
+from urllib.request import urlopen
+
+class color(object):
+    PURPLE = '\033[95m'
+    CYAN = '\033[96m'
+    DARKCYAN = '\033[36m'
+    BLUE = '\033[94m'
+    GREEN = '\033[92m'
+    YELLOW = '\033[93m'
+    RED = '\033[91m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'
+    END = '\033[0m'
+
+
+vcstypes = ('bzr', 'git', 'hg', 'svn')    
+
+class pkgChk(object):
+    def __init__(self, pkg):
+        # pkg should be a string of a PKGBUILD,
+        # not the path to a file.
+        self.pkg = pkg
+        # The below holds parsed data from the PKGBUILD.
+        self.pkgdata = {'pkgver': self.getLex('pkgver', 'var'),
+                        '_pkgver': self.getLex('_pkgver', 'var'),
+                        'pkgname': self.getLex('pkgname', 'var'),
+                        'sources': self.getLex('source', 'array')}
+
+    def getLex(self, attrib, attrtype):
+        # Parse the PKGBUILD and return actual values from it.
+        # attrtype should be "var" or "array".
+        # var returns a string and array returns a list.
+        # If the given attrib isn't in the pkgbuild, None is returned.
+        # The sources array is special, though - it returns a tuple of:
+        # (hashtype, dict) where dict is a mapping of:
+        # filename: hash
+        # filename2: hash2
+        # etc.
+        if attrtype not in ('var', 'array'):
+            raise ValueError('{0} is not a valid attribute type.'.format(attrib))
+        _sums = ('sha512', 'sha384', 'sha256', 'sha1', 'md5')  # in order of preference
+        _attrmap = {'var': 'echo ${{{0}}}'.format(attrib),
+                    'array': 'echo ${{{}[@]}}'.format(attrib)}
+        _tempfile = tempfile.mkstemp(text = True)
+        with open(_tempfile[1], 'w') as f:
+            f.write(self.pkg)
+        _cmd = ['/bin/bash',
+                '--restricted', '--noprofile',
+                '--init-file', _tempfile[1],
+                '-i', '-c', _attrmap[attrtype]]
+        with open(os.devnull, 'wb') as devnull:
+            _out = subprocess.run(_cmd, env = {'PATH': ''},
+                                  stdout = subprocess.PIPE,
+                                  stderr = devnull).stdout.decode('utf-8').strip()
+        if _out == '':
+            os.remove(_tempfile[1])
+            return(None)
+        if attrtype == 'var':
+            os.remove(_tempfile[1])
+            return(_out)
+        else:  # it's an array
+            if attrib == 'source':
+                _sources = {}
+                _source = shlex.split(_out)
+                _sumarr = [None] * len(_source)
+                for h in _sums:
+                    _cmd[-1] = 'echo ${{{0}[@]}}'.format(h + 'sums')
+                    with open(os.devnull, 'wb') as devnull:
+                        _out = subprocess.run(_cmd, env = {'PATH': ''},
+                                              stdout = subprocess.PIPE,
+                                              stderr = devnull).stdout.decode('utf-8').strip()
+                    if _out != '':
+                        os.remove(_tempfile[1])
+                        return(h, OrderedDict(zip(_source, shlex.split(_out))))
+                    else:
+                        continue
+                    # No match for checksums.
+                    os.remove(_tempfile[1])
+                    return(None, OrderedDict(zip(_source, shlex.split(_out))))
+            else:
+                os.remove(_tempfile[1])
+                return(shlex.split(_out))
+            return()
+        
+    def getURL(self, url):
+        with urlopen(url) as http:
+            code = http.getcode()
+        return(code)
+    
+    def chkVer(self):
+        _separators = []
+        # TODO: this is to explicitly prevent parsing
+        # VCS packages, so might need some re-tooling in the future.
+        if self.pkgdata['pkgname'].split('-')[-1] in vcstypes:
+            return(None)
+        # transform the current version into a list of various components.
+        if not self.pkgdata['pkgver']:
+            return(None)
+        if self.pkgdata['_pkgver']:
+            _cur_ver = self.pkgdata['_pkgver']
+        else:
+            _cur_ver = self.pkgdata['pkgver']
+        # This will catch like 90% of the software versions out there.
+        # Unfortunately, it won't catch all of them. I dunno how to
+        # handle that quite yet. TODO.
+        _split_ver = _cur_ver.split('.')
+        _idx = len(_split_ver) - 1
+        while _idx >= 0:
+            _url = re.sub('^[A-Za-z0-9]+::',
+                          '',
+                          list(self.pkgdata['sources'].keys())[0])
+            _code = self.getURL(_url)
+            _idx -= 1
+
+def parseArgs():
+    _ini = '~/.config/optools/buildup.ini'
+    _defini = os.path.abspath(os.path.expanduser(_ini))
+    args = argparse.ArgumentParser()
+    args.add_argument('-c', '--config',
+                      default = _defini,
+                      dest = 'config',
+                      help = ('The path to the config file. ' +
+                              'Default: {0}{1}{2}').format(color.BOLD,
+                                                           _defini,
+                                                           color.END))
+    args.add_argument('-R', '--no-recurse',
+                      action = 'store_false',
+                      dest = 'recurse',
+                      help = ('If specified, and the path provided is a directory, ' +
+                              'do NOT recurse into subdirectories.'))
+    args.add_argument('-p', '--path',
+                      metavar = 'path/to/dir/or/PKGBUILD',
+                      default = None,
+                      dest = 'pkgpath',
+                      help = ('The path to either a directory containing PKGBUILDs (recursion ' +
+                              'enabled - see {0}-R/--no-recurse{1}) ' +
+                              'or a single PKGBUILD. Use to override ' +
+                              'the config\'s PKG:paths.').format(color.BOLD, color.END)) 
+    return(args)
+
+def parsePkg(pkgbuildstr):
+    p = pkgChk(pkgbuildstr)
+    p.chkVer()
+    return()
+
+def iterDir(pkgpath, recursion = True):
+    filepaths = []
+    if os.path.isfile(pkgpath):
+        return([pkgpath])
+    if recursion:
+        for root, subdirs, files in os.walk(pkgpath):
+            for vcs in vcstypes:
+                if '.{0}'.format(vcs) in subdirs:
+                    subdirs.remove('.{0}'.format(vcs))
+            for f in files:
+                if 'PKGBUILD' in f:
+                    filepaths.append(os.path.join(root, f))
+    else:
+        for f in os.listdir(pkgpath):
+            if 'PKGBUILD' in f:
+                filepaths.append(f)
+    filepaths.sort()
+    return(filepaths)
+
+def parseCfg(cfgfile):
+    def getPath(p):
+        return(os.path.abspath(os.path.expanduser(p)))
+    _defcfg = '[PKG]\npaths = \ntestbuild = no\n[VCS]\n'
+    for vcs in vcstypes:
+        _defcfg += '{0} = no\n'.format(vcs)
+    _cfg = configparser.ConfigParser()
+    _cfg._interpolation = configparser.ExtendedInterpolation()
+    _cfg.read((_defcfg, cfgfile))
+    # We convert to a dict so we can do things like list comprehension.
+    cfg = {s:dict(_cfg.items(s)) for s in _cfg.sections()}
+    if 'paths' not in cfg['PKG'].keys():
+        raise ValueError('You must provide a valid configuration ' +
+                          'file with the PKG:paths setting specified and valid.')
+    cfg['PKG']['paths'] = sorted([getPath(p.strip()) for p in cfg['PKG']['paths'].split(',')],
+                                 reverse = True)
+    for p in cfg['PKG']['paths'][:]:
+        if not os.path.exists(p):
+            print('WARNING: {0} does not exist; skipping...'.format(p))
+            cfg['PKG']['paths'].remove(p)
+    # We also want to convert these to pythonic True/False
+    cfg['PKG']['testbuild'] = _cfg['PKG'].getboolean('testbuild')
+    for k in vcstypes:
+        cfg['VCS'][k] = _cfg['VCS'].getboolean(k)
+    return(cfg)
+
+if __name__ == '__main__':
+    args = vars(parseArgs().parse_args())
+    if not os.path.isfile(args['config']):
+        raise FileNotFoundError('{0} does not exist.'.format(cfg))
+    cfg = parseCfg(args['config'])
+    if args['pkgpath']:
+        args['pkgpath'] = os.path.abspath(os.path.expanduser(args['pkgpath']))
+        if os.path.isdir(args['pkgpath']):
+            iterDir(args['pkgpath'], recursion = args['recurse'])
+        elif os.path.isfile(args['pkgpath']):
+            parsePkg(args['pkgpath'])
+        else:
+            raise FileNotFoundError('{0} does not exist.'.format(args['pkgpath']))
+    else:
+        files = []
+        for p in cfg['PKG']['paths']:
+            files.extend(iterDir(p))
+        files.sort()
+        for p in files:
+            with open(p, 'r') as f:
+                parsePkg(f.read())

arch/buildup/sample.buildup.ini

@@ -0,0 +1,39 @@
+## This configuration file will allow you to perform more
+## fine-grained control of BuildUp.
+## It supports the syntax shortcuts found here:
+## https://docs.python.org/3/library/configparser.html#configparser.ExtendedInterpolation
+
+[PKG]
+# The path(s) to your PKGBUILD(s), or a directory/directories containing them.
+# If you have more than one, separate with a comma.
+paths = path/to/pkgbuilds,another/path/to/pkgbuilds
+
+# If 'yes', try building the package with the new version.
+# If 'no' (the default), don't try to build with the new version.
+# This can be a good way to test that you don't need to modify the PKGBUILD,
+# but can be error-prone (missing makedeps, etc.).
+testbuild = no
+
+[VCS]
+# Here you can enable or disable which VCS platforms you want to support.
+# Note that it will increase the time of your check, as it will
+# actually perform a checkout/clone/etc. of the source and check against
+# the version function inside the PKGBUILD.
+# It's also generally meaningless, as VCS PKGBUILDs are intended
+# to be dynamic. Nonetheless, the options are there.
+# Use 'yes' to enable, or 'no' to disable (the default).
+# Currently only the given types are supported (i.e. no CVS).
+
+# THESE ARE CURRENTLY NOT SUPPORTED.
+
+# Check revisions for -git PKGBUILDs
+git = no
+
+# Check revisions for -svn PKGBUILDs
+svn = no
+
+# Check revisions for -hg PKGBUILDs
+hg = no
+
+# Check revisions for -bzr PKGBUILDs
+bzr = no

arch/mirrorchk.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+
+import os
+import re
+import subprocess
+import tempfile
+from urllib.request import urlopen
+
+# The local list of mirrors
+mfile = '/etc/pacman.d/mirrorlist'
+# The URL for the list of mirros
+# TODO: customize with country in a config
+rlist = 'https://www.archlinux.org/mirrorlist/?country=US&protocol=http&protocol=https&ip_version=4&use_mirror_status=on'
+# If local_mirror is set to None, don't do any modifications.
+# If it's a dict in the format of:
+#   local_mirror = {'profile': 'PROFILE_NAME',
+#                   'url': 'http://host/arch/%os/$arch',
+#                   'state_file': '/var/lib/netctl/netctl.state'}
+# Then we will check 'state_file'. If its contents match 'profile',
+# then we will add 'url' to the *top* of mfile.
+# TODO: I need to move this to a config.
+local_mirror = {'profile': '<PROFILENAME>',
+                'url': 'http://<REPOBOX>/arch/$repo/os/$arch',
+                'state_file': '/var/lib/netctl/netctl.state'}
+
+def getList(url):
+    with urlopen(url) as http:
+        l = http.read().decode('utf-8')
+    return(l)
+
+def uncomment(url_list):
+    urls = []
+    if isinstance(url_list, str):
+        url_list = [u.strip() for u in url_list.splitlines()]
+    for u in url_list:
+        u = u.strip()
+        if u == '':
+            continue
+        urls.append(re.sub('^\s*#', '', u))
+    return(urls)
+
+def rankList(mfile):
+    c = ['rankmirrors',
+         '-n', '6',
+         mfile]
+    ranked_urls = subprocess.run(c, stdout = subprocess.PIPE)
+    url_list = ranked_urls.stdout.decode('utf-8').splitlines()
+    for u in url_list[:]:
+        if u.strip() == '':
+            url_list.remove(u)
+            continue
+        if re.match('^\s*(#.*)$', u, re.MULTILINE | re.DOTALL):
+            url_list.remove(u)
+    return(url_list)
+
+def localMirror(url_list):
+    # If checking the state_file doesn't work out, use netctl
+    # directly.
+    if not isinstance(local_mirror, dict):
+        return(url_list)
+    with open(local_mirror['state_file'], 'r') as f:
+        state = f.read().strip()
+    state = [s.strip() for s in state]
+    if local_mirror['profile'] in state:
+        url_list.insert(0, 'Server = {0}'.format(local_mirror['url']))
+    return(url_list)
+
+def writeList(mirrorfile, url_list):
+    with open(mirrorfile, 'w') as f:
+        f.write('{0}\n'.format('\n'.join(url_list)))
+    return()
+
+if __name__ == '__main__':
+    if os.geteuid() != 0:
+        exit('Must be run as root.')
+    urls = getList(rlist)
+    t = tempfile.mkstemp(text = True)
+    writeList(t[1], uncomment(urls))
+    ranked_mirrors = localMirror(rankList(t[1]))
+    writeList(mfile, ranked_mirrors)
+    os.remove(t[1])

arch/repo-maint.py

@@ -0,0 +1,288 @@
+#!/usr/bin/env python3
+
+import argparse
+import io
+import os
+import pprint
+import re
+import sys
+import tarfile
+
+
+# PREREQS:
+# Mostly stdlib.
+#
+# IF:
+# 1.) You want to sign or verify packages (-s/--sign and -v/--verify, respectively),
+# 2.) You want to work with delta updates,
+# THEN:
+# 1.) You need to install the python GnuPG GPGME bindings (the "gpg" module; NOT the "gpgme" module). They're
+#     distributed with the GPG source. They're also in PyPI (https://pypi.org/project/gpg/).
+# 2.) You need to install the xdelta3 module (https://pypi.org/project/xdelta3/).
+
+_delta_re = re.compile('(.*)-*-*_to*')
+
+
+class RepoMaint(object):
+    def __init__(self, **kwargs):
+        # https://stackoverflow.com/a/2912884/733214
+        user_params = kwargs
+        # Define a set of defaults to update with kwargs since we
+        # aren't explicitly defining params.
+        self.args = {'color': True,
+                     'db': './repo.db.tar.xz',
+                     'key': None,
+                     'pkgs': [],
+                     'quiet': False,
+                     'sign': False,
+                     'verify': False}
+        self.args.update(user_params)
+        self.db_exts = {'db.tar': False,  # No compression
+                        'db.tar.xz': 'xz',
+                        'db.tar.gz': 'gz',
+                        'db.tar.bz2': 'bz2',
+                        # We explicitly check False vs. None.
+                        # For None, we do a custom check and wrap it.
+                        # In .Z's case, we use the lzw module. It's the only non-stdlib compression
+                        # that Arch Linux repo DB files support.
+                        'db.tar.Z': None}
+        self.args['db'] = os.path.abspath(os.path.expanduser(self.args['db']))
+        self.db = None
+        _is_valid_repo_db = False
+        if not _is_valid_repo_db:
+            raise ValueError(('Repo DB {0} is not a valid DB type. '
+                              'Must be one of {1}.').format(self.args['db'],
+                                                            ', '.join(['*.{0}'.format(i) for i in self.db_exts])))
+        self.repo_dir = os.path.dirname(self.args['db'])
+        self.lockfile = '{0}.lck'.format(self.args['db'])
+        os.makedirs(self.repo_dir, exist_ok = True)
+        self.gpg = None
+        self.sigkey = None
+        if self.args['sign'] or self.args['verify']:
+            # Set up GPG handler.
+            self._initGPG()
+        self._importDB()
+
+    def _initGPG(self):
+        import gpg
+        self.gpg = gpg.Context()
+        if self.args['sign']:
+            _seckeys = [k for k in self.gpg.keylist(secret = True) if k.can_sign]
+            if self.args['key']:
+                for k in _seckeys:
+                    if self.sigkey:
+                        break
+                    for s in k.subkeys:
+                        if self.sigkey:
+                            break
+                        if s.can_sign:
+                            if self.args['key'].lower() in (s.keyid.lower(),
+                                                            s.fpr.lower()):
+                                self.sigkey = k
+                                self.gpg.signers = [k]
+            else:
+                # Grab the first key that can sign.
+                if _seckeys:
+                    self.sigkey = _seckeys[0]
+                    self.gpg.signers = [_seckeys[0]]
+                if not self.args['quiet']:
+                    print('Key ID not specified; using {0} as the default'.format(self.sigkey.fpr))
+            if not self.sigkey:
+                raise RuntimeError('Private key ID not found, cannot sign, or no secret keys exist.')
+        # TODO: confirm verifying works without a key
+        return()
+
+    def _LZWcompress(self, data):
+        # Based largely on:
+        # https://github.com/HugoPouliquen/lzw-tools/blob/master/utils/compression.py
+        data_arr = []
+        rawdata = io.BytesIO(data)
+        for i in range(int(len(data) / 2)):
+            data_arr.insert(i, rawdata.read(2))
+        w = bytes()
+        b_size = 256
+        b = []
+        compressed = io.BytesIO()
+        for c in data_arr:
+            c = c.to_bytes(2, 'big')
+            wc = w + c
+            if wc in b:
+                w = wc
+            else:
+                b.insert(b_size, wc)
+                compressed.write(b.index(wc).to_bytes(2, 'big'))
+                b_size += 1
+                w = c
+        return(compressed.getvalue())
+
+    def _LZWdecompress(self, data):
+        # Based largely on:
+        # https://github.com/HugoPouliquen/lzw-tools/blob/master/utils/decompression.py
+        b_size = 256
+        b = []
+        out = io.BytesIO()
+        for i in range(b_size):
+            b.insert(i, i.to_bytes(2, 'big'))
+        w = data.pop(0)
+        out.write(w)
+        i = 0
+        for byte in data:
+            x = int.from_bytes(byte, byteorder = 'big')
+            if x < b_size:
+                entry = b[x]
+            elif x == b_size:
+                entry = w + w
+            else:
+                raise ValueError('Bad uncompressed value for "{0}"'.format(byte))
+            for y in entry:
+                if i % 2 == 1:
+                    out.write(y.to_bytes(1, byteorder = 'big'))
+                i += 1
+            b.insert(b_size, w + x)
+            b_size += 1
+            w = entry
+        return(out.getvalue())
+
+    def _importDB(self):
+        # Get the compression type.
+        for ct in self.db_exts:
+            if self.args['db'].lower().endswith(ct):
+                if self.db_exts[ct] == False:
+                    if ct.endswith('.Z'):  # Currently the only custom one.
+                        pass
+
+
+    def add(self):
+        # Fresh pkg set (in case the instance was re-used).
+        self.pkgs = {}
+        # First handle any wildcard
+        for p in self.args['pkgs'][:]:
+            if p.strip() == '*':
+                for root, dirs, files in os.walk(self.repo_dir):
+                    for f in files:
+                        abspath = os.path.join(root, f)
+                        if f.endswith('.pkg.tar.xz'):  # Recommended not to be changed per makepkg.conf
+                            if abspath not in self.args['pkgs']:
+                                self.args['pkgs'].append(abspath)
+                        if self.args['delta']:
+                            if f.endswith('.delta'):
+                                if abspath not in self.args['pkgs']:
+                                    self.args['pkgs'].append(abspath)
+                self.args['pkgs'].remove(p)
+        # Then de-dupe and convert to full path.
+        self.args['pkgs'] = sorted(list(set([os.path.abspath(os.path.expanduser(d)) for d in self.args['pkgs']])))
+        for p in self.args['pkgs']:
+            pkgfnm = os.path.basename(p)
+            if p.endswith('.delta'):
+                pkgnm = _delta_re.sub('\g<1>', os.path.basename(pkgfnm))
+
+        return()
+
+    def remove(self):
+        for p in self.args['pkgs']:
+            pass
+        return()
+
+
+def hatch():
+    import base64
+    import lzma
+    import random
+    h = ((
+          '/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AB6AEtdABBok+MQCtEh'
+          'BisubEtc2ebacaLGrSRAMmHrcwUr39J24q4iODdNz7wfQl9e6I3C'
+          'ooyuOkptNISdo50CRdknGAU4JBBh+IQTkHwiAAAABW1d7drLmkUA'
+          'AWd7/+DtzR+2830BAAAAAARZWg=='
+         ).encode('utf-8'),
+         (
+          '/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AHEALtdABBpE/AVEKFC'
+          'fdT16ly2cCwT/MnXTY2D4r8nWgH6mLetLPn17nza3ZK+tSFU7d5j'
+          'my91M8fvPGu9Tf0NYkWlRU7vJM8r2V3kK/Gs6/GS7tq2qIum/C/X'
+          'sOnYUewVB2yMvlACqwp3gWJlmXSfwcpGiU662EmATS8kUgF+OdP+'
+          'EATXhM/1bAn07wJbVWPoAL2SBmJBo2zL1tXQklbQu1J20eWfd1bD'
+          'cgSBGqcU1/CdHnW6lcb6BmWKTg0p9IAAAEoEyN1gLkAMAAHXAcUD'
+          'AACXcduyscRn+wIAAAAABFla'
+         ).encode('utf-8'))
+    h = lzma.decompress(base64.b64decode(h[random.randint(0, 1)]))
+    return(h.decode('utf-8'))
+
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = ('Python implementation of repo-add/repo-remove.'),
+                                   epilog = ('See https://wiki.archlinux.org/index.php/Pacman/'
+                                             'Tips_and_tricks#Custom_local_repository for more information.\n'
+                                             'Each operation has sub-help (e.g. "... add -h")'),
+                                   formatter_class = argparse.RawDescriptionHelpFormatter)
+    operargs = args.add_subparsers(dest = 'oper',
+                                   help = ('Operation to perform'))
+    commonargs = argparse.ArgumentParser(add_help = False)
+    commonargs.add_argument('db',
+                            metavar = '</path/to/repository/repo.db.tar.xz>',
+                            help = ('The path to the repository DB (required)'))
+    commonargs.add_argument('pkgs',
+                            nargs = '+',
+                            metavar = '<package|delta>',
+                            help = ('Package filepath (for adding)/name (for removing) or delta; '
+                                    'can be specified multiple times (at least 1 required)'))
+    commonargs.add_argument('--nocolor',
+                            dest = 'color',
+                            action = 'store_false',
+                            help = ('If specified, turn off color in output (currently does nothing; '
+                                    'output is currently not colorized)'))
+    commonargs.add_argument('-q', '--quiet',
+                            dest = 'quiet',
+                            action = 'store_true',
+                            help = ('Minimize output'))
+    commonargs.add_argument('-s', '--sign',
+                            dest = 'sign',
+                            action = 'store_true',
+                            help = ('If specified, sign database with GnuPG after update'))
+    commonargs.add_argument('-k', '--key',
+                            metavar = 'KEY_ID',
+                            nargs = 1,
+                            help = ('Use the specified GPG key to sign the database '
+                                    '(only used if -s/--sign is active)'))
+    commonargs.add_argument('-v', '--verify',
+                            dest = 'verify',
+                            action = 'store_true',
+                            help = ('If specified, verify the database\'s signature before update'))
+    addargs = operargs.add_parser('add',
+                                  parents = [commonargs],
+                                  help = ('Add package(s) to a repository'))
+    remargs = operargs.add_parser('remove',
+                                  parents = [commonargs],
+                                  help = ('Remove package(s) from a repository'))
+    addargs.add_argument('-d', '--delta',
+                         dest = 'delta',
+                         action = 'store_true',
+                         help = ('If specified, generate and add package deltas for the update'))
+    addargs.add_argument('-n', '--new',
+                         dest = 'new_only',
+                         action = 'store_true',
+                         help = ('If specified, only add packages that are not already in the database'))
+    addargs.add_argument('-R', '--remove',
+                         dest = 'remove_old',
+                         action = 'store_true',
+                         help = ('If specified, remove old packages from disk after updating the database'))
+    # Removal args have no add'l arguments, just the common ones.
+    return(args)
+
+def main():
+    if (len(sys.argv) == 2) and (sys.argv[1] == 'elephant'):
+        print(hatch())
+        return()
+    else:
+        rawargs = parseArgs()
+        args = rawargs.parse_args()
+        if not args.oper:
+            rawargs.print_help()
+            exit()
+        rm = RepoMaint(**vars(args))
+        if args.oper == 'add':
+            rm.add()
+        elif args.oper == 'remove':
+            rm.remove()
+    return()
+
+if __name__ == '__main__':
+    main()

arch/repoclone.py

@@ -1,151 +0,0 @@
-#!/usr/bin/env python3
-
-import argparse
-import configparser
-import datetime
-import os
-import pprint
-import subprocess
-import sys
-
-cfgfile = os.path.join(os.environ['HOME'], '.arch.repoclone.ini')
-
-# Rsync options
-opts = [
-        '--recursive',  # recurse into directories
-        '--times',  # preserve modification times
-        '--links',  # copy symlinks as symlinks
-        '--hard-links',  # preserve hard links
-        '--quiet',  # suppress non-error messages
-        '--delete-after',  # receiver deletes after transfer, not during
-        '--delay-updates',  # put all updated files into place at end
-        '--copy-links',  # transform symlink into referent file/dir
-        '--safe-links',  # ignore symlinks that point outside the tree
-        #'--max-delete',  # don't delete more than NUM files
-        '--delete-excluded',  # also delete excluded files from dest dirs
-        '--exclude=.*'  # exclude files matching PATTERN
-       ]
-
-def sync(args):
-    with open(os.devnull, 'w') as devnull:
-        mntchk = subprocess.run(['findmnt', args['mount']], stdout = devnull, stderr = devnull)
-    if mntchk.returncode != 0:
-        exit('!! BAILING OUT; {0} isn\'t mounted !!'.format(args['mount']))
-    if args['bwlimit'] >= 1:
-        opts.insert(10, '--bwlimit=' + str(args['bwlimit']))  # limit socket I/O bandwidth
-    for k in ('destination', 'logfile', 'lockfile'):
-        os.makedirs(os.path.dirname(args[k]), exist_ok = True)
-    paths = os.environ['PATH'].split(':')
-    rsync = '/usr/bin/rsync'  # set the default
-    for p in paths:
-        testpath = os.path.join(p, 'rsync')
-        if os.path.isfile(testpath):
-            rsync = testpath  # in case rsync isn't in /usr/bin/rsync
-            break
-    cmd = [rsync]  # the path to the binary
-    cmd.extend(opts)  # the arguments
-    # TODO: implement repos here?
-    cmd.append(os.path.join(args['mirror'], '.'))  # the path on the remote mirror
-    cmd.append(os.path.join(args['destination'], '.'))  # the local destination
-    if os.path.isfile(args['lockfile']):
-        with open(args['lockfile'], 'r') as f:
-            existingpid = f.read().strip()
-        if os.isatty(sys.stdin.fileno()):
-            # Running from shell
-            exit('!! A repo synchronization seems to already be running (PID: {0}). Quitting. !!'.format(existingpid))
-        else:
-            exit()  # we're running in cron, shut the hell up.
-    else:
-        with open(args['lockfile'], 'w') as f:
-            f.write(str(os.getpid()))
-    with open(args['logfile'], 'a') as log:
-        c = subprocess.run(cmd, stdout = log, stderr = subprocess.PIPE)
-        now = int(datetime.datetime.timestamp(datetime.datetime.utcnow()))
-        with open(os.path.join(args['destination'], 'lastsync'), 'w') as f:
-            f.write(str(now) + '\n')
-        os.remove(args['lockfile'])
-        # Only report errors at the end of the run if we aren't running in cron. Otherwise, log them.
-        errors = c.stderr.decode('utf-8').splitlines()
-        if os.isatty(sys.stdin.fileno()):
-            print('We encountered some errors:')
-            for e in errors:
-                if e.startswith('symlink has no referent: '):
-                    print('Broken upstream symlink: {0}'.format(e.split()[1].replace('"', '')))
-                else:
-                    print(e)
-        else:
-            with open(args['logfile'], 'a') as f:
-                for e in errors:
-                    f.write('{0}\n'.format(e))
-    return()
-
-def getDefaults():
-    # Hardcoded defaults
-    dflt = {'mirror': 'rsync://mirror.square-r00t.net/arch/',
-            'repos': 'core,extra,community,multilib,iso/latest',
-            'destination': '/srv/repos/arch',
-            'mount': '/',
-            'bwlimit': 0,
-            'lockfile': '/var/run/repo-sync.lck',
-            'logfile': '/var/log/repo/arch.log'}
-    realcfg = configparser.ConfigParser(defaults = dflt)
-    if not os.path.isfile(cfgfile):
-        with open(cfgfile, 'w') as f:
-            realcfg.write(f)
-    realcfg.read(cfgfile)
-    return(realcfg)
-
-def parseArgs():
-    cfg = getDefaults()
-    liveopts = cfg['DEFAULT']
-    args = argparse.ArgumentParser(description = 'Synchronization for a remote Arch repository to a local one.',
-                                   epilog = ('This program will write a default configuration file to {0} ' +
-                                            'if one is not found.'.format(cfgfile)))
-    args.add_argument('-m',
-                      '--mirror',
-                      dest = 'mirror',
-                      default = liveopts['mirror'],
-                      help = ('The upstream mirror to sync from, must be an rsync URI '+
-                              '(Default: {0}').format(liveopts['mirror']))
-# TODO: can we do this?
-#    args.add_argument('-r',
-#                      '--repos',
-#                      dest = 'repos',
-#                      default = liveopts['repos'],
-#                      help = ('The repositories to sync; must be a comma-separated list. ' +
-#                              '(Currently not used.) Default: {0}').format(','.join(liveopts['repos'])))
-    args.add_argument('-d',
-                      '--destination',
-                      dest = 'destination',
-                      default = liveopts['destination'],
-                      help = 'The destination directory to sync to. Default: {0}'.format(liveopts['destination']))
-    args.add_argument('-b',
-                      '--bwlimit',
-                      dest = 'bwlimit',
-                      default = liveopts['bwlimit'],
-                      type = int,
-                      help = 'The amount, in Kilobytes per second, to throttle the sync to. Default is to not throttle (0).')
-    args.add_argument('-l',
-                      '--log',
-                      dest = 'logfile',
-                      default = liveopts['logfile'],
-                      help = 'The path to the logfile. Default: {0}'.format(liveopts['logfile']))
-    args.add_argument('-L',
-                      '--lock',
-                      dest = 'lockfile',
-                      default = liveopts['lockfile'],
-                      help = 'The path to the lockfile. Default: {0}'.format(liveopts['lockfile']))
-    args.add_argument('-M',
-                      '--mount',
-                      dest = 'mount',
-                      default = liveopts['mount'],
-                      help = 'The mountpoint for your --destination. The script will exit if this point is not mounted. ' +
-                             'If you don\'t need mount checking, just use /. Default: {0}'.format(liveopts['mount']))
-    return(args)
-
-def main():
-    args = vars(parseArgs().parse_args())
-    sync(args)
-
-if __name__ == '__main__':
-    main()

centos/extract_files_package.py

@@ -0,0 +1,207 @@
+#!/usr/bin/env python
+
+# Supports CentOS 6.9 and up, untested on lower versions.
+# Lets you extract files for a given package name(s) without installing
+# any extra packages (such as yum-utils for repoquery).
+
+# NOTE: If you're on CentOS 6.x, since it uses such an ancient version of python you need to either install
+# python-argparse OR just resign to using it for all packages with none of the features.
+try:
+    import argparse
+    has_argparse = True
+except ImportError:
+    has_argparse = False
+import os
+import re
+import shutil
+import tempfile
+# For when CentOS/RHEL switch to python 3 by default (if EVER).
+import sys
+pyver = sys.version_info
+try:
+    import yum
+    # Needed for verbosity
+    from yum.logginglevels import __NO_LOGGING as yum_nolog
+    has_yum = True
+except ImportError:
+    has_yum = False
+    exit('This script only runs on the system-provided Python on RHEL/CentOS/other RPM-based distros.')
+try:
+    # pip install libarchive
+    # https://github.com/dsoprea/PyEasyArchive
+    import libarchive.public as lap
+    is_ctype = False
+except ImportError:
+    try:
+        # pip install libarchive
+        # https://github.com/Changaco/python-libarchive-c
+        import libarchive
+        if 'file_reader' in dir(libarchive):
+            is_legacy = False
+        else:
+            # https://code.google.com/archive/p/python-libarchive
+            is_legacy = True
+        is_ctype = True
+    except ImportError:
+        raise ImportError('Try yum -y install python-libarchive')
+
+
+class FileExtractor(object):
+    def __init__(self, dest_dir, paths, verbose = False, *args, **kwargs):
+        self.dest_dir = os.path.abspath(os.path.expanduser(dest_dir))
+        self.verbose = verbose  # TODO: print file name as extracting? Verbose as argument?
+        self.rpms = {}
+        if 'pkgs' in kwargs and kwargs['pkgs']:
+            self.pkgs = kwargs['pkgs']
+            self.yum_getFiles()
+        if 'rpm_files' in kwargs and kwargs['rpm_files']:
+            self.rpm_files = kwargs['rpm_files']
+            self.getFiles()
+        if '*' in paths:
+            self.paths = None
+        else:
+            self.paths = [re.sub('^', '.', os.path.abspath(i)) for i in paths]
+
+    def yum_getFiles(self):
+        import logging
+        yumloggers = ['yum.filelogging.RPMInstallCallback', 'yum.verbose.Repos', 'yum.verbose.plugin', 'yum.Depsolve',
+                      'yum.verbose', 'yum.plugin', 'yum.Repos', 'yum', 'yum.verbose.YumBase', 'yum.filelogging',
+                      'yum.verbose.YumPlugins', 'yum.RepoStorage', 'yum.YumBase', 'yum.filelogging.YumBase',
+                      'yum.verbose.Depsolve']
+        # This actually silences everything. Nice.
+        # https://stackoverflow.com/a/46716482/733214
+        if not self.verbose:
+            for loggerName in yumloggers:
+               logger = logging.getLogger(loggerName)
+               logger.setLevel(yum_nolog)
+        # http://yum.baseurl.org/api/yum/yum/__init__.html#yumbase
+        yb = yum.YumBase()
+        yb.conf.downloadonly = True
+        yb.conf.downloaddir = os.path.join(self.dest_dir, '.CACHE')
+        yb.conf.quiet = True
+        yb.conf.assumeyes = True
+        for pkg in self.pkgs:
+            try:
+                p = yb.reinstall(name = pkg)
+            except yum.Errors.ReinstallRemoveError:
+                p = yb.install(name = pkg)
+            p = p[0]
+            # I am... not 100% certain on this. Might be a better way?
+            fname = '{0}-{3}-{4}.{1}.rpm'.format(*p.pkgtup)
+            self.rpms[pkg] =  os.path.join(yb.conf.downloaddir, fname)
+        yb.buildTransaction()
+        try:
+            yb.processTransaction()
+        except SystemExit:
+            pass  # It keeps passing an exit because it's downloading only. Get it together, RH.
+        yb.closeRpmDB()
+        yb.close()
+        return()
+
+    def getFiles(self):
+        for rf in self.rpm_files:
+            # TODO: check if we have the rpm module and if so, rip pkg name from it? use that as key instead of rf?
+            self.rpms[os.path.basename(rf)] = os.path.abspath(os.path.expanduser(rf))
+        return()
+
+    def extractFiles(self):
+        # TODO: globbing or regex on self.paths?
+        # If we have yum, we can, TECHNICALLY, do this with:
+        # http://yum.baseurl.org/api/yum/rpmUtils/miscutils.html#rpmUtils.miscutils.rpm2cpio
+        # But nope. We can't selectively decompress members based on path with rpm2cpio-like funcs.
+        # We keep getting extraction artefacts, at least with legacy libarchive_c, so we use a hammer.
+        _curdir = os.getcwd()
+        _tempdir = tempfile.mkdtemp()
+        os.chdir(_tempdir)
+        for rpm_file in self.rpms:
+            rf = self.rpms[rpm_file]
+            if is_ctype:
+                if not is_legacy:
+                    # ctype - extracts to pwd
+                    with libarchive.file_reader(rf) as reader:
+                        for entry in reader:
+                            if self.paths and entry.path not in self.paths:
+                                continue
+                            if entry.isdir():
+                                continue
+                            fpath = os.path.join(self.dest_dir, rpm_file, entry.path)
+                            if not os.path.isdir(os.path.dirname(fpath)):
+                                os.makedirs(os.path.dirname(fpath))
+                            with open(fpath, 'wb') as f:
+                                for b in entry.get_blocks():
+                                    f.write(b)
+                else:
+                    with libarchive.Archive(rf) as reader:
+                        for entry in reader:
+                            if (self.paths and entry.pathname not in self.paths) or (entry.isdir()):
+                                continue
+                            fpath = os.path.join(self.dest_dir, rpm_file, entry.pathname)
+                            if not os.path.isdir(os.path.dirname(fpath)):
+                                os.makedirs(os.path.dirname(fpath))
+                            reader.readpath(fpath)
+            else:
+                # pyEasyArchive/"pypi/libarchive"
+                with lap.file_reader(rf) as reader:
+                    for entry in reader:
+                        if (self.paths and entry.pathname not in self.paths) or (entry.filetype.IFDIR):
+                            continue
+                        fpath = os.path.join(self.dest_dir, rpm_file, entry.pathname)
+                        if not os.path.isdir(os.path.dirname(fpath)):
+                            os.makedirs(os.path.dirname(fpath))
+                        with open(fpath, 'wb') as f:
+                            for b in entry.get_blocks():
+                                f.write(b)
+        os.chdir(_curdir)
+        shutil.rmtree(_tempdir)
+        return()
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = ('This script allows you to extract files for a given package '
+                                                  '{0}without installing any extra packages (such as yum-utils '
+                                                  'for repoquery). '
+                                                  'You must use at least one -r/--rpm{1}.').format(
+                                                                        ('name(s) ' if has_yum else ''),
+                                                                        (', -p/--package, or both' if has_yum else '')))
+    args.add_argument('-d', '--dest-dir',
+                      dest = 'dest_dir',
+                      default = '/var/tmp/rpm_extract',
+                      help = ('The destination for the extracted package file tree (in the format of '
+                              '<dest_dir>/<pkg_nm>/<tree>). '
+                              'Default: /var/tmp/rpm_extract'))
+    args.add_argument('-r', '--rpm',
+                      dest = 'rpm_files',
+                      metavar = 'PATH/TO/RPM',
+                      action = 'append',
+                      default = [],
+                      help = ('If specified, use this RPM file instead of the system\'s RPM database. Can be '
+                              'specified multiple times'))
+    if has_yum:
+        args.add_argument('-p', '--package',
+                          dest = 'pkgs',
+                          #nargs = 1,
+                          metavar = 'PKGNAME',
+                          action = 'append',
+                          default = [],
+                          help = ('If specified, restrict the list of packages to check against to only this package. '
+                                  'Can be specified multiple times. HIGHLY RECOMMENDED'))
+    args.add_argument('paths',
+                      nargs = '+',
+                      metavar = 'path/file/name.ext',
+                      help = ('The path(s) of files to extract. If \'*\' is used, extract all files'))
+    return(args)
+
+def main():
+    if has_argparse:
+        args = vars(parseArgs().parse_args())
+        args['rpm_files'] = [os.path.abspath(os.path.expanduser(i)) for i in args['rpm_files']]
+        if not any((args['rpm_files'], args['pkgs'])):
+            exit(('You have not specified any package files{0}.\n'
+                  'This is so dumb we are bailing out.\n').format((' or package names') if has_yum else ''))
+    else:
+        raise RuntimeError('Please yum -y install python-argparse')
+    fe = FileExtractor(**args)
+    fe.extractFiles()
+    return()
+
+if __name__ == '__main__':
+    main()

centos/find_changed_confs.py

@@ -0,0 +1,171 @@
+#!/usr/bin/env python
+
+# Supports CentOS 6.9 and up, untested on lower versions.
+# Definitely probably won't work on 5.x since they use MD5(?), and 6.5? and up
+# use SHA256.
+
+# TODO: add support for .rpm files (like list_files_package.py)
+
+import argparse
+import copy
+import datetime
+import hashlib
+import os
+import re
+from sys import version_info as py_ver
+try:
+    import rpm
+except ImportError:
+    exit('This script only runs on RHEL/CentOS/other RPM-based distros.')
+
+# Thanks, dude!
+# https://blog.fpmurphy.com/2011/08/programmatically-retrieve-rpm-package-details.html
+
+class PkgChk(object):
+    def __init__(self, dirpath, symlinks = True, pkgs = None):
+        self.path = dirpath
+        self.pkgs = pkgs
+        self.symlinks = symlinks
+        self.orig_pkgs = copy.deepcopy(pkgs)
+        self.pkgfilemap = {}
+        self.flatfiles = []
+        self.flst = {}
+        self.trns = rpm.TransactionSet()
+        self.getFiles()
+        self.getActualFiles()
+
+    def getFiles(self):
+        if not self.pkgs:
+            for p in self.trns.dbMatch():
+                self.pkgs.append(p['name'])
+        for p in self.pkgs:
+            for pkg in self.trns.dbMatch('name', p):
+                # Get the canonical package name
+                _pkgnm = pkg.sprintf('%{NAME}')
+                self.pkgfilemap[_pkgnm] = {}
+                # Get the list of file(s) and their MD5 hash(es)
+                for f in pkg.fiFromHeader():
+                    if not f[0].startswith(self.path):
+                        continue
+                    if f[12] == '0' * 64:
+                        _hash = None
+                    else:
+                        _hash = f[12]
+                    self.pkgfilemap[_pkgnm][f[0]] = {'hash': _hash,
+                                                     'date': f[3],
+                                                     'size': f[1]}
+                    self.flatfiles.append(f[0])
+        return()
+
+    def getActualFiles(self):
+        print('Getting a list of local files and their hashes.')
+        print('Please wait...\n')
+        for root, dirs, files in os.walk(self.path):
+            for f in files:
+                _fpath = os.path.join(root, f)
+                _stat = os.stat(_fpath)
+                if _fpath in self.flatfiles:
+                    _hash = hashlib.sha256()
+                    with open(_fpath, 'rb') as r:
+                        for chunk in iter(lambda: r.read(4096), b''):
+                            _hash.update(chunk)
+                    self.flst[_fpath] = {'hash': str(_hash.hexdigest()),
+                                         'date': int(_stat.st_mtime),
+                                         'size': _stat.st_size}
+                else:
+                    # It's not even in the package, so don't waste time
+                    # with generating hashes or anything else.
+                    self.flst[_fpath] = {'hash': None}
+        return()
+
+    def compareFiles(self):
+        for f in self.flst.keys():
+            if f not in self.flatfiles:
+                if not self.orig_pkgs:
+                    print(('{0} is not installed by any package.').format(f))
+                else:
+                    print(('{0} is not installed by package(s) ' +
+                           'specified.').format(f))
+            else:
+                for p in self.pkgs:
+                    if f not in self.pkgfilemap[p].keys():
+                        continue
+                    if (f in self.flst.keys() and
+                            (self.flst[f]['hash'] !=
+                             self.pkgfilemap[p][f]['hash'])):
+                        if not self.symlinks:
+                            if ((not self.pkgfilemap[p][f]['hash'])
+                                or re.search('^0+$',
+                                             self.pkgfilemap[p][f]['hash'])):
+                                continue
+                        r_time = datetime.datetime.fromtimestamp(
+                                                self.pkgfilemap[p][f]['date'])
+                        r_hash = self.pkgfilemap[p][f]['hash']
+                        r_size = self.pkgfilemap[p][f]['size']
+                        l_time = datetime.datetime.fromtimestamp(
+                                                        self.flst[f]['date'])
+                        l_hash = self.flst[f]['hash']
+                        l_size = self.flst[f]['size']
+                        r_str = ('\n{0} differs per {1}:\n' +
+                                 '\tRPM:\n' +
+                                 '\t\tSHA256: {2}\n' +
+                                 '\t\tBYTES:  {3}\n' +
+                                 '\t\tDATE:   {4}').format(f, p,
+                                                           r_hash,
+                                                           r_size,
+                                                           r_time)
+                        l_str = ('\tLOCAL:\n' +
+                                 '\t\tSHA256: {0}\n' +
+                                 '\t\tBYTES:  {1}\n' +
+                                 '\t\tDATE:   {2}').format(l_hash,
+                                                           l_size,
+                                                           l_time)
+                        print(r_str)
+                        print(l_str)
+        # Now we print missing files
+        for f in sorted(list(set(self.flatfiles))):
+            if not os.path.exists(f):
+                print('{0} was deleted from the filesystem.'.format(f))
+        return()
+
+def parseArgs():
+    def dirchk(path):
+        p = os.path.abspath(path)
+        if not os.path.isdir(p):
+            raise argparse.ArgumentTypeError(('{0} is not a valid ' +
+                                              'directory').format(path))
+        return(p)
+    args = argparse.ArgumentParser(description = ('Get a list of config ' +
+                                                  'files that have changed ' +
+                                                  'from the package\'s ' +
+                                                  'defaults'))
+    args.add_argument('-l', '--ignore-symlinks',
+                      dest = 'symlinks',
+                      action = 'store_false',
+                      help = ('If specified, don\'t track files that are ' +
+                              'symlinks in the RPM'))
+    args.add_argument('-p', '--package',
+                      dest = 'pkgs',
+                      #nargs = 1,
+                      metavar = 'PKGNAME',
+                      action = 'append',
+                      default = [],
+                      help = ('If specified, restrict the list of ' +
+                              'packages to check against to only this ' +
+                              'package. Can be specified multiple times. ' +
+                              'HIGHLY RECOMMENDED'))
+    args.add_argument('dirpath',
+                      type = dirchk,
+                      metavar = 'path/to/directory',
+                      help = ('The path to the directory containing the ' +
+                              'configuration files to check against (e.g. ' +
+                              '"/etc/ssh")'))
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    p = PkgChk(**args)
+    p.compareFiles()
+
+if __name__ == '__main__':
+    main()

centos/isomirror_sort.py

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+
+# requires python lxml module as well
+import os
+import socket
+import time
+from urllib.request import urlopen
+from urllib.parse import urlparse
+from bs4 import BeautifulSoup
+
+# The page that contains the list of (authoritative ISO) mirrors
+URL = 'http://isoredirect.centos.org/centos/7/isos/x86_64/'
+# The formatting on the page is pretty simple - no divs, etc. - so we need to
+# blacklist some links we pull in.
+blacklisted_link_URLs = ('http://bittorrent.com/',
+                         'http://wiki.centos.org/AdditionalResources/Repositories')
+
+mirrors = {}
+
+dflt_ports = {'https': 443,  # unlikely. "HTTPS is currently not used for mirrors." per https://wiki.centos.org/HowTos/CreatePublicMirrors
+              'http': 80,  # most likely.
+              'ftp': 21,
+              'rsync': 873}
+
+def getMirrors():
+    mirrors = []
+    with urlopen(URL) as u:
+        pg_src = u.read().decode('utf-8')
+    soup = BeautifulSoup(pg_src, 'lxml')
+    for tag in soup.find_all('br')[4].next_siblings:
+        if tag.name == 'a' and tag['href'] not in blacklisted_link_URLs:
+            mirrors.append(tag['href'].strip())
+    return(mirrors)
+
+def getHosts(mirror):
+    port = None
+    fqdn = None
+    login = ''
+    # "mirror" should be a base URI of the CentOS mirror path.
+    # mirrors.centos.org is pointless to use for this!
+    #url = os.path.join(mirror, 'sha256sum.txt.asc')
+    uri = urlparse(mirror)
+    spl_dom = uri.netloc.split(':')
+    if len(spl_dom) >= 2:  # more complex URI
+        if len(spl_dom) == 2:  # probably domain:port?
+            try:
+                port = int(spl_dom[-1:])
+            except ValueError:  # ooookay, so it's not domain:port, it's a user:pass@
+                if '@' in uri.netloc:
+                    auth = uri.netloc.split('@')
+                    fqdn = auth[1]
+                    login = auth[0] + '@'
+        elif len(spl_dom) > 2:  # even more complex URI, which ironically makes parsing easier
+            auth = uri.netloc.split('@')
+            fqdn = spl_dom[1].split('@')[1]
+            port = int(spl_dom[-1:])
+            login = auth[0] + '@'
+    # matches missing values and simple URI. like, 99%+ of mirror URIs being passed.
+    if not fqdn:
+        fqdn = uri.netloc
+    if not port:
+        port = dflt_ports[uri.scheme]
+    mirrors[fqdn] = {'proto': uri.scheme,
+                     'port': port,
+                     'path': uri.path,
+                     'auth': login}
+    return()
+
+def getSpeeds():
+    for fqdn in mirrors.keys():
+        start = time.time()
+        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        sock.connect((fqdn, mirrors[fqdn]['port']))
+        mirrors[fqdn]['time'] = time.time() - start
+        sock.close()
+    return()
+
+def main():
+    for m in getMirrors():
+        getHosts(m)
+    getSpeeds()
+    ranking = sorted(mirrors.keys(), key = lambda k: (mirrors[k]['time']))
+    for i in ranking:
+        str_port = ':' + str(mirrors[i]['port'])
+        if mirrors[i]['port'] in dflt_ports.values():
+            str_port = ''
+        print('{proto}://{auth}{0}{p}{path}'.format(i,
+                                                    **mirrors[i],
+                                                    p = str_port))
+
+if __name__ == '__main__':
+    main()

centos/list_files_package.py

@@ -0,0 +1,155 @@
+#!/usr/bin/env python
+
+# Supports CentOS 6.9 and up, untested on lower versions.
+# Lets you get a list of files for a given package name(s) without installing
+# any extra packages (such as yum-utils for repoquery).
+
+# NOTE: If you're on CentOS 6.x, since it uses such an ancient version of python you need to either install
+# python-argparse OR just resign to using it for all packages with none of the features.
+try:
+    import argparse
+    has_argparse = True
+except ImportError:
+    has_argparse = False
+import json
+import os
+import re
+# For when CentOS/RHEL switch to python 3 by default (if EVER).
+import sys
+pyver = sys.version_info
+try:
+    import rpm
+except ImportError:
+    exit('This script only runs on the system-provided Python on RHEL/CentOS/other RPM-based distros.')
+
+def all_pkgs():
+    # Gets a list of all packages.
+    pkgs = []
+    trns = rpm.TransactionSet()
+    for p in trns.dbMatch():
+        pkgs.append(p['name'])
+    pkgs = list(sorted(set(pkgs)))
+    return(pkgs)
+
+class FileGetter(object):
+    def __init__(self, symlinks = True, verbose = False, *args, **kwargs):
+        self.symlinks = symlinks
+        self.verbose = verbose
+        self.trns = rpm.TransactionSet()
+        self.files = {}
+        for p in kwargs['pkgs']:
+            if p not in self.files.keys():
+                self.getFiles(p)
+        if kwargs['rpm_files']:
+            self.getLocalFiles(kwargs['rpm_files'])
+
+    def getLocalFiles(self, rpm_files):
+        # Needed because the rpm module can't handle arbitrary rpm files??? If it can, someone let me know.
+        # According to http://rpm5.org/docs/api/classRpmhdr.html#_details I can.
+        import yum
+        for r in rpm_files:
+            pkg = yum.YumLocalPackage(ts = self.trns,
+                                      filename = r)
+            _pkgnm = pkg.hdr.sprintf('%{NAME}')
+            if _pkgnm in self.files:
+                continue
+            if self.verbose:
+                self.files[_pkgnm] = {}
+            else:
+                self.files[_pkgnm] = []
+            for f in pkg.hdr.fiFromHeader():
+                _symlink = (True if re.search('^0+$', f[12]) else False)
+                if self.verbose:
+                    if _symlink:
+                        if self.symlinks:
+                            self.files[_pkgnm][f[0]] = '(symbolic link or directory)'
+                        continue
+                    self.files[_pkgnm][f[0]] = f[12]
+                else:
+                    # Skip if it is a symlink but they aren't enabled
+                    if _symlink and not self.symlinks:
+                        continue
+                    else:
+                        self.files[_pkgnm].append(f[0])
+                    self.files[_pkgnm].sort()
+        return()
+
+    def getFiles(self, pkgnm):
+        for pkg in self.trns.dbMatch('name', pkgnm):
+            # The canonical package name
+            _pkgnm = pkg.sprintf('%{NAME}')
+            # Return just a list of files, or a dict of filepath:hash if verbose is enabled.
+            if self.verbose:
+                self.files[_pkgnm] = {}
+            else:
+                self.files[_pkgnm] = []
+            for f in pkg.fiFromHeader():
+                _symlink = (True if re.search('^0+$', f[12]) else False)
+                if self.verbose:
+                    if _symlink:
+                        if self.symlinks:
+                            self.files[_pkgnm][f[0]] = '(symbolic link)'
+                        continue
+                    self.files[_pkgnm][f[0]] = f[12]
+                else:
+                    # Skip if it is a symlink but they aren't enabled
+                    if _symlink and not self.symlinks:
+                        continue
+                    else:
+                        self.files[_pkgnm].append(f[0])
+                    self.files[_pkgnm].sort()
+        return()
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = ('This script allows you get a list of files for a given package '
+                                                  'name(s) without installing any extra packages (such as yum-utils '
+                                                  'for repoquery). It is highly recommended to use at least one '
+                                                  '-r/--rpm, -p/--package, or both.'))
+    args.add_argument('-l', '--ignore-symlinks',
+                      dest = 'symlinks',
+                      action = 'store_false',
+                      help = ('If specified, don\'t report files that are symlinks in the RPM'))
+    args.add_argument('-v', '--verbose',
+                      dest = 'verbose',
+                      action = 'store_true',
+                      help = ('If specified, include the hashes of the files'))
+    args.add_argument('-r', '--rpm',
+                      dest = 'rpm_files',
+                      metavar = 'PATH/TO/RPM',
+                      action = 'append',
+                      default = [],
+                      help = ('If specified, use this RPM file instead of the system\'s RPM database. Can be '
+                              'specified multiple times'))
+    args.add_argument('-p', '--package',
+                      dest = 'pkgs',
+                      #nargs = 1,
+                      metavar = 'PKGNAME',
+                      action = 'append',
+                      default = [],
+                      help = ('If specified, restrict the list of packages to check against to only this package. Can '
+                              'be specified multiple times. HIGHLY RECOMMENDED'))
+    return(args)
+
+def main():
+    if has_argparse:
+        args = vars(parseArgs().parse_args())
+        args['rpm_files'] = [os.path.abspath(os.path.expanduser(i)) for i in args['rpm_files']]
+        if not any((args['rpm_files'], args['pkgs'])):
+            prompt_str = ('You have not specified any package names.\nThis means we will get file lists for EVERY SINGLE '
+                          'installed package.\nThis is a LOT of output and can take a few moments.\nIf this was a mistake, '
+                          'you can hit ctrl-c now.\nOtherwise, hit the enter key to continue.\n')
+            sys.stderr.write(prompt_str)
+            if pyver.major >= 3:
+                input()
+            elif pyver.major == 2:
+                raw_input()
+            args['pkgs'] = all_pkgs()
+    else:
+        args = {'pkgs': all_pkgs(),
+                'rpm_files': []}
+    gf = FileGetter(**args)
+    print(json.dumps(gf.files, indent = 4))
+    return()
+
+if __name__ == '__main__':
+    main()

centos/list_pkgs.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python
+
+# Supports CentOS 6.9 and up, untested on lower versions.
+# Lets you dump a list of installed packages for backup purposes
+# Reference: https://blog.fpmurphy.com/2011/08/programmatically-retrieve-rpm-package-details.html
+
+import argparse
+import copy
+import datetime
+import io
+import re
+import sys
+try:
+    import yum
+except ImportError:
+    exit('This script only runs on RHEL/CentOS/other yum-based distros.')
+# Detect RH version.
+ver_re = re.compile('^(centos( linux)? release) ([0-9\.]+) .*$', re.IGNORECASE)
+# distro module isn't stdlib, and platform.linux_distribution() (AND platform.distro()) are both deprecated in 3.7.
+# So we get hacky.
+with open('/etc/redhat-release', 'r') as f:
+    ver = [int(i) for i in ver_re.sub('\g<3>', f.read().strip()).split('.')]
+import pprint
+
+repo_re = re.compile('^@')
+
+class PkgIndexer(object):
+    def __init__(self, **args):
+        self.pkgs = []
+        self.args = args
+        self.yb = yum.YumBase()
+        # Make the Yum API shut the heck up.
+        self.yb.preconf.debuglevel = 0
+        self.yb.preconf.errorlevel = 0
+        self._pkgs = self._pkglst()
+        self._build_pkginfo()
+        if self.args['report'] == 'csv':
+            self._gen_csv()
+        elif self.args['report'] == 'json':
+            self._gen_json()
+        elif self.args['report'] == 'xml':
+            self._gen_xml()
+
+    def _pkglst(self):
+        pkgs = []
+        # Get the list of packages
+        if self.args['reason'] != 'all':
+            for p in sorted(self.yb.rpmdb.returnPackages()):
+                if 'reason' not in p.yumdb_info:
+                    continue
+                reason = getattr(p.yumdb_info, 'reason')
+                if reason == self.args['reason']:
+                    pkgs.append(p)
+        else:
+            pkgs = sorted(self.yb.rpmdb.returnPackages())
+        return(pkgs)
+
+    def _build_pkginfo(self):
+        for p in self._pkgs:
+            _pkg = {'name': p.name,
+                    'desc': p.summary,
+                    'version': p.ver,
+                    'release': p.release,
+                    'arch': p.arch,
+                    'built': datetime.datetime.fromtimestamp(p.buildtime),
+                    'installed': datetime.datetime.fromtimestamp(p.installtime),
+                    'repo': repo_re.sub('', p.ui_from_repo),
+                    'sizerpm': p.packagesize,
+                    'sizedisk': p.installedsize}
+            self.pkgs.append(_pkg)
+
+    def _gen_csv(self):
+        if self.args['plain']:
+            _fields = ['name']
+        else:
+            _fields = ['name', 'version', 'release', 'arch', 'desc', 'built',
+                       'installed', 'repo', 'sizerpm', 'sizedisk']
+        import csv
+        if sys.hexversion >= 0x30000f0:
+            _buf = io.StringIO()
+        else:
+            _buf = io.BytesIO()
+        _csv = csv.writer(_buf, delimiter = self.args['sep_char'])
+        if self.args['header']:
+            if self.args['plain']:
+                _csv.writerow(['Name'])
+            else:
+                _csv.writerow(['Name', 'Version', 'Release', 'Architecture', 'Description', 'Build Time',
+                               'Install Time', 'Repository', 'Size (RPM)', 'Size (On-Disk)'])
+        _csv = csv.DictWriter(_buf, fieldnames = _fields, extrasaction = 'ignore', delimiter = self.args['sep_char'])
+        for p in self.pkgs:
+            _csv.writerow(p)
+        _buf.seek(0, 0)
+        self.report = _buf.read().replace('\r\n', '\n')
+        return()
+
+    def _gen_json(self):
+        import json
+        if self.args['plain']:
+            self.report = json.dumps([p['name'] for p in self.pkgs], indent = 4)
+        else:
+            self.report = json.dumps(self.pkgs, default = str, indent = 4)
+        return()
+
+    def _gen_xml(self):
+        from lxml import etree
+        _xml = etree.Element('packages')
+        for p in self.pkgs:
+            _attrib = copy.deepcopy(p)
+            for i in ('built', 'installed', 'sizerpm', 'sizedisk'):
+                _attrib[i] = str(_attrib[i])
+            if self.args['plain']:
+                _pkg = etree.Element('package', attrib = {'name': p['name']})
+            else:
+                _pkg = etree.Element('package', attrib = _attrib)
+            _xml.append(_pkg)
+            #del(_attrib['name'])  # I started to make it a more complex, nested structure... is that necessary?
+        if self.args['header']:
+            self.report = etree.tostring(_xml, pretty_print = True, xml_declaration = True, encoding = 'UTF-8')
+        else:
+            self.report = etree.tostring(_xml, pretty_print = True)
+        return()
+
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = ('This script lets you dump the list of installed packages'))
+    args.add_argument('-p', '--plain',
+                      dest = 'plain',
+                      action = 'store_true',
+                      help = 'If specified, only create a list of plain package names (i.e. don\'t include extra '
+                             'information)')
+    args.add_argument('-n', '--no-header',
+                      dest = 'header',
+                      action = 'store_false',
+                      help = 'If specified, do not print column headers/XML headers')
+    args.add_argument('-s', '--separator',
+                      dest = 'sep_char',
+                      default = ',',
+                      help = 'The separator used to split fields in the output (default: ,) (only used for CSV '
+                             'reports)')
+    rprt = args.add_mutually_exclusive_group()
+    rprt.add_argument('-c', '--csv',
+                      dest = 'report',
+                      default = 'csv',
+                      action = 'store_const',
+                      const = 'csv',
+                      help = 'Generate CSV output (this is the default). See -n/--no-header, -s/--separator')
+    rprt.add_argument('-x', '--xml',
+                      dest = 'report',
+                      default = 'csv',
+                      action = 'store_const',
+                      const = 'xml',
+                      help = 'Generate XML output (requires the LXML module: yum install python-lxml)')
+    rprt.add_argument('-j', '--json',
+                      dest = 'report',
+                      default = 'csv',
+                      action = 'store_const',
+                      const = 'json',
+                      help = 'Generate JSON output')
+    rsn = args.add_mutually_exclusive_group()
+    rsn.add_argument('-a', '--all',
+                     dest = 'reason',
+                     default = 'all',
+                     action = 'store_const',
+                     const = 'all',
+                     help = ('Parse/report all packages that are currently installed. '
+                             'Conflicts with -u/--user and -d/--dep. '
+                             'This is the default'))
+    rsn.add_argument('-u', '--user',
+                     dest = 'reason',
+                     default = 'all',
+                     action = 'store_const',
+                     const = 'user',
+                     help = ('Parse/report only packages which were explicitly installed. '
+                             'Conflicts with -a/--all and -d/--dep'))
+    rsn.add_argument('-d', '--dep',
+                     dest = 'reason',
+                     default = 'all',
+                     action = 'store_const',
+                     const = 'dep',
+                     help = ('Parse/report only packages which were installed to satisfy a dependency. '
+                             'Conflicts with -a/--all and -u/--user'))
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    p = PkgIndexer(**args)
+    print(p.report)
+    return()
+
+if __name__ == '__main__':
+    main()

git/remotehooks.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+
+import ast  # Needed for localhost cmd strings
+import json
+import os
+import re
+import sys
+modules = {}
+try:
+    import git
+    modules['git'] = True
+except ImportError:
+    import subprocess
+    modules['git'] = False
+try:
+    import paramiko
+    import socket
+    modules['ssh'] = True
+except ImportError:
+    modules['ssh'] = False
+
+
+
+repos = {}
+repos['bdisk'] = {'remotecmds': {'g.rainwreck.com': {'gitbot': {'cmds': ['git -C /var/lib/gitbot/clonerepos/BDisk pull',
+                                                                         'git -C /var/lib/gitbot/clonerepos/BDisk pull --tags',
+                                                                         'asciidoctor /var/lib/gitbot/clonerepos/BDisk/docs/manual/HEAD.adoc -o /srv/http/bdisk/index.html']}}}}
+repos['test'] = {'remotecmds': {'g.rainwreck.com': {'gitbot': {'cmds': ['echo $USER']}}}}
+repos['games-site'] = {'remotecmds': {'games.square-r00t.net':
+                                    {'gitbot':
+                                        {'cmds': ['cd /srv/http/games-site && git pull']}}}}
+repos['aif-ng'] = {'cmds': [['asciidoctor', '/opt/git/repo.checkouts/aif-ng/docs/README.adoc', '-o', '/srv/http/aif/index.html']]}
+
+def execHook(gitinfo = False):
+    if not gitinfo:
+        gitinfo = getGitInfo()
+    repo = gitinfo['repo'].lower()
+    print('Executing hooks for {0}:{1}...'.format(repo, gitinfo['branch']))
+    print('This commit: {0}\nLast commit: {1}'.format(gitinfo['currev'], gitinfo['oldrev']))
+    # Execute local commands first
+    if 'cmds' in repos[repo].keys():
+        for cmd in repos[repo]['cmds']:
+            print('\tExecuting {0}...'.format(' '.join(cmd)))
+            subprocess.call(cmd)
+    if 'remotecmds' in repos[repo].keys():
+        for host in repos[repo]['remotecmds'].keys():
+            if 'port' in repos[repo]['remotecmds'][host].keys():
+                port = int(repos[repo]['remotecmds'][host]['port'])
+            else:
+                port = 22
+            for user in repos[repo]['remotecmds'][host].keys():
+                print('{0}@{1}:'.format(user, host))
+                if paramikomodule:
+                    ssh = paramiko.SSHClient()
+                    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+                    ssh.connect(host, username = user, port = port)
+                    try:
+                        for cmd in repos[repo]['remotecmds'][host][user]['cmds']:
+                            print('\tExecuting \'{0}\'...'.format(cmd))
+                            stdin, stdout, stderr = ssh.exec_command(cmd)
+                            stdout = stdout.read().decode('utf-8')
+                            stderr = stderr.read().decode('utf-8')
+                            print(stdout)
+                            if stderr != '':
+                                print(stderr)
+                    except paramiko.AuthenticationException:
+                        print('({0}@{1}) AUTHENTICATION FAILED!'.format(user, host))
+                    except paramiko.BadHostKeyException:
+                        print('({0}@{1}) INCORRECT HOSTKEY!'.format(user, host))
+                    except paramiko.SSHException:
+                        print('({0}@{1}) FAILED TO ESTABLISH SSH!'.format(user, host))
+                    except socket.error:
+                        print('({0}@{1}) SOCKET CONNECTION FAILURE! (DNS, timeout/firewall, etc.)'.format(user, host))
+                else:
+                    for cmd in repos[repo]['remotecmds'][host][user]['cmds']:
+                        try:
+                            print('\tExecuting \'{0}\'...'.format(cmd))
+                            subprocess.call(['ssh', '{0}@{1}'.format(user, host), cmd])
+                        except:
+                            print('({0}@{1}) An error occurred!'.format(user, host))
+
+def getGitInfo():
+    refs = sys.argv[1].split('/')
+    gitinfo = {}
+    if refs[1] == 'tags':
+        gitinfo['branch'] = False
+        gitinfo['tag'] = refs[2]
+    elif refs[1] == 'heads':
+        gitinfo['branch'] = refs[2]
+        gitinfo['tag'] = False
+    gitinfo['repo'] = os.environ['GL_REPO']
+    gitinfo['user'] = os.environ['GL_USER']
+    clientinfo = os.environ['SSH_CONNECTION'].split()
+    gitinfo['ssh'] = {'client': {'ip': clientinfo[0], 'port': clientinfo[1]},
+                      'server': {'ip': clientinfo[2], 'port': clientinfo[3]},
+                      'user': os.environ['USER']
+                     }
+    if os.environ['GIT_DIR'] == '.':
+        gitinfo['dir'] = os.environ['PWD']
+    else:
+        #gitinfo['dir'] = os.path.join(os.environ['GL_REPO_BASE'], gitinfo['repo'], '.git')
+        gitinfo['dir'] = os.path.abspath(os.path.expanduser(os.environ['GIT_DIR']))
+    if gitmodule:
+        # This is preferred, because it's a lot more faster and a lot more flexible.
+        #https://gitpython.readthedocs.io/en/stable
+        gitobj = git.Repo(gitinfo['dir'])
+        commits = list(gitobj.iter_commits(gitobj.head.ref.name, max_count = 2))
+    else:
+        commits = subprocess.check_output(['git', 'rev-parse', 'HEAD..HEAD^1']).decode('utf-8').splitlines()
+    gitinfo['oldrev'] = re.sub('^\^', '', commits[1])
+    gitinfo['currev'] = re.sub('^\^', '', commits[0])
+    return(gitinfo)
+    #sys.exit(0)
+
+def main():
+    execHook()
+
+if __name__ == '__main__':
+    main()

git/remotehooks2.py

@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+
+import json
+import os
+import re
+import sys
+# Can we use paramiko for remotecmds?
+try:
+    import paramiko
+    import socket
+    has_ssh = True
+except ImportError:
+    has_ssh = False
+# Can we use the python git module?
+try:
+    import git  # "python-gitpython" in Arch; https://github.com/gitpython-developers/gitpython
+    has_git = True
+except ImportError:
+    has_git = False
+
+
+class repoHooks(object):
+    def __init__(self):
+        with open(os.path.join(os.environ['HOME'],
+                               '.gitolite',
+                               'local',
+                               'hooks',
+                               'repo-specific',
+                               'githooks.json'), 'r') as f:
+            self.cfg = json.loads(f.read())
+        self.repos = list(self.cfg.keys())
+        self.env = os.environ.copy()
+        if 'GIT_DIR' in self.env.keys():
+            del(self.env['GIT_DIR'])
+        self.repo = self.env['GL_REPO']
+
+    def remoteExec(self):
+        for _host in self.repos[self.repo]['remotecmds'].keys():
+            if len(_host.split(':')) == 2:
+                _server, _port = [i.strip() for i in _host.split(':')]
+            else:
+                _port = 22
+                _server = _host.split(':')[0]
+            _h = self.repos[self.repo]['remotecmds'][_host]
+            for _user in _h.keys():
+                _u = _h[_user]
+                if has_ssh:
+                    _ssh = paramiko.SSHClient()
+                    _ssh.load_system_host_keys()
+                    _ssh.missing_host_key_policy(paramiko.AutoAddPolicy())
+                    _ssh.connect(_server,
+                                 int(_port),
+                                 _user)
+                    for _cmd in _h.keys():
+                        pass  # DO STUFF HERE
+                else:
+                    return()  # no-op; no paramiko
+
+    def localExec(self):
+        pass
+
+def main():
+    h = repoHooks()
+    if h.repo not in h.repos:
+        return()
+
+
+if __name__ == '__main__':
+    main()

git/sample.githooks.json

@@ -0,0 +1,27 @@
+# remotehooks.py should go in your <gitolite repo>/local/hooks/repo-specific directory,
+# along with the (uncommented) format of this file configured for your particular hooks
+# "cmds" is a list of commands performed locally on the gitolite server,
+# "remotecmds" contains a recursive directory of commands to run remotely
+
+{
+    "<REPO_NAME>": {
+        "remotecmds": {
+            "<HOST_OR_IP_ADDRESS>": {
+                "<USER>": {
+                    "cmds": [
+                        "<COMMAND_1>",
+                        "<COMMAND_2>"
+                    ]
+                }
+            }
+        }
+    },
+    "<REPO2_NAME>": {
+        "cmds": [
+            [
+                "<LOCAL_COMMAND_1>",
+                "<LOCAL_COMMAND_2>"
+            ]
+        ]
+    }
+}

gpg/keystats.py

@@ -0,0 +1,285 @@
+#!/usr/bin/env python3
+
+# Get various information about an SKS keyserver from its status page
+# without opening a browser.
+# Requires BeautifulSoup4 and (optional but recommended) the lxml module.
+
+# stdlib
+import argparse
+import datetime
+import os
+import re
+import socket
+from urllib.request import urlopen, urlparse
+# pypi/pip
+from bs4 import BeautifulSoup
+try:
+    import lxml
+    bs_parser = 'lxml'
+except ImportError:
+    bs_parser = 'html.parser'
+
+socket_orig = socket.getaddrinfo
+
+def ForceProtov4(host, port, family = 0, socktype = 0, proto = 0,
+                       flags = 0):
+    return(socket_orig(host, port, socket.AF_INET, socktype, proto, flags))
+
+def ForceProtov6(host, port, family = 0, socktype = 0, proto = 0,
+                       flags = 0):
+    return(socket_orig(host, port, socket.AF_INET6, socktype, proto, flags))
+
+class KeyStats(object):
+    def __init__(self, server, port = None, tls = True, netproto = None,
+                 proto = 'http', output = 'py', verbose = True):
+        self.stats = {'server': {},
+                      'keys': 0}
+        if verbose:
+            self.stats['peers'] = {}
+            self.stats['histograms'] = {}
+        # Currently I only support scraping the stats page of the keyserver.
+        # TODO: Can I do this directly via HKP/HKPS? Is there a python module
+        # for it?
+        self.port_dflts = {'http': {True: 443,
+                                    False: 80,
+                                    None: 80}}
+        self.server = server
+        self.tls = tls
+        self.netproto = netproto
+        # We need to do some... ugly, hacky stuff to *force* a particular
+        # network stack (IPv4 vs. IPv6).
+        # https://stackoverflow.com/a/6319043/733214
+        if self.netproto:
+            if self.netproto == 'ipv6':
+                socket.getaddrinfo = ForceProtov6
+            elif self.netproto == 'ipv4':
+                socket.getaddrinfo = ForceProtov4
+        self.verbose = verbose
+        self.output = output
+        self.proto = proto.lower()
+        # TODO: would need to add add'l protocol support here.
+        if self.proto in ('http', 'https'):
+            self.proto = 'http'
+        if not port:
+            self.port = self.port_dflts[self.proto][self.tls]
+        else:
+            self.port = int(port)
+        if self.proto == 'http':
+            self.getStatsPage()
+
+    def getStatsPage(self):
+        if self.proto is not 'http':
+            # Something went wrong; this function shouldn't be used for
+            # non-http.
+            return()
+        _str_map = {'Hostname': 'name',
+                    'Nodename': 'hostname',
+                    'Version': 'version',
+                    'Server contact': 'contact',
+                    'HTTP port': 'hkp_port',
+                    'Recon port': 'recon_port',
+                    'Debug level': 'debug'}
+        _uri = 'pks/lookup?op=stats'
+        _url = '{0}://{1}:{2}/{3}'.format(('https' if self.tls else 'http'),
+                                          self.server,
+                                          self.port,
+                                          _uri)
+        with urlopen(_url) as u:
+            _webdata = u.read()
+        _soup = BeautifulSoup(_webdata, bs_parser)
+        for e in _soup.find_all('h2'):
+            # General server info
+            if e.text == 'Settings':
+                t = e.find_next('table',
+                                attrs = {'summary': 'Keyserver Settings'})
+                for r in t.find_all('tr'):
+                    h = None
+                    row = [re.sub(':$', '',
+                                  i.text.strip()) for i in r.find_all('td')]
+                    h = row[0]
+                    if h in _str_map.keys():
+                        if _str_map[h] in ('debug', 'hkp_port', 'recon_port'):
+                            self.stats['server'][_str_map[h]] = int(row[1])
+                        elif _str_map[h] == 'version':
+                            self.stats['server'][_str_map[h]] = tuple(
+                                                            row[1].split('.'))
+                        else:
+                            self.stats['server'][_str_map[h]] = row[1]
+            # "Gossip" (recon) peers list
+            elif e.text == 'Gossip Peers' and self.verbose:
+                self.stats['peers']['recon'] = []
+                t = e.find_next('table',
+                                attrs = {'summary': 'Gossip Peers'})
+                for r in t.find_all('tr'):
+                    _peer = list(r.children)[0].text.split()
+                    # A tuple consisting of host/name, port.
+                    self.stats['peers']['recon'].append((_peer[0],
+                                                         int(_peer[1])))
+            # Mailsync peers list
+            elif e.text == 'Outgoing Mailsync Peers' and self.verbose:
+                self.stats['peers']['mailsync'] = []
+                t = e.find_next('table', attrs = {'summary': 'Mailsync Peers'})
+                for r in t.find_all('tr'):
+                    _address = list(r.children)[0].text.strip()
+                    self.stats['peers']['mailsync'].append(_address)
+            # Number of keys
+            elif e.text == 'Statistics':
+                self.stats['keys'] = int(e.find_next('p').text.split()[-1])
+        # Histograms
+        for e in _soup.find_all('h3'):
+            # Dailies
+            if e.text == 'Daily Histogram' and self.verbose:
+                _dfmt = '%Y-%m-%d'
+                t = e.find_next('table', attrs = {'summary': 'Statistics'})
+                for r in t.find_all('tr'):
+                    row = [i.text.strip() for i in r.find_all('td')]
+                    if row[0] == 'Time':
+                        continue
+                    _date = datetime.datetime.strptime(row[0], _dfmt)
+                    _new = int(row[1])
+                    _updated = int(row[2])
+                    # JSON can't convert datetime objects to strings
+                    # automatically like PyYAML can.
+                    if self.output == 'json':
+                        k = str(_date)
+                    else:
+                        k = _date
+                    self.stats['histograms'][k] = {'total': {'new': _new,
+                                                             'updated': \
+                                                                    _updated},
+                                                   'hourly': {}}
+            # Hourlies
+            elif e.text == 'Hourly Histogram' and self.verbose:
+                _dfmt = '%Y-%m-%d %H'
+                t = e.find_next('table', attrs = {'summary': 'Statistics'})
+                for r in t.find_all('tr'):
+                    row = [i.text.strip() for i in r.find_all('td')]
+                    if row[0] == 'Time':
+                        continue
+                    _date = datetime.datetime.strptime(row[0], _dfmt)
+                    _new = int(row[1])
+                    _updated = int(row[2])
+                    _day = datetime.datetime(year = _date.year,
+                                             month = _date.month,
+                                             day = _date.day)
+                    if self.output == 'json':
+                        k1 = str(_day)
+                        k2 = str(_date)
+                    else:
+                        k1 = _day
+                        k2 = _date
+                    self.stats['histograms'][k1]['hourly'][k2] = {'new': _new,
+                                                                  'updated': \
+                                                                     _updated}
+        return()
+
+    def print(self):
+        if self.output == 'json':
+            import json
+            print(json.dumps(self.stats,
+                             #indent = 4,
+                             default = str))
+        elif self.output == 'yaml':
+            has_yaml = False
+            if 'YAML_MOD' in os.environ.keys():
+                _mod = os.environ['YAML_MOD']
+                try:
+                    import importlib
+                    yaml = importlib.import_module(_mod)
+                    has_yaml = True
+                except (ImportError, ModuleNotFoundError):
+                    raise RuntimeError(('Module "{0}" is not ' +
+                                        'installed').format(_mod))
+            else:
+                try:
+                    import yaml
+                    has_yaml = True
+                except ImportError:
+                    pass
+                try:
+                    import pyaml as yaml
+                    has_yaml = True
+                except ImportError:
+                    pass
+            if not has_yaml:
+                raise RuntimeError(('You must have the PyYAML or pyaml ' +
+                                    'module installed to use YAML ' +
+                                    'formatting'))
+            print(yaml.dump(self.stats))
+        elif self.output == 'py':
+            import pprint
+            pprint.pprint(self.stats)
+        return()
+
+def parseArgs():
+    args = argparse.ArgumentParser()
+    args.add_argument('-i', '--insecure',
+                      dest = 'tls',
+                      action = 'store_false',
+                      help = ('If specified, do not use TLS encryption ' +
+                              'querying the server (default is to use TLS)'))
+    args.add_argument('-P', '--port',
+                      dest = 'port',
+                      type = int,
+                      default = None,
+                      help = ('The port number to use. If not specified, ' +
+                              'use the default port per the normal protocol ' +
+                              '(i.e. for HTTPS, use 443)'))
+    fmt = args.add_mutually_exclusive_group()
+    fmt.add_argument('-j', '--json',
+                     default = 'py',
+                     dest = 'output',
+                     action = 'store_const',
+                     const = 'json',
+                     help = ('Output the data in JSON format'))
+    fmt.add_argument('-y', '--yaml',
+                     default = 'py',
+                     dest = 'output',
+                     action = 'store_const',
+                     const = 'yaml',
+                     help = ('Output the data in YAML format (requires ' +
+                             'PyYAML or pyaml module). You can prefer which ' +
+                             'one by setting an environment variable, ' +
+                             'YAML_MOD, to "yaml" or "pyaml" (for PyYAML or ' +
+                             'pyaml respectively); otherwise preference ' +
+                             'will be PyYAML > pyaml'))
+    fmt.add_argument('-p', '--python',
+                     default = 'py',
+                     dest = 'output',
+                     action = 'store_const',
+                     const = 'py',
+                     help = ('Output the data in pythonic format (default)'))
+    args.add_argument('-v', '--verbose',
+                      dest = 'verbose',
+                      action = 'store_true',
+                      help = ('If specified, print out ALL info (peers, ' +
+                              'histogram, etc.), not just the settings/' +
+                              'number of keys/contact info/server info'))
+    proto_grp = args.add_mutually_exclusive_group()
+    proto_grp.add_argument('-4', '--ipv4',
+                           dest = 'netproto',
+                           default = None,
+                           action = 'store_const',
+                           const = 'ipv4',
+                           help = ('If specified, force IPv4 (default is ' +
+                                   'system\'s preference)'))
+    proto_grp.add_argument('-6', '--ipv6',
+                           dest = 'netproto',
+                           default = None,
+                           action = 'store_const',
+                           const = 'ipv6',
+                           help = ('If specified, force IPv6 (default is ' +
+                                   'system\'s preference)'))
+    args.add_argument('server',
+                      help = ('The keyserver ((sub)domain, IP address, etc.)'))
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    import pprint
+    #pprint.pprint(args)
+    ks = KeyStats(**args)
+    ks.print()
+
+if __name__ == '__main__':
+    main()

gpg/sksdump.py

@@ -3,6 +3,7 @@
 # Thanks to Matt Rude and https://gist.github.com/mattrude/b0ac735d07b0031bb002 so I can know what the hell I'm doing.
 
 import argparse
+import base64
 import configparser
 import datetime
 import getpass
@@ -16,8 +17,10 @@ NOWstr = NOW.strftime('%Y-%m-%d')
 
 # TODO:
 # - cleanup/rotation should be optional
+# - turn into a class so we can more easily share vars across functions
+# - also, create the "CURRENT" symlink *AFTER* the dump completes?
 
-cfgfile = os.path.join(os.environ['HOME'], '.sksdump.ini')
+cfgfile = os.path.join(os.environ['HOME'], '.config', 'optools', 'sksdump.ini')
 
 def getDefaults():
     # Hardcoded defaults
@@ -28,56 +31,78 @@ def getDefaults():
                        'logfile': '/var/log/sksdump.log',
                        'days': 1,
                        'dumpkeys': 15000},
+            'sync': {'throttle': 0},
             'paths': {'basedir': '/var/lib/sks',
                       'destdir': '/srv/http/sks/dumps',
-                      'rsync': 'root@mirror.square-r00t.net:/srv/http/sks/dumps'},
+                      'rsync': ('root@mirror.square-r00t.net:' +
+                                '/srv/http/sks/dumps'),
+                      'sksbin': '/usr/bin/sks'},
             'runtime': {'nodump': None, 'nocompress': None, 'nosync': None}}
     ## Build out the default .ini.
-    dflt_str = ('# IMPORTANT: This script uses certain permissions functions that require some forethought.\n' +
-                '# You can either run as root, which is the "easy" way, OR you can run as the sks user.\n' +
-                '# Has to be one or the other; you\'ll SERIOUSLY mess things up otherwise.\n' +
-                '# If you run as the sks user, MAKE SURE the following is set in your sudoers\n' +
-                '# (where SKSUSER is the username sks runs as):\n#\tCmnd_Alias SKSCMDS = ' +
-                '/usr/bin/systemctl start sks-db,\\\n#\t\t/usr/bin/systemctl stop sks-db,\\\n#\t\t' +
-                '/usr/bin/systemctl start sks-recon,\\\n#\t\t/usr/bin/systemctl stop sks-recon\n#\t' +
-                'SKSUSER ALL = NOPASSWD: SKSCMDS\n\n')
-    dflt_str += ('# This was written for systemd systems only. Tweaking would be needed for non-systemd systems\n' +
-                 '# (since every non-systemd uses their own init system callables...)\n\n')
-    # [system]
-    d = dflt['system']
-    dflt_str += ('## SKSDUMP CONFIG FILE ##\n\n# This section controls various system configuration.\n' +
-                '[system]\n# This should be the user SKS runs as.\nuser = {0}\n# This is the group that' +
-                'SKS runs as.\ngroup = {1}\n# If None, don\'t compress dumps.\n# If one of: ' +
-                'xz, gz, bz2, or lrz (for lrzip) then use that compression algo.\ncompress = {2}\n' +
-                '# These services will be started/stopped, in order, before/after dumps. ' +
-                'Comma-separated.\nsvcs = {3}\n# The path to the logfile.\nlogfile = {4}\n# The number ' +
-                'of days of rotated key dumps. If None, don\'t rotate.\ndays = {5}\n# How many keys to include in each ' +
-                'dump file.\ndumpkeys = {6}\n\n').format(d['user'],
-                                                         d['group'],
-                                                         d['compress'],
-                                                         ','.join(d['svcs']),
-                                                         d['logfile'],
-                                                         d['days'],
-                                                         d['dumpkeys'])
-    # [paths]
-    d = dflt['paths']
-    dflt_str += ('# This section controls where stuff goes and where we should find it.\n[paths]\n# ' +
-                 'Where your SKS DB is.\nbasedir = {0}\n# This is the base directory where the dumps should go.\n' +
-                 '# There will be a sub-directory created for each date.\ndestdir = {1}\n# The ' +
-                 'path for rsyncing the dumps. If None, don\'t rsync.\nrsync = {2}\n\n').format(d['basedir'],
-                                                                                                d['destdir'],
-                                                                                                d['rsync'])
-    # [runtime]
-    d = dflt['runtime']
-    dflt_str += ('# This section controls runtime options. These can be overridden at the commandline.\n' +
-                 '# They take no values; they\'re merely options.\n[runtime]\n# Don\'t dump any keys.\n' +
-                 '# Useful for dedicated in-transit/prep boxes.\n;nodump\n# Don\'t compress the dumps, even if ' +
-                 'we have a compression scheme specified in [system:compress].\n;nocompress\n# Don\'t sync to' +
-                 'another server/path, even if one is specified in [paths:rsync].\n;nosync\n')
+    dflt_b64 = ("""IyBJTVBPUlRBTlQ6IFRoaXMgc2NyaXB0IHVzZXMgY2VydGFpbiBwZXJtaXNz
+                   aW9ucyBmdW5jdGlvbnMgdGhhdCByZXF1aXJlIHNvbWUKIyBmb3JldGhvdWdo
+                   dC4KIyBZb3UgY2FuIGVpdGhlciBydW4gYXMgcm9vdCwgd2hpY2ggaXMgdGhl
+                   ICJlYXN5IiB3YXksIE9SIHlvdSBjYW4gcnVuIGFzIHRoZQojIHNrcyB1c2Vy
+                   IChvci4uLiB3aGF0ZXZlciB1c2VyIHlvdXIgU0tTIGluc3RhbmNlIHJ1bnMg
+                   YXMpLgojIEl0IGhhcyB0byBiZSBvbmUgb3IgdGhlIG90aGVyOyB5b3UnbGwg
+                   U0VSSU9VU0xZIG1lc3MgdGhpbmdzIHVwIG90aGVyd2lzZS4KIyBJZiB5b3Ug
+                   cnVuIGFzIHRoZSBza3MgdXNlciwgTUFLRSBTVVJFIHRoZSBmb2xsb3dpbmcg
+                   aXMgc2V0IGluIHlvdXIgc3Vkb2VycwojICh3aGVyZSBTS1NVU0VSIGlzIHRo
+                   ZSB1c2VybmFtZSBza3MgcnVucyBhcyk6CiMJQ21uZF9BbGlhcyBTS1NDTURT
+                   ID0gL3Vzci9iaW4vc3lzdGVtY3RsIHN0YXJ0IHNrcy1kYixcCiMJICAgICAg
+                   ICAgICAgICAgICAgICAgL3Vzci9iaW4vc3lzdGVtY3RsIHN0b3Agc2tzLWRi
+                   LFwKIyAgICAgICAgICAgICAgICAgICAgICAgIC91c3IvYmluL3N5c3RlbWN0
+                   bCBzdGFydCBza3MtcmVjb24sXAojCQkgICAgICAgICAgICAgICAgIC91c3Iv
+                   YmluL3N5c3RlbWN0bCBzdG9wIHNrcy1yZWNvbgojCVNLU1VTRVIgQUxMID0g
+                   Tk9QQVNTV0Q6IFNLU0NNRFMKCiMgVGhpcyB3YXMgd3JpdHRlbiBmb3Igc3lz
+                   dGVtZCBzeXN0ZW1zIG9ubHkuIFR3ZWFraW5nIHdvdWxkIGJlIG5lZWRlZCBm
+                   b3IKIyBub24tc3lzdGVtZCBzeXN0ZW1zIChzaW5jZSBldmVyeSBub24tc3lz
+                   dGVtZCB1c2VzIHRoZWlyIG93biBpbml0IHN5c3RlbQojIGNhbGxhYmxlcy4u
+                   LikKCiMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj
+                   IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMKCiMgVGhp
+                   cyBzZWN0aW9uIGNvbnRyb2xzIHZhcmlvdXMgc3lzdGVtIGNvbmZpZ3VyYXRp
+                   b24uCltzeXN0ZW1dCgojIFRoaXMgc2hvdWxkIGJlIHRoZSB1c2VyIFNLUyBy
+                   dW5zIGFzLgp1c2VyID0gc2tzCgojIFRoaXMgaXMgdGhlIGdyb3VwIHRoYXQg
+                   U0tTIHJ1bnMgYXMuCmdyb3VwID0gc2tzCgojIElmIGVtcHR5LCBkb24ndCBj
+                   b21wcmVzcyBkdW1wcy4KIyBJZiBvbmUgb2Y6IHh6LCBneiwgYnoyLCBvciBs
+                   cnogKGZvciBscnppcCkgdGhlbiB1c2UgdGhhdCBjb21wcmVzc2lvbiBhbGdv
+                   LgojIE5vdGUgdGhhdCBscnppcCByZXF1aXJlcyBleHRyYSBpbnN0YWxsYXRp
+                   b24uCmNvbXByZXNzID0geHoKCiMgVGhlc2Ugc2VydmljZXMgd2lsbCBiZSBz
+                   dG9wcGVkL3N0YXJ0ZWQsIGluIG9yZGVyLCBiZWZvcmUvYWZ0ZXIgZHVtcHMu
+                   IElmIG1vcmUKIyB0aGFuIG9uZSwgc2VwZXJhdGUgYnkgY29tbWFzLgpzdmNz
+                   ID0gc2tzLWRiLHNrcy1yZWNvbgoKIyBUaGUgcGF0aCB0byB0aGUgbG9nZmls
+                   ZS4KbG9nZmlsZSA9IC92YXIvbG9nL3Nrc2R1bXAubG9nCgojIFRoZSBudW1i
+                   ZXIgb2YgZGF5cyBvZiByb3RhdGVkIGtleSBkdW1wcy4gSWYgZW1wdHksIGRv
+                   bid0IHJvdGF0ZS4KZGF5cyA9IDEKCiMgSG93IG1hbnkga2V5cyB0byBpbmNs
+                   dWRlIGluIGVhY2ggZHVtcCBmaWxlLgpkdW1wa2V5cyA9IDE1MDAwCgoKIyBU
+                   aGlzIHNlY3Rpb24gY29udHJvbHMgc3luYyBzZXR0aW5ncy4KW3N5bmNdCgoj
+                   IFRoaXMgc2V0dGluZyBpcyB3aGF0IHRoZSBzcGVlZCBzaG91bGQgYmUgdGhy
+                   b3R0bGVkIHRvLCBpbiBLaUIvcy4gSWYgZW1wdHkgb3IKIyAwLCBwZXJmb3Jt
+                   IG5vIHRocm90dGxpbmcuCnRocm90dGxlID0gMAoKCiMgVGhpcyBzZWN0aW9u
+                   IGNvbnRyb2xzIHdoZXJlIHN0dWZmIGdvZXMgYW5kIHdoZXJlIHdlIHNob3Vs
+                   ZCBmaW5kIGl0LgpbcGF0aHNdCgojIFdoZXJlIHlvdXIgU0tTIERCIGlzLgpi
+                   YXNlZGlyID0gL3Zhci9saWIvc2tzCgojIFRoaXMgaXMgdGhlIGJhc2UgZGly
+                   ZWN0b3J5IHdoZXJlIHRoZSBkdW1wcyBzaG91bGQgZ28uCiMgVGhlcmUgd2ls
+                   bCBiZSBhIHN1Yi1kaXJlY3RvcnkgY3JlYXRlZCBmb3IgZWFjaCBkYXRlLgpk
+                   ZXN0ZGlyID0gL3Nydi9odHRwL3Nrcy9kdW1wcwoKIyBUaGUgcGF0aCBmb3Ig
+                   cnN5bmNpbmcgdGhlIGR1bXBzLiBJZiBlbXB0eSwgZG9uJ3QgcnN5bmMuCnJz
+                   eW5jID0gcm9vdEBtaXJyb3Iuc3F1YXJlLXIwMHQubmV0Oi9zcnYvaHR0cC9z
+                   a3MvZHVtcHMKCiMgVGhlIHBhdGggdG8gdGhlIHNrcyBiaW5hcnkgdG8gdXNl
+                   Lgpza3NiaW4gPSAvdXNyL2Jpbi9za3MKCgojIFRoaXMgc2VjdGlvbiBjb250
+                   cm9scyBydW50aW1lIG9wdGlvbnMuIFRoZXNlIGNhbiBiZSBvdmVycmlkZGVu
+                   IGF0IHRoZQojIGNvbW1hbmRsaW5lLiBUaGV5IHRha2Ugbm8gdmFsdWVzOyB0
+                   aGV5J3JlIG1lcmVseSBvcHRpb25zLgpbcnVudGltZV0KCiMgRG9uJ3QgZHVt
+                   cCBhbnkga2V5cy4KIyBVc2VmdWwgZm9yIGRlZGljYXRlZCBpbi10cmFuc2l0
+                   L3ByZXAgYm94ZXMuCjtub2R1bXAKCiMgRG9uJ3QgY29tcHJlc3MgdGhlIGR1
+                   bXBzLCBldmVuIGlmIHdlIGhhdmUgYSBjb21wcmVzc2lvbiBzY2hlbWUgc3Bl
+                   Y2lmaWVkIGluCiMgdGhlIFtzeXN0ZW06Y29tcHJlc3NdIHNlY3Rpb246ZGly
+                   ZWN0aXZlLgo7bm9jb21wcmVzcwoKIyBEb24ndCBzeW5jIHRvIGFub3RoZXIg
+                   c2VydmVyL3BhdGgsIGV2ZW4gaWYgb25lIGlzIHNwZWNpZmllZCBpbiBbcGF0
+                   aHM6cnN5bmNdLgo7bm9zeW5j""")
     realcfg = configparser.ConfigParser(defaults = dflt, allow_no_value = True)
     if not os.path.isfile(cfgfile):
         with open(cfgfile, 'w') as f:
-            f.write(dflt_str)
+            f.write(base64.b64decode(dflt_b64).decode('utf-8'))
     realcfg.read(cfgfile)
     return(realcfg)
 
@@ -115,7 +140,10 @@ def destPrep(args):
             _dir = os.path.join(thisdir, d)
             if os.path.isdir(_dir):
                 if len(os.listdir(_dir)) == 0:
-                    os.rmdir(os.path.join(thisdir, d))
+                    try:
+                        os.rmdir(os.path.join(thisdir, d))
+                    except NotADirectoryError:
+                        pass  # in case it grabs the "current" symlink
     #try:
     #    os.removedirs(sks['destdir'])  # Remove empty dirs
     #except:
@@ -124,18 +152,23 @@ def destPrep(args):
     if getpass.getuser() == 'root':
         uid = getpwnam(args['user']).pw_uid
         gid = getgrnam(args['group']).gr_gid
-        for d in (args['destdir'], nowdir):  # we COULD set it as part of the os.makedirs, but iirc it doesn't set it for existing dirs
+        # we COULD set it as part of the os.makedirs, but iirc it doesn't set
+        # it for existing dirs.
+        for d in (args['destdir'], nowdir):
             os.chown(d, uid, gid)
     if os.path.isdir(curdir):
         os.remove(curdir)
-    os.symlink(NOWstr, curdir, target_is_directory = True)
+    try:
+        os.symlink(NOWstr, curdir, target_is_directory = True)
+    except FileExistsError:
+        pass  # Ignore if it was set earlier
     return()
 
 def dumpDB(args):
     destPrep(args)
     os.chdir(args['basedir'])
     svcMgmt('stop', args)
-    cmd = ['sks',
+    cmd = [args['sksbin'],
            'dump',
            str(args['dumpkeys']),  # How many keys per dump?
            os.path.join(args['destdir'], NOWstr),  # Where should it go?
@@ -154,7 +187,9 @@ def compressDB(args):
     if not args['compress']:
         return()
     curdir = os.path.join(args['destdir'], NOWstr)
-    for thisdir, dirs, files in os.walk(curdir):  # I use os.walk here because we might handle this differently in the future...
+    # I use os.walk here because we might handle this differently in the
+    # future...
+    for thisdir, dirs, files in os.walk(curdir):
         files.sort()
         for f in files:
             fullpath = os.path.join(thisdir, f)
@@ -163,22 +198,30 @@ def compressDB(args):
             # However, I can't do this on memory-constrained systems for lrzip.
             # See: https://github.com/kata198/python-lrzip/issues/1
             with open(args['logfile'], 'a') as f:
-                f.write('===== {0} Now compressing {1} =====\n'.format(str(datetime.datetime.utcnow()), fullpath))
+                f.write('===== {0} Now compressing {1} =====\n'.format(
+                                            str(datetime.datetime.utcnow()),
+                                                                    fullpath))
             if args['compress'].lower() == 'gz':
                 import gzip
-                with open(fullpath, 'rb') as fh_in, gzip.open(newfile, 'wb') as fh_out:
+                with open(fullpath, 'rb') as fh_in, gzip.open(newfile,
+                                                              'wb') as fh_out:
                     fh_out.writelines(fh_in)
             elif args['compress'].lower() == 'xz':
                 import lzma
-                with open(fullpath, 'rb') as fh_in, lzma.open(newfile, 'wb', preset = 9|lzma.PRESET_EXTREME) as fh_out:
+                with open(fullpath, 'rb') as fh_in, \
+                        lzma.open(newfile,
+                                  'wb',
+                                  preset = 9|lzma.PRESET_EXTREME) as fh_out:
                     fh_out.writelines(fh_in)
             elif args['compress'].lower() == 'bz2':
                 import bz2
-                with open(fullpath, 'rb') as fh_in, bz2.open(newfile, 'wb') as fh_out:
+                with open(fullpath, 'rb') as fh_in, bz2.open(newfile,
+                                                             'wb') as fh_out:
                     fh_out.writelines(fh_in)
             elif args['compress'].lower() == 'lrz':
                 import lrzip
-                with open(fullpath, 'rb') as fh_in, open(newfile, 'wb') as fh_out:
+                with open(fullpath, 'rb') as fh_in, open(newfile,
+                                                         'wb') as fh_out:
                     fh_out.write(lrzip.compress(fh_in.read()))
             os.remove(fullpath)
             if getpass.getuser() == 'root':
@@ -195,8 +238,11 @@ def syncDB(args):
            '--delete',
            os.path.join(args['destdir'], '.'),
            args['rsync']]
+    if args['throttle'] > 0.0:
+        cmd.insert(-1, '--bwlimit={0}'.format(str(args['throttle'])))
     with open(args['logfile'], 'a') as f:
-        f.write('===== {0} Rsyncing to mirror =====\n'.format(str(datetime.datetime.utcnow())))
+        f.write('===== {0} Rsyncing to mirror =====\n'.format(
+                                            str(datetime.datetime.utcnow())))
     with open(args['logfile'], 'a') as f:
         subprocess.run(cmd, stdout = f, stderr = f)
     return()
@@ -205,9 +251,12 @@ def parseArgs():
     cfg = getDefaults()
     system = cfg['system']
     paths = cfg['paths']
+    sync = cfg['sync']
     runtime = cfg['runtime']
-    args = argparse.ArgumentParser(description = 'sksdump - a tool for dumping the SKS Database',
-                                   epilog = 'brent s. || 2017 || https://square-r00t.net')
+    args = argparse.ArgumentParser(description = ('sksdump - a tool for ' +
+                                                  'dumping an SKS Database'),
+                                   epilog = ('brent s. || 2018 || ' +
+                                             'https://square-r00t.net'))
     args.add_argument('-u',
                       '--user',
                       default = system['user'],
@@ -228,7 +277,9 @@ def parseArgs():
                       '--services',
                       default = system['svcs'],
                       dest = 'svcs',
-                      help = 'A comma-separated list of services that will be stopped/started for the dump (in the provided order).')
+                      help = ('A comma-separated list of services that will ' +
+                              'be stopped/started for the dump (in the ' +
+                              'provided order).'))
     args.add_argument('-l',
                       '--log',
                       default = system['logfile'],
@@ -251,16 +302,32 @@ def parseArgs():
                       default = paths['basedir'],
                       dest = 'basedir',
                       help = 'The directory which holds your SKS DB.')
+    args.add_argument('-x',
+                      '--sks-binary',
+                      default = paths['sksbin'],
+                      dest = 'sksbin',
+                      help = ('The path to the SKS binary/executable to use ' +
+                              'to perform the dump.'))
     args.add_argument('-e',
                       '--destdir',
                       default = paths['destdir'],
                       dest = 'destdir',
-                      help = 'The directory where the dumps should be saved (a sub-directory with the date will be created).')
+                      help = ('The directory where the dumps should be ' +
+                              'saved (a sub-directory with the date will be ' +
+                              'created).'))
     args.add_argument('-r',
                       '--rsync',
                       default = paths['rsync'],
                       dest = 'rsync',
-                      help = 'The remote (user@host:/path/) or local (/path/) path to use to sync the dumps to.')
+                      help = ('The remote (user@host:/path/) or local '+
+                              '(/path/) path to use to sync the dumps to.'))
+    args.add_argument('-t',
+                      '--throttle',
+                      default = float(sync['throttle']),
+                      dest = 'throttle',
+                      type = float,
+                      help = ('The amount in KiB/s to throttle the rsync ' +
+                              'to. Use 0 for no throttling.'))
     args.add_argument('-D',
                       '--no-dump',
                       dest = 'nodump',
@@ -272,7 +339,8 @@ def parseArgs():
                       dest = 'nocompress',
                       action = 'store_true',
                       default = ('nocompress' in runtime),
-                      help = 'Don\'t compress the DB dumps (default is to compress)')
+                      help = ('Don\'t compress the DB dumps (default is to ' +
+                              'compress)'))
     args.add_argument('-S',
                       '--no-sync',
                       dest = 'nosync',
@@ -287,7 +355,8 @@ def main():
     if getpass.getuser() not in ('root', args['user']):
         exit('ERROR: You must be root or {0}!'.format(args['user']))
     with open(args['logfile'], 'a') as f:
-        f.write('===== {0} STARTING =====\n'.format(str(datetime.datetime.utcnow())))
+        f.write('===== {0} STARTING =====\n'.format(
+                                            str(datetime.datetime.utcnow())))
     if not args['nodump']:
         dumpDB(args)
     if not args['nocompress']:
@@ -295,7 +364,11 @@ def main():
     if not args['nosync']:
         syncDB(args)
     with open(args['logfile'], 'a') as f:
-        f.write('===== {0} DONE =====\n'.format(str(datetime.datetime.utcnow())))
+        f.write('===== {0} DONE =====\n'.format(
+                                            str(datetime.datetime.utcnow())))
+    with open(os.path.join(args['destdir'], 'LAST_COMPLETED_DUMP.txt'),
+              'w') as f:
+        f.write(str(datetime.datetime.utcnow()) + ' UTC\n')
 
 
 if __name__ == '__main__':

ldap/loglevel.py

@@ -0,0 +1,4 @@
+#!/usr/bin/env python3
+
+# https://www.openldap.org/doc/admin24/slapdconfig.html#loglevel%20%3Clevel%3E
+# https://www.zytrax.com/books/ldap/ch6/#loglevel

lib/python/logger.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+
+# The logfile.
+dflt_logfile = '/var/log/optools/optools.log'
+
+# The default log level. Can be one of (in increasing levels of output):
+# critical
+# error
+# warning
+# info
+# debug
+# "debug" may log sensitive information! Do *not* use it unless ABSOLUTELY
+# NECESSARY.
+dflt_loglevel = 'warning'
+
+# stdlib
+import datetime
+import logging
+import logging.handlers
+import os
+
+class log(object):
+    def __init__(self, loglvl = dflt_loglevel, logfile = dflt_logfile,
+                 logname = 'optools'):
+        # Loglevel mappings.
+        self.loglvls = {'critical': logging.CRITICAL,
+                        'error': logging.ERROR,
+                        'warning': logging.WARNING,
+                        'info': logging.INFO,
+                        'debug': logging.DEBUG}
+        self.loglvl = loglvl.lower()
+        if self.loglvl not in self.loglvls:
+            raise ValueError(('{0} is not one of: ' +
+                              '{1}').format(loglvl,
+                                            ', '.join(self.loglvls.keys())))
+        self.Logger = logging.getLogger(logname)
+        self.logfile = os.path.abspath(os.path.expanduser(logfile))
+        try:
+            os.makedirs(os.path.dirname(self.logfile),
+                        exist_ok = True,
+                        mode = 0o700)
+        except Exception as e:
+            # Make this non-fatal since we also log to journal for systemd?
+            raise e
+        self.chkSystemd()
+        self.journald()
+        self.Logger.setLevel(self.loglvls[self.loglvl])
+        self.log_handlers()
+
+    def chkSystemd(self):
+        # Add journald support if we're on systemd.
+        # We probably are since we're most likely on Arch, but we don't want to
+        # make assumptions.
+        self.systemd = False
+        _sysd_chk = ['/run/systemd/system',
+                     '/dev/.run/systemd',
+                     '/dev/.systemd']
+        for _ in _sysd_chk:
+            if os.path.isdir(_):
+                self.systemd = True
+                break
+        return()
+
+    def journald(self):
+        if not self.systemd:
+            return()
+        try:
+            from systemd import journal
+        except ImportError:
+            try:
+                import pip
+                pip.main(['install', '--user', 'systemd'])
+                from systemd import journal
+            except Exception as e:
+                # Build failed. Missing gcc, disk too full, whatever.
+                self.systemd = False
+        return()
+
+    def log_handlers(self):
+        # Log formats
+        if self.systemd:
+            _jrnlfmt = logging.Formatter(fmt = ('{levelname}: {message} ' +
+                                                '({filename}:{lineno})'),
+                                         style = '{',
+                                         datefmt = '%Y-%m-%d %H:%M:%S')
+        _logfmt = logging.Formatter(fmt = ('{asctime}:{levelname}: {message} (' +
+                                           '{filename}:{lineno})'),
+                                    style = '{',
+                                    datefmt = '%Y-%m-%d %H:%M:%S')
+        # Add handlers
+        _dflthandler = logging.handlers.RotatingFileHandler(self.logfile,
+                                                            encoding = 'utf8',
+                                                            # 1GB
+                                                            maxBytes = 1073741824,
+                                                            backupCount = 5)
+        _dflthandler.setFormatter(_logfmt)
+        _dflthandler.setLevel(self.loglvls[self.loglvl])
+        if self.systemd:
+            from systemd import journal
+            try:
+                h = journal.JournaldLogHandler()
+            except AttributeError:  # Uses the other version
+                h = journal.JournalHandler()
+            h.setFormatter(_jrnlfmt)
+            h.setLevel(self.loglvls[self.loglvl])
+            self.Logger.addHandler(h)
+        self.Logger.addHandler(_dflthandler)
+        self.Logger.info('Logging initialized')
+        return()

libvirt/README

@@ -0,0 +1 @@
+These projects/scripts have been moved to https://git.square-r00t.net/LibvirtTools/.

mumble/.gitignore

@@ -0,0 +1,2 @@
+/docs
+/testcertimport.py

mumble/Mumble.proto

@@ -0,0 +1,563 @@
+// Copyright 2005-2017 The Mumble Developers. All rights reserved.
+// Use of this source code is governed by a BSD-style license
+// that can be found in the LICENSE file at the root of the
+// Mumble source tree or at <https://www.mumble.info/LICENSE>.
+
+syntax = "proto2";
+
+package MumbleProto;
+
+option optimize_for = SPEED;
+
+message Version {
+	// 2-byte Major, 1-byte Minor and 1-byte Patch version number.
+	optional uint32 version = 1;
+	// Client release name.
+	optional string release = 2;
+	// Client OS name.
+	optional string os = 3;
+	// Client OS version.
+	optional string os_version = 4;
+}
+
+// Not used. Not even for tunneling UDP through TCP.
+message UDPTunnel {
+	// Not used.
+	required bytes packet = 1;
+}
+
+// Used by the client to send the authentication credentials to the server.
+message Authenticate {
+	// UTF-8 encoded username.
+	optional string username = 1;
+	// Server or user password.
+	optional string password = 2;
+	// Additional access tokens for server ACL groups.
+	repeated string tokens = 3;
+	// A list of CELT bitstream version constants supported by the client.
+	repeated int32 celt_versions = 4;
+	optional bool opus = 5 [default = false];
+}
+
+// Sent by the client to notify the server that the client is still alive.
+// Server must reply to the packet with the same timestamp and its own
+// good/late/lost/resync numbers. None of the fields is strictly required.
+message Ping {
+	// Client timestamp. Server should not attempt to decode.
+	optional uint64 timestamp = 1;
+	// The amount of good packets received.
+	optional uint32 good = 2;
+	// The amount of late packets received.
+	optional uint32 late = 3;
+	// The amount of packets never received.
+	optional uint32 lost = 4;
+	// The amount of nonce resyncs.
+	optional uint32 resync = 5;
+	// The total amount of UDP packets received.
+	optional uint32 udp_packets = 6;
+	// The total amount of TCP packets received.
+	optional uint32 tcp_packets = 7;
+	// UDP ping average.
+	optional float udp_ping_avg = 8;
+	// UDP ping variance.
+	optional float udp_ping_var = 9;
+	// TCP ping average.
+	optional float tcp_ping_avg = 10;
+	// TCP ping variance.
+	optional float tcp_ping_var = 11;
+}
+
+// Sent by the server when it rejects the user connection.
+message Reject {
+	enum RejectType {
+		// The rejection reason is unknown (details should be available
+		// in Reject.reason).
+		None = 0;
+		// The client attempted to connect with an incompatible version.
+		WrongVersion = 1;
+		// The user name supplied by the client was invalid.
+		InvalidUsername = 2;
+		// The client attempted to authenticate as a user with a password but it
+		// was wrong.
+		WrongUserPW = 3;
+		// The client attempted to connect to a passworded server but the password
+		// was wrong.
+		WrongServerPW = 4;
+		// Supplied username is already in use.
+		UsernameInUse = 5;
+		// Server is currently full and cannot accept more users.
+		ServerFull = 6;
+		// The user did not provide a certificate but one is required.
+		NoCertificate = 7;
+		AuthenticatorFail = 8;
+	}
+	// Rejection type.
+	optional RejectType type = 1;
+	// Human readable rejection reason.
+	optional string reason = 2;
+}
+
+// ServerSync message is sent by the server when it has authenticated the user
+// and finished synchronizing the server state.
+message ServerSync {
+	// The session of the current user.
+	optional uint32 session = 1;
+	// Maximum bandwidth that the user should use.
+	optional uint32 max_bandwidth = 2;
+	// Server welcome text.
+	optional string welcome_text = 3;
+	// Current user permissions in the root channel.
+	optional uint64 permissions = 4;
+}
+
+// Sent by the client when it wants a channel removed. Sent by the server when
+// a channel has been removed and clients should be notified.
+message ChannelRemove {
+	required uint32 channel_id = 1;
+}
+
+// Used to communicate channel properties between the client and the server.
+// Sent by the server during the login process or when channel properties are
+// updated. Client may use this message to update said channel properties.
+message ChannelState {
+	// Unique ID for the channel within the server.
+	optional uint32 channel_id = 1;
+	// channel_id of the parent channel.
+	optional uint32 parent = 2;
+	// UTF-8 encoded channel name.
+	optional string name = 3;
+	// A collection of channel id values of the linked channels. Absent during
+	// the first channel listing.
+	repeated uint32 links = 4;
+	// UTF-8 encoded channel description. Only if the description is less than
+	// 128 bytes
+	optional string description = 5;
+	// A collection of channel_id values that should be added to links.
+	repeated uint32 links_add = 6;
+	// A collection of channel_id values that should be removed from links.
+	repeated uint32 links_remove = 7;
+	// True if the channel is temporary.
+	optional bool temporary = 8 [default = false];
+	// Position weight to tweak the channel position in the channel list.
+	optional int32 position = 9 [default = 0];
+	// SHA1 hash of the description if the description is 128 bytes or more.
+	optional bytes description_hash = 10;
+	// Maximum number of users allowed in the channel. If this value is zero,
+	// the maximum number of users allowed in the channel is given by the
+	// server's "usersperchannel" setting.
+	optional uint32 max_users = 11;
+}
+
+// Used to communicate user leaving or being kicked. May be sent by the client
+// when it attempts to kick a user. Sent by the server when it informs the
+// clients that a user is not present anymore.
+message UserRemove {
+	// The user who is being kicked, identified by their session, not present
+	// when no one is being kicked.
+	required uint32 session = 1;
+	// The user who initiated the removal. Either the user who performs the kick
+	// or the user who is currently leaving.
+	optional uint32 actor = 2;
+	// Reason for the kick, stored as the ban reason if the user is banned.
+	optional string reason = 3;
+	// True if the kick should result in a ban.
+	optional bool ban = 4;
+}
+
+// Sent by the server when it communicates new and changed users to client.
+// First seen during login procedure. May be sent by the client when it wishes
+// to alter its state.
+message UserState {
+	// Unique user session ID of the user whose state this is, may change on
+	// reconnect.
+	optional uint32 session = 1;
+	// The session of the user who is updating this user.
+	optional uint32 actor = 2;
+	// User name, UTF-8 encoded.
+	optional string name = 3;
+	// Registered user ID if the user is registered.
+	optional uint32 user_id = 4;
+	// Channel on which the user is.
+	optional uint32 channel_id = 5;
+	// True if the user is muted by admin.
+	optional bool mute = 6;
+	// True if the user is deafened by admin.
+	optional bool deaf = 7;
+	// True if the user has been suppressed from talking by a reason other than
+	// being muted.
+	optional bool suppress = 8;
+	// True if the user has muted self.
+	optional bool self_mute = 9;
+	// True if the user has deafened self.
+	optional bool self_deaf = 10;
+	// User image if it is less than 128 bytes.
+	optional bytes texture = 11;
+	// The positional audio plugin identifier.
+	// Positional audio information is only sent to users who share
+	// identical plugin contexts.
+	//
+	// This value is not trasmitted to clients.
+	optional bytes plugin_context = 12;
+	// The user's plugin-specific identity.
+	// This value is not transmitted to clients.
+	optional string plugin_identity = 13;
+	// User comment if it is less than 128 bytes.
+	optional string comment = 14;
+	// The hash of the user certificate.
+	optional string hash = 15;
+	// SHA1 hash of the user comment if it 128 bytes or more.
+	optional bytes comment_hash = 16;
+	// SHA1 hash of the user picture if it 128 bytes or more.
+	optional bytes texture_hash = 17;
+	// True if the user is a priority speaker.
+	optional bool priority_speaker = 18;
+	// True if the user is currently recording.
+	optional bool recording = 19;
+}
+
+// Relays information on the bans. The client may send the BanList message to
+// either modify the list of bans or query them from the server. The server
+// sends this list only after a client queries for it.
+message BanList {
+	message BanEntry {
+		// Banned IP address.
+		required bytes address = 1;
+		// The length of the subnet mask for the ban.
+		required uint32 mask = 2;
+		// User name for identification purposes (does not affect the ban).
+		optional string name = 3;
+		// The certificate hash of the banned user.
+		optional string hash = 4;
+		// Reason for the ban (does not affect the ban).
+		optional string reason = 5;
+		// Ban start time.
+		optional string start = 6;
+		// Ban duration in seconds.
+		optional uint32 duration = 7;
+	}
+	// List of ban entries currently in place.
+	repeated BanEntry bans = 1;
+	// True if the server should return the list, false if it should replace old
+	// ban list with the one provided.
+	optional bool query = 2 [default = false];
+}
+
+// Used to send and broadcast text messages.
+message TextMessage {
+	// The message sender, identified by its session.
+	optional uint32 actor = 1;
+	// Target users for the message, identified by their session.
+	repeated uint32 session = 2;
+	// The channels to which the message is sent, identified by their
+	// channel_ids.
+	repeated uint32 channel_id = 3;
+	// The root channels when sending message recursively to several channels,
+	// identified by their channel_ids.
+	repeated uint32 tree_id = 4;
+	// The UTF-8 encoded message. May be HTML if the server allows.
+	required string message = 5;
+}
+
+message PermissionDenied {
+	enum DenyType {
+		// Operation denied for other reason, see reason field.
+		Text = 0;
+		// Permissions were denied.
+		Permission = 1;
+		// Cannot modify SuperUser.
+		SuperUser = 2;
+		// Invalid channel name.
+		ChannelName = 3;
+		// Text message too long.
+		TextTooLong = 4;
+		// The flux capacitor was spelled wrong.
+		H9K = 5;
+		// Operation not permitted in temporary channel.
+		TemporaryChannel = 6;
+		// Operation requires certificate.
+		MissingCertificate = 7;
+		// Invalid username.
+		UserName = 8;
+		// Channel is full.
+		ChannelFull = 9;
+		NestingLimit = 10;
+	}
+	// The denied permission when type is Permission.
+	optional uint32 permission = 1;
+	// channel_id for the channel where the permission was denied when type is
+	// Permission.
+	optional uint32 channel_id = 2;
+	// The user who was denied permissions, identified by session.
+	optional uint32 session = 3;
+	// Textual reason for the denial.
+	optional string reason = 4;
+	// Type of the denial.
+	optional DenyType type = 5;
+	// The name that is invalid when type is UserName.
+	optional string name = 6;
+}
+
+message ACL {
+	message ChanGroup {
+		// Name of the channel group, UTF-8 encoded.
+		required string name = 1;
+		// True if the group has been inherited from the parent (Read only).
+		optional bool inherited = 2 [default = true];
+		// True if the group members are inherited.
+		optional bool inherit = 3 [default = true];
+		// True if the group can be inherited by sub channels.
+		optional bool inheritable = 4 [default = true];
+		// Users explicitly included in this group, identified by user_id.
+		repeated uint32 add = 5;
+		// Users explicitly removed from this group in this channel if the group
+		// has been inherited, identified by user_id.
+		repeated uint32 remove = 6;
+		// Users inherited, identified by user_id.
+		repeated uint32 inherited_members = 7;
+	}
+	message ChanACL {
+		// True if this ACL applies to the current channel.
+		optional bool apply_here = 1 [default = true];
+		// True if this ACL applies to the sub channels.
+		optional bool apply_subs = 2 [default = true];
+		// True if the ACL has been inherited from the parent.
+		optional bool inherited = 3 [default = true];
+		// ID of the user that is affected by this ACL.
+		optional uint32 user_id = 4;
+		// ID of the group that is affected by this ACL.
+		optional string group = 5;
+		// Bit flag field of the permissions granted by this ACL.
+		optional uint32 grant = 6;
+		// Bit flag field of the permissions denied by this ACL.
+		optional uint32 deny = 7;
+	}
+	// Channel ID of the channel this message affects.
+	required uint32 channel_id = 1;
+	// True if the channel inherits its parent's ACLs.
+	optional bool inherit_acls = 2 [default = true];
+	// User group specifications.
+	repeated ChanGroup groups = 3;
+	// ACL specifications.
+	repeated ChanACL acls = 4;
+	// True if the message is a query for ACLs instead of setting them.
+	optional bool query = 5 [default = false];
+}
+
+// Client may use this message to refresh its registered user information. The
+// client should fill the IDs or Names of the users it wants to refresh. The
+// server fills the missing parts and sends the message back.
+message QueryUsers {
+	// user_ids.
+	repeated uint32 ids = 1;
+	// User names in the same order as ids.
+	repeated string names = 2;
+}
+
+// Used to initialize and resync the UDP encryption. Either side may request a
+// resync by sending the message without any values filled. The resync is
+// performed by sending the message with only the client or server nonce
+// filled.
+message CryptSetup {
+	// Encryption key.
+	optional bytes key = 1;
+	// Client nonce.
+	optional bytes client_nonce = 2;
+	// Server nonce.
+	optional bytes server_nonce = 3;
+}
+
+message ContextActionModify {
+	enum Context {
+		// Action is applicable to the server.
+		Server = 0x01;
+		// Action can target a Channel.
+		Channel = 0x02;
+		// Action can target a User.
+		User = 0x04;
+	}
+	enum Operation {
+		Add = 0;
+		Remove = 1;
+	}
+	// The action name.
+	required string action = 1;
+	// The display name of the action.
+	optional string text = 2;
+	// Context bit flags defining where the action should be displayed.
+	optional uint32 context = 3;
+	optional Operation operation = 4;
+}
+
+// Sent by the client when it wants to initiate a Context action.
+message ContextAction {
+	// The target User for the action, identified by session.
+	optional uint32 session = 1;
+	// The target Channel for the action, identified by channel_id.
+	optional uint32 channel_id = 2;
+	// The action that should be executed.
+	required string action = 3;
+}
+
+// Lists the registered users.
+message UserList {
+	message User {
+		// Registered user ID.
+		required uint32 user_id = 1;
+		// Registered user name.
+		optional string name = 2;
+		optional string last_seen = 3;
+		optional uint32 last_channel = 4;
+	}
+	// A list of registered users.
+	repeated User users = 1;
+}
+
+// Sent by the client when it wants to register or clear whisper targets.
+//
+// Note: The first available target ID is 1 as 0 is reserved for normal
+// talking. Maximum target ID is 30.
+message VoiceTarget {
+	message Target {
+		// Users that are included as targets.
+		repeated uint32 session = 1;
+		// Channel that is included as a target.
+		optional uint32 channel_id = 2;
+		// ACL group that is included as a target.
+		optional string group = 3;
+		// True if the voice should follow links from the specified channel.
+		optional bool links = 4 [default = false];
+		// True if the voice should also be sent to children of the specific
+		// channel.
+		optional bool children = 5 [default = false];
+	}
+	// Voice target ID.
+	optional uint32 id = 1;
+	// The receivers that this voice target includes.
+	repeated Target targets = 2;
+}
+
+// Sent by the client when it wants permissions for a certain channel. Sent by
+// the server when it replies to the query or wants the user to resync all
+// channel permissions.
+message PermissionQuery {
+	// channel_id of the channel for which the permissions are queried.
+	optional uint32 channel_id = 1;
+	// Channel permissions.
+	optional uint32 permissions = 2;
+	// True if the client should drop its current permission information for all
+	// channels.
+	optional bool flush = 3 [default = false];
+}
+
+// Sent by the server to notify the users of the version of the CELT codec they
+// should use. This may change during the connection when new users join.
+message CodecVersion {
+	// The version of the CELT Alpha codec.
+	required int32 alpha = 1;
+	// The version of the CELT Beta codec.
+	required int32 beta = 2;
+	// True if the user should prefer Alpha over Beta.
+	required bool prefer_alpha = 3 [default = true];
+	optional bool opus = 4 [default = false];
+}
+
+// Used to communicate user stats between the server and clients.
+message UserStats {
+	message Stats {
+		// The amount of good packets received.
+		optional uint32 good = 1;
+		// The amount of late packets received.
+		optional uint32 late = 2;
+		// The amount of packets never received.
+		optional uint32 lost = 3;
+		// The amount of nonce resyncs.
+		optional uint32 resync = 4;
+	}
+
+	// User whose stats these are.
+	optional uint32 session = 1;
+	// True if the message contains only mutable stats (packets, ping).
+	optional bool stats_only = 2 [default = false];
+	// Full user certificate chain of the user certificate in DER format.
+	repeated bytes certificates = 3;
+	// Packet statistics for packets received from the client.
+	optional Stats from_client = 4;
+	// Packet statistics for packets sent by the server.
+	optional Stats from_server = 5;
+
+	// Amount of UDP packets sent.
+	optional uint32 udp_packets = 6;
+	// Amount of TCP packets sent.
+	optional uint32 tcp_packets = 7;
+	// UDP ping average.
+	optional float udp_ping_avg = 8;
+	// UDP ping variance.
+	optional float udp_ping_var = 9;
+	// TCP ping average.
+	optional float tcp_ping_avg = 10;
+	// TCP ping variance.
+	optional float tcp_ping_var = 11;
+
+	// Client version.
+	optional Version version = 12;
+	// A list of CELT bitstream version constants supported by the client of this
+	// user.
+	repeated int32 celt_versions = 13;
+	// Client IP address.
+	optional bytes address = 14;
+	// Bandwith used by this client.
+	optional uint32 bandwidth = 15;
+	// Connection duration.
+	optional uint32 onlinesecs = 16;
+	// Duration since last activity.
+	optional uint32 idlesecs = 17;
+	// True if the user has a strong certificate.
+	optional bool strong_certificate = 18 [default = false];
+	optional bool opus = 19 [default = false];
+}
+
+// Used by the client to request binary data from the server. By default large
+// comments or textures are not sent within standard messages but instead the
+// hash is. If the client does not recognize the hash it may request the
+// resource when it needs it. The client does so by sending a RequestBlob
+// message with the correct fields filled with the user sessions or channel_ids
+// it wants to receive. The server replies to this by sending a new
+// UserState/ChannelState message with the resources filled even if they would
+// normally be transmitted as hashes.
+message RequestBlob {
+	// sessions of the requested UserState textures.
+	repeated uint32 session_texture = 1;
+	// sessions of the requested UserState comments.
+	repeated uint32 session_comment = 2;
+	// channel_ids of the requested ChannelState descriptions.
+	repeated uint32 channel_description = 3;
+}
+
+// Sent by the server when it informs the clients on server configuration
+// details.
+message ServerConfig {
+	// The maximum bandwidth the clients should use.
+	optional uint32 max_bandwidth = 1;
+	// Server welcome text.
+	optional string welcome_text = 2;
+	// True if the server allows HTML.
+	optional bool allow_html = 3;
+	// Maximum text message length.
+	optional uint32 message_length = 4;
+	// Maximum image message length.
+	optional uint32 image_message_length = 5;
+	// The maximum number of users allowed on the server.
+	optional uint32 max_users = 6;
+}
+
+// Sent by the server to inform the clients of suggested client configuration
+// specified by the server administrator.
+message SuggestConfig {
+	// Suggested client version.
+	optional uint32 version = 1;
+	// True if the administrator suggests positional audio to be used on this
+	// server.
+	optional bool positional = 2;
+	// True if the administrator suggests push to talk to be used on this server.
+	optional bool push_to_talk = 3;
+}

mumble/MurmurRPC.proto

@@ -0,0 +1,823 @@
+// Copyright 2005-2017 The Mumble Developers. All rights reserved.
+// Use of this source code is governed by a BSD-style license
+// that can be found in the LICENSE file at the root of the
+// Mumble source tree or at <https://www.mumble.info/LICENSE>.
+
+syntax = "proto2";
+
+package MurmurRPC;
+
+// Note about embedded messages:
+//
+// To help save bandwidth, the protocol does not always send complete embedded
+// messages (i.e. an embeddded message with all of the fields filled in). These
+// incomplete messages only contain enough identifying information to get more
+// information from the message's corresponding "Get" method. For example:
+//
+// User.server only ever contains the server ID. Calling ServerGet(User.server)
+// will return a Server message with the server's status and uptime.
+
+message Void {
+}
+
+message Version {
+	// 2-byte Major, 1-byte Minor and 1-byte Patch version number.
+	optional uint32 version = 1;
+	// Client release name.
+	optional string release = 2;
+	// Client OS name.
+	optional string os = 3;
+	// Client OS version.
+	optional string os_version = 4;
+}
+
+message Uptime {
+	// The number of seconds from the starting time.
+	optional uint64 secs = 1;
+}
+
+message Server {
+	// The unique server ID.
+	required uint32 id = 1;
+	// Is the server currently running?
+	optional bool running = 2;
+	// The update of the server.
+	optional Uptime uptime = 3;
+
+	message Event {
+		enum Type {
+			UserConnected = 0;
+			UserDisconnected = 1;
+			UserStateChanged = 2;
+			UserTextMessage = 3;
+			ChannelCreated = 4;
+			ChannelRemoved = 5;
+			ChannelStateChanged = 6;
+		};
+		// The server on which the event happened.
+		optional Server server = 1;
+		// The type of event that happened.
+		optional Type type = 2;
+		// The user tied to the event (if applicable).
+		optional User user = 3;
+		// The text message tied to the event (if applicable).
+		optional TextMessage message = 4;
+		// The channel tied to the event (if applicable).
+		optional Channel channel = 5;
+	}
+
+	message Query {
+	}
+
+	message List {
+		// The servers.
+		repeated Server servers = 1;
+	}
+}
+
+message Event {
+	enum Type {
+		ServerStopped = 0;
+		ServerStarted = 1;
+	};
+	// The server for which the event happened.
+	optional Server server = 1;
+	// The type of event that happened.
+	optional Type type = 2;
+}
+
+message ContextAction {
+	enum Context {
+		Server = 0x01;
+		Channel = 0x02;
+		User = 0x04;
+	};
+	// The server on which the action is.
+	optional Server server = 1;
+	// The context in which the action is.
+	optional uint32 context = 2;
+	// The action name.
+	optional string action = 3;
+	// The user-visible descriptive name of the action.
+	optional string text = 4;
+	// The user that triggered the ContextAction.
+	optional User actor = 5;
+	// The user on which the ContextAction was triggered.
+	optional User user = 6;
+	// The channel on which the ContextAction was triggered.
+	optional Channel channel = 7;
+}
+
+message TextMessage {
+	// The server on which the TextMessage originates.
+	optional Server server = 1;
+	// The user who sent the message.
+	optional User actor = 2;
+	// The users to whom the message is sent.
+	repeated User users = 3;
+	// The channels to which the message is sent.
+	repeated Channel channels = 4;
+	// The channels to which the message is sent, including the channels'
+	// ancestors.
+	repeated Channel trees = 5;
+	// The message body that is sent.
+	optional string text = 6;
+
+	message Filter {
+		enum Action {
+			// Accept the message.
+			Accept = 0;
+			// Reject the message with a permission error.
+			Reject = 1;
+			// Silently drop the message.
+			Drop = 2;
+		}
+		// The server on which the message originated.
+		optional Server server = 1;
+		// The action to perform for the message.
+		optional Action action = 2;
+		// The text message.
+		optional TextMessage message = 3;
+	}
+}
+
+message Log {
+	// The server on which the log message was generated.
+	optional Server server = 1;
+	// The unix timestamp of when the message was generated.
+	optional int64 timestamp = 2;
+	// The log message.
+	optional string text = 3;
+
+	message Query {
+		// The server whose logs will be queried.
+		optional Server server = 1;
+		// The minimum log index to receive.
+		optional uint32 min = 2;
+		// The maximum log index to receive.
+		optional uint32 max = 3;
+	}
+
+	message List {
+		// The server where the log entries are from.
+		optional Server server = 1;
+		// The total number of logs entries on the server.
+		optional uint32 total = 2;
+		// The minimum log index that was sent.
+		optional uint32 min = 3;
+		// The maximum log index that was sent.
+		optional uint32 max = 4;
+		// The log entries.
+		repeated Log entries = 5;
+	}
+}
+
+message Config {
+	// The server for which the configuration is for.
+	optional Server server = 1;
+	// The configuration keys and values.
+	map<string, string> fields = 2;
+
+	message Field {
+		// The server for which the configuration field is for.
+		optional Server server = 1;
+		// The field key.
+		optional string key = 2;
+		// The field value.
+		optional string value = 3;
+	}
+}
+
+message Channel {
+	// The server on which the channel exists.
+	optional Server server = 1;
+	// The unique channel identifier.
+	optional uint32 id = 2;
+	// The channel name.
+	optional string name = 3;
+	// The channel's parent.
+	optional Channel parent = 4;
+	// Linked channels.
+	repeated Channel links = 5;
+	// The channel's description.
+	optional string description = 6;
+	// Is the channel temporary?
+	optional bool temporary = 7;
+	// The position in which the channel should appear in a sorted list.
+	optional int32 position = 8;
+
+	message Query {
+		// The server on which the channels are.
+		optional Server server = 1;
+	}
+
+	message List {
+		// The server on which the channels are.
+		optional Server server = 1;
+		// The channels.
+		repeated Channel channels = 2;
+	}
+}
+
+message User {
+	// The server to which the user is connected.
+	optional Server server = 1;
+	// The user's session ID.
+	optional uint32 session = 2;
+	// The user's registered ID.
+	optional uint32 id = 3;
+	// The user's name.
+	optional string name = 4;
+	// Is the user muted?
+	optional bool mute = 5;
+	// Is the user deafened?
+	optional bool deaf = 6;
+	// Is the user suppressed?
+	optional bool suppress = 7;
+	// Is the user a priority speaker?
+	optional bool priority_speaker = 8;
+	// Has the user muted him/herself?
+	optional bool self_mute = 9;
+	// Has the user muted him/herself?
+	optional bool self_deaf = 10;
+	// Is the user recording?
+	optional bool recording = 11;
+	// The channel the user is in.
+	optional Channel channel = 12;
+	// How long the user has been connected to the server.
+	optional uint32 online_secs = 13;
+	// How long the user has been idle on the server.
+	optional uint32 idle_secs = 14;
+	// How many bytes per second is the user transmitting to the server.
+	optional uint32 bytes_per_sec = 15;
+	// The user's client version.
+	optional Version version = 16;
+	// The user's  plugin context.
+	optional bytes plugin_context = 17;
+	// The user's plugin identity.
+	optional string plugin_identity = 18;
+	// The user's comment.
+	optional string comment = 19;
+	// The user's texture.
+	optional bytes texture = 20;
+	// The user's IP address.
+	optional bytes address = 21;
+	// Is the user in TCP-only mode?
+	optional bool tcp_only = 22;
+	// The user's UDP ping in milliseconds.
+	optional float udp_ping_msecs = 23;
+	// The user's TCP ping in milliseconds.
+	optional float tcp_ping_msecs = 24;
+
+	message Query {
+		// The server whose users will be queried.
+		optional Server server = 1;
+	}
+
+	message List {
+		// The server to which the users are connected.
+		optional Server server = 1;
+		// The users.
+		repeated User users = 2;
+	}
+
+	message Kick {
+		// The server to which the user is connected.
+		optional Server server = 1;
+		// The user to kick.
+		optional User user = 2;
+		// The user who performed the kick.
+		optional User actor = 3;
+		// The reason for why the user is being kicked.
+		optional string reason = 4;
+	}
+}
+
+message Tree {
+	// The server which the tree represents.
+	optional Server server = 1;
+	// The current channel.
+	optional Channel channel = 2;
+	// Channels below the current channel.
+	repeated Tree children = 3;
+	// The users in the current channel.
+	repeated User users = 4;
+
+	message Query {
+		// The server to query.
+		optional Server server = 1;
+	}
+}
+
+message Ban {
+	// The server on which the ban is applied.
+	optional Server server = 1;
+	// The banned IP address.
+	optional bytes address = 2;
+	// The number of leading bits in the address to which the ban applies.
+	optional uint32 bits = 3;
+	// The name of the banned user.
+	optional string name = 4;
+	// The certificate hash of the banned user.
+	optional string hash = 5;
+	// The reason for the ban.
+	optional string reason = 6;
+	// The ban start time (in epoch form).
+	optional int64 start = 7;
+	// The ban duration.
+	optional int64 duration_secs = 8;
+
+	message Query {
+		// The server whose bans to query.
+		optional Server server = 1;
+	}
+
+	message List {
+		// The server for which the bans apply.
+		optional Server server = 1;
+		// The bans.
+		repeated Ban bans = 2;
+	}
+}
+
+message ACL {
+	enum Permission {
+		None = 0x00;
+		Write = 0x01;
+		Traverse = 0x02;
+		Enter = 0x04;
+		Speak = 0x08;
+		Whisper = 0x100;
+		MuteDeafen = 0x10;
+		Move = 0x20;
+		MakeChannel = 0x40;
+		MakeTemporaryChannel = 0x400;
+		LinkChannel = 0x80;
+		TextMessage = 0x200;
+
+		Kick = 0x10000;
+		Ban = 0x20000;
+		Register = 0x40000;
+		RegisterSelf = 0x80000;
+	}
+
+	message Group {
+		// The ACL group name.
+		optional string name = 1;
+		// Is the group inherited?
+		optional bool inherited = 2;
+		// Does the group inherit members?
+		optional bool inherit = 3;
+		// Can this group be inherited by its children?
+		optional bool inheritable = 4;
+
+		// The users explicitly added by this group.
+		repeated DatabaseUser users_add = 5;
+		// The users explicitly removed by this group.
+		repeated DatabaseUser users_remove = 6;
+		// All of the users who are part of this group.
+		repeated DatabaseUser users = 7;
+	}
+
+	// Does the ACL apply to the current channel?
+	optional bool apply_here = 3;
+	// Does the ACL apply to the current channel's sub-channels?
+	optional bool apply_subs = 4;
+	// Was the ACL inherited?
+	optional bool inherited = 5;
+
+	// The user to whom the ACL applies.
+	optional DatabaseUser user = 6;
+	// The group to whom the ACL applies.
+	optional ACL.Group group = 7;
+
+	// The permissions granted by the ACL (bitmask of ACL.Permission).
+	optional uint32 allow = 8;
+	// The permissions denied by the ACL (bitmask of ACL.Permission).
+	optional uint32 deny = 9;
+
+	message Query {
+		// The server where the user and channel exist.
+		optional Server server = 1;
+		// The user to query.
+		optional User user = 2;
+		// The channel to query.
+		optional Channel channel = 3;
+	}
+
+	message List {
+		// The server on which the ACLs exist.
+		optional Server server = 1;
+		// The channel to which the ACL refers.
+		optional Channel channel = 2;
+		// The ACLs part of the given channel.
+		repeated ACL acls = 3;
+		// The groups part of the given channel.
+		repeated ACL.Group groups = 4;
+		// Should ACLs be inherited from the parent channel.
+		optional bool inherit = 5;
+	}
+
+	message TemporaryGroup {
+		// The server where the temporary group exists.
+		optional Server server = 1;
+		// The channel to which the temporary user group is added.
+		optional Channel channel = 2;
+		// The user who is added to the group.
+		optional User user = 3;
+		// The name of the temporary group.
+		optional string name = 4;
+	}
+}
+
+message Authenticator {
+	message Request {
+		// An authentication request for a connecting user.
+		message Authenticate {
+			// The user's name.
+			optional string name = 1;
+			// The user's password.
+			optional string password = 2;
+			// The user's certificate chain in DER format.
+			repeated bytes certificates = 3;
+			// The hexadecimal hash of the user's certificate.
+			optional string certificate_hash = 4;
+			// If the user is connecting with a strong certificate.
+			optional bool strong_certificate = 5;
+		}
+
+		// A request for information about a user, given by either the user's ID
+		// or name.
+		message Find {
+			// The user's ID used for lookup.
+			optional uint32 id = 1;
+			// The user's name used for lookup.
+			optional string name = 2;
+		}
+
+		// A query of all the registered users, optionally filtered by the given
+		// filter string.
+		message Query {
+			// A user name filter (% is often used as a wildcard)
+			optional string filter = 1;
+		}
+
+		// A request for a new user registration.
+		message Register {
+			// The database user to register.
+			optional DatabaseUser user = 1;
+		}
+
+		// A request for deregistering a registered user.
+		message Deregister {
+			// The database user to deregister.
+			optional DatabaseUser user = 1;
+		}
+
+		// A request to update a registered user's information. The information
+		// provided should be merged with existing data.
+		message Update {
+			// The database user to update.
+			optional DatabaseUser user = 1;
+		}
+
+		optional Authenticate authenticate = 1;
+		optional Find find = 2;
+		optional Query query = 3;
+		optional Register register = 4;
+		optional Deregister deregister = 5;
+		optional Update update = 6;
+	}
+
+	message Response {
+		// The initialization for the authenticator stream. This message must be
+		// sent before authentication requests will start streaming.
+		message Initialize {
+			optional Server server = 1;
+		}
+
+		enum Status {
+			// The request should fallthrough to murmur's default action.
+			Fallthrough = 0;
+			// The request was successful.
+			Success = 1;
+			// The request failed; there was some error.
+			Failure = 2;
+			// A temporary failure prevented the request from succeeding (e.g. a
+			// database was unavailable).
+			TemporaryFailure = 3;
+		}
+
+		message Authenticate {
+			// The status of the request.
+			optional Status status = 1;
+			// The user's registered ID.
+			optional uint32 id = 2;
+			// The corrected user's name;
+			optional string name = 3;
+			// Additional ACL groups that the user belongs too.
+			repeated ACL.Group groups = 4;
+		}
+
+		message Find {
+			// The database user (if found).
+			optional DatabaseUser user = 1;
+		}
+
+		message Query {
+			// The matched database users.
+			repeated DatabaseUser users = 1;
+		}
+
+		message Register {
+			// The status of the request.
+			optional Status status = 1;
+			// The registered database user (must contain the registered user's ID).
+			optional DatabaseUser user = 2;
+		}
+
+		message Deregister {
+			// The status of the request.
+			optional Status status = 1;
+		}
+
+		message Update {
+			// The status of the request.
+			optional Status status = 1;
+		}
+
+		optional Initialize initialize = 1;
+		optional Authenticate authenticate = 2;
+		optional Find find = 3;
+		optional Query query = 4;
+		optional Register register = 5;
+		optional Deregister deregister = 6;
+		optional Update update = 7;
+	}
+}
+
+message DatabaseUser {
+	// The server on which the user is registered.
+	optional Server server = 1;
+	// The unique user ID.
+	optional uint32 id = 2;
+	// The user's name.
+	optional string name = 3;
+	// The user's email address.
+	optional string email = 4;
+	// The user's comment.
+	optional string comment = 5;
+	// The user's certificate hash.
+	optional string hash = 6;
+	// The user's password (never sent; used only when updating).
+	optional string password = 7;
+	// When the user was last on the server.
+	optional string last_active = 8;
+	// The user's texture.
+	optional bytes texture = 9;
+
+	message Query {
+		// The server whose users will be queried.
+		optional Server server = 1;
+		// A string to filter the users by.
+		optional string filter = 2;
+	}
+
+	message List {
+		// The server on which the users are registered.
+		optional Server server = 1;
+		// The users.
+		repeated DatabaseUser users = 2;
+	}
+
+	message Verify {
+		// The server on which the user-password pair will be authenticated.
+		optional Server server = 1;
+		// The user's name.
+		optional string name = 2;
+		// The user's password.
+		optional string password  = 3;
+	}
+}
+
+message RedirectWhisperGroup {
+	// The server on which the whisper redirection will take place.
+	optional Server server = 1;
+	// The user to whom the redirection will be applied.
+	optional User user = 2;
+	// The source group.
+	optional ACL.Group source = 3;
+	// The target group.
+	optional ACL.Group target = 4;
+}
+
+service V1 {
+	//
+	// Meta
+	//
+
+	// GetUptime returns murmur's uptime.
+	rpc GetUptime(Void) returns(Uptime);
+	// GetVersion returns murmur's version.
+	rpc GetVersion(Void) returns(Version);
+	// Events returns a stream of murmur events.
+	rpc Events(Void) returns(stream Event);
+
+	//
+	// Servers
+	//
+
+	// ServerCreate creates a new virtual server. The returned server object
+	// contains the newly created server's ID.
+	rpc ServerCreate(Void) returns(Server);
+	// ServerQuery returns a list of servers that match the given query.
+	rpc ServerQuery(Server.Query) returns(Server.List);
+	// ServerGet returns information about the given server.
+	rpc ServerGet(Server) returns(Server);
+	// ServerStart starts the given stopped server.
+	rpc ServerStart(Server) returns(Void);
+	// ServerStop stops the given virtual server.
+	rpc ServerStop(Server) returns(Void);
+	// ServerRemove removes the given virtual server and its configuration.
+	rpc ServerRemove(Server) returns(Void);
+	// ServerEvents returns a stream of events that happen on the given server.
+	rpc ServerEvents(Server) returns(stream Server.Event);
+
+	//
+	// ContextActions
+	//
+
+	// ContextActionAdd adds a context action to the given user's client. The
+	// following ContextAction fields must be set:
+	//   context, action, text, and user.
+	//
+	// Added context actions are valid until:
+	//  - The context action is removed with ContextActionRemove, or
+	//  - The user disconnects from the server, or
+	//  - The server stops.
+	rpc ContextActionAdd(ContextAction) returns(Void);
+	// ContextActionRemove removes a context action from the given user's client.
+	// The following ContextAction must be set:
+	//   action
+	// If no user is given, the context action is removed from all users.
+	rpc ContextActionRemove(ContextAction) returns(Void);
+	// ContextActionEvents returns a stream of context action events that are
+	// triggered by users.
+	rpc ContextActionEvents(ContextAction) returns(stream ContextAction);
+
+	//
+	// TextMessage
+	//
+
+	// TextMessageSend sends the given TextMessage to the server.
+	//
+	// If no users, channels, or trees are added to the TextMessage, the message
+	// will be broadcast the entire server. Otherwise, the message will be
+	// targeted to the specified users, channels, and trees.
+	rpc TextMessageSend(TextMessage) returns(Void);
+	// TextMessageFilter filters text messages on a given server.
+
+	// TextMessageFilter filters text messages for a given server.
+	//
+	// When a filter stream is active, text messages sent from users to the
+	// server are sent over the stream. The RPC client then sends a message back
+	// on the same stream, containing an action: whether the message should be
+	// accepted, rejected, or dropped.
+	//
+	// To activate the filter stream, an initial TextMessage.Filter message must
+	// be sent that contains the server on which the filter will be active.
+	rpc TextMessageFilter(stream TextMessage.Filter) returns(stream TextMessage.Filter);
+
+	//
+	// Logs
+	//
+
+	// LogQuery returns a list of log entries from the given server.
+	//
+	// To get the total number of log entries, omit min and/or max from the
+	// query.
+	rpc LogQuery(Log.Query) returns(Log.List);
+
+	//
+	// Config
+	//
+
+	// ConfigGet returns the explicitly set configuration for the given server.
+	rpc ConfigGet(Server) returns(Config);
+	// ConfigGetField returns the configuration value for the given key.
+	rpc ConfigGetField(Config.Field) returns(Config.Field);
+	// ConfigSetField sets the configuration value to the given value.
+	rpc ConfigSetField(Config.Field) returns(Void);
+	// ConfigGetDefault returns the default server configuration.
+	rpc ConfigGetDefault(Void) returns(Config);
+
+	//
+	// Channels
+	//
+
+	// ChannelQuery returns a list of channels that match the given query.
+	rpc ChannelQuery(Channel.Query) returns(Channel.List);
+	// ChannelGet returns the channel with the given ID.
+	rpc ChannelGet(Channel) returns(Channel);
+	// ChannelAdd adds the channel to the given server. The parent and name of
+	// the channel must be set.
+	rpc ChannelAdd(Channel) returns(Channel);
+	// ChannelRemove removes the given channel from the server.
+	rpc ChannelRemove(Channel) returns(Void);
+	// ChannelUpdate updates the given channel's attributes. Only the fields that
+	// are set will be updated.
+	rpc ChannelUpdate(Channel) returns(Channel);
+
+	//
+	// Users
+	//
+
+	// UserQuery returns a list of connected users who match the given query.
+	rpc UserQuery(User.Query) returns(User.List);
+	// UserGet returns information on the connected user, given by the user's
+	// session or name.
+	rpc UserGet(User) returns(User);
+	// UserUpdate changes the given user's state. Only the following fields can
+	// be changed:
+	//   name, mute, deaf, suppress, priority_speaker, channel, comment.
+	rpc UserUpdate(User) returns(User);
+	// UserKick kicks the user from the server.
+	rpc UserKick(User.Kick) returns(Void);
+
+	//
+	// Tree
+	//
+
+	// TreeQuery returns a representation of the given server's channel/user
+	// tree.
+	rpc TreeQuery(Tree.Query) returns(Tree);
+
+	//
+	// Bans
+	//
+
+	// BansGet returns a list of bans for the given server.
+	rpc BansGet(Ban.Query) returns(Ban.List);
+	// BansSet replaces the server's ban list with the given list.
+	rpc BansSet(Ban.List) returns(Void);
+
+	//
+	// ACL
+	//
+
+	// ACLGet returns the ACL for the given channel.
+	rpc ACLGet(Channel) returns(ACL.List);
+	// ACLSet overrides the ACL of the given channel to what is provided.
+	rpc ACLSet(ACL.List) returns(Void);
+	// ACLGetEffectivePermissions returns the effective permissions for the given
+	// user in the given channel.
+	rpc ACLGetEffectivePermissions(ACL.Query) returns(ACL);
+	// ACLAddTemporaryGroup adds a user to a temporary group.
+	rpc ACLAddTemporaryGroup(ACL.TemporaryGroup) returns(Void);
+	// ACLRemoveTemporaryGroup removes a user from a temporary group.
+	rpc ACLRemoveTemporaryGroup(ACL.TemporaryGroup) returns(Void);
+
+	//
+	// Authenticator
+	//
+
+	// AuthenticatorStream opens an authentication stream to the server.
+	//
+	// There can only be one RPC client with an open Stream. If a new
+	// authenticator connects, the open connected will be closed.
+	rpc AuthenticatorStream(stream Authenticator.Response) returns(stream Authenticator.Request);
+
+	//
+	// Database
+	//
+
+	// DatabaseUserQuery returns a list of registered users who match given
+	// query.
+	rpc DatabaseUserQuery(DatabaseUser.Query) returns(DatabaseUser.List);
+	// DatabaseUserGet returns the database user with the given ID.
+	rpc DatabaseUserGet(DatabaseUser) returns(DatabaseUser);
+	// DatabaseUserUpdate updates the given database user.
+	rpc DatabaseUserUpdate(DatabaseUser) returns(Void);
+	// DatabaseUserRegister registers a user with the given information on the
+	// server. The returned DatabaseUser will contain the newly registered user's
+	// ID.
+	rpc DatabaseUserRegister(DatabaseUser) returns(DatabaseUser);
+	// DatabaseUserDeregister deregisters the given user.
+	rpc DatabaseUserDeregister(DatabaseUser) returns(Void);
+	// DatabaseUserVerify verifies the that the given user-password pair is
+	// correct.
+	rpc DatabaseUserVerify(DatabaseUser.Verify) returns(DatabaseUser);
+
+	//
+	// Audio
+	//
+
+	// AddRedirectWhisperGroup add a whisper targets redirection for the given
+	// user. Whenever a user whispers to group "source", the whisper will be
+	// redirected to group "target".
+	rpc RedirectWhisperGroupAdd(RedirectWhisperGroup) returns(Void);
+
+	// RemoveRedirectWhisperGroup removes a whisper target redirection for
+	// the the given user.
+	rpc RedirectWhisperGroupRemove(RedirectWhisperGroup) returns(Void);
+}

mumble/MurmurRPC_pb2_grpc.py

@@ -0,0 +1,912 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+import MurmurRPC_pb2 as MurmurRPC__pb2
+
+
+class V1Stub(object):
+  """
+  Meta
+
+  """
+
+  def __init__(self, channel):
+    """Constructor.
+
+    Args:
+      channel: A grpc.Channel.
+    """
+    self.GetUptime = channel.unary_unary(
+        '/MurmurRPC.V1/GetUptime',
+        request_serializer=MurmurRPC__pb2.Void.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Uptime.FromString,
+        )
+    self.GetVersion = channel.unary_unary(
+        '/MurmurRPC.V1/GetVersion',
+        request_serializer=MurmurRPC__pb2.Void.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Version.FromString,
+        )
+    self.Events = channel.unary_stream(
+        '/MurmurRPC.V1/Events',
+        request_serializer=MurmurRPC__pb2.Void.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Event.FromString,
+        )
+    self.ServerCreate = channel.unary_unary(
+        '/MurmurRPC.V1/ServerCreate',
+        request_serializer=MurmurRPC__pb2.Void.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Server.FromString,
+        )
+    self.ServerQuery = channel.unary_unary(
+        '/MurmurRPC.V1/ServerQuery',
+        request_serializer=MurmurRPC__pb2.Server.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Server.List.FromString,
+        )
+    self.ServerGet = channel.unary_unary(
+        '/MurmurRPC.V1/ServerGet',
+        request_serializer=MurmurRPC__pb2.Server.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Server.FromString,
+        )
+    self.ServerStart = channel.unary_unary(
+        '/MurmurRPC.V1/ServerStart',
+        request_serializer=MurmurRPC__pb2.Server.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ServerStop = channel.unary_unary(
+        '/MurmurRPC.V1/ServerStop',
+        request_serializer=MurmurRPC__pb2.Server.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ServerRemove = channel.unary_unary(
+        '/MurmurRPC.V1/ServerRemove',
+        request_serializer=MurmurRPC__pb2.Server.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ServerEvents = channel.unary_stream(
+        '/MurmurRPC.V1/ServerEvents',
+        request_serializer=MurmurRPC__pb2.Server.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Server.Event.FromString,
+        )
+    self.ContextActionAdd = channel.unary_unary(
+        '/MurmurRPC.V1/ContextActionAdd',
+        request_serializer=MurmurRPC__pb2.ContextAction.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ContextActionRemove = channel.unary_unary(
+        '/MurmurRPC.V1/ContextActionRemove',
+        request_serializer=MurmurRPC__pb2.ContextAction.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ContextActionEvents = channel.unary_stream(
+        '/MurmurRPC.V1/ContextActionEvents',
+        request_serializer=MurmurRPC__pb2.ContextAction.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.ContextAction.FromString,
+        )
+    self.TextMessageSend = channel.unary_unary(
+        '/MurmurRPC.V1/TextMessageSend',
+        request_serializer=MurmurRPC__pb2.TextMessage.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.TextMessageFilter = channel.stream_stream(
+        '/MurmurRPC.V1/TextMessageFilter',
+        request_serializer=MurmurRPC__pb2.TextMessage.Filter.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.TextMessage.Filter.FromString,
+        )
+    self.LogQuery = channel.unary_unary(
+        '/MurmurRPC.V1/LogQuery',
+        request_serializer=MurmurRPC__pb2.Log.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Log.List.FromString,
+        )
+    self.ConfigGet = channel.unary_unary(
+        '/MurmurRPC.V1/ConfigGet',
+        request_serializer=MurmurRPC__pb2.Server.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Config.FromString,
+        )
+    self.ConfigGetField = channel.unary_unary(
+        '/MurmurRPC.V1/ConfigGetField',
+        request_serializer=MurmurRPC__pb2.Config.Field.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Config.Field.FromString,
+        )
+    self.ConfigSetField = channel.unary_unary(
+        '/MurmurRPC.V1/ConfigSetField',
+        request_serializer=MurmurRPC__pb2.Config.Field.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ConfigGetDefault = channel.unary_unary(
+        '/MurmurRPC.V1/ConfigGetDefault',
+        request_serializer=MurmurRPC__pb2.Void.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Config.FromString,
+        )
+    self.ChannelQuery = channel.unary_unary(
+        '/MurmurRPC.V1/ChannelQuery',
+        request_serializer=MurmurRPC__pb2.Channel.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Channel.List.FromString,
+        )
+    self.ChannelGet = channel.unary_unary(
+        '/MurmurRPC.V1/ChannelGet',
+        request_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Channel.FromString,
+        )
+    self.ChannelAdd = channel.unary_unary(
+        '/MurmurRPC.V1/ChannelAdd',
+        request_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Channel.FromString,
+        )
+    self.ChannelRemove = channel.unary_unary(
+        '/MurmurRPC.V1/ChannelRemove',
+        request_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ChannelUpdate = channel.unary_unary(
+        '/MurmurRPC.V1/ChannelUpdate',
+        request_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Channel.FromString,
+        )
+    self.UserQuery = channel.unary_unary(
+        '/MurmurRPC.V1/UserQuery',
+        request_serializer=MurmurRPC__pb2.User.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.User.List.FromString,
+        )
+    self.UserGet = channel.unary_unary(
+        '/MurmurRPC.V1/UserGet',
+        request_serializer=MurmurRPC__pb2.User.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.User.FromString,
+        )
+    self.UserUpdate = channel.unary_unary(
+        '/MurmurRPC.V1/UserUpdate',
+        request_serializer=MurmurRPC__pb2.User.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.User.FromString,
+        )
+    self.UserKick = channel.unary_unary(
+        '/MurmurRPC.V1/UserKick',
+        request_serializer=MurmurRPC__pb2.User.Kick.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.TreeQuery = channel.unary_unary(
+        '/MurmurRPC.V1/TreeQuery',
+        request_serializer=MurmurRPC__pb2.Tree.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Tree.FromString,
+        )
+    self.BansGet = channel.unary_unary(
+        '/MurmurRPC.V1/BansGet',
+        request_serializer=MurmurRPC__pb2.Ban.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Ban.List.FromString,
+        )
+    self.BansSet = channel.unary_unary(
+        '/MurmurRPC.V1/BansSet',
+        request_serializer=MurmurRPC__pb2.Ban.List.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ACLGet = channel.unary_unary(
+        '/MurmurRPC.V1/ACLGet',
+        request_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.ACL.List.FromString,
+        )
+    self.ACLSet = channel.unary_unary(
+        '/MurmurRPC.V1/ACLSet',
+        request_serializer=MurmurRPC__pb2.ACL.List.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ACLGetEffectivePermissions = channel.unary_unary(
+        '/MurmurRPC.V1/ACLGetEffectivePermissions',
+        request_serializer=MurmurRPC__pb2.ACL.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.ACL.FromString,
+        )
+    self.ACLAddTemporaryGroup = channel.unary_unary(
+        '/MurmurRPC.V1/ACLAddTemporaryGroup',
+        request_serializer=MurmurRPC__pb2.ACL.TemporaryGroup.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.ACLRemoveTemporaryGroup = channel.unary_unary(
+        '/MurmurRPC.V1/ACLRemoveTemporaryGroup',
+        request_serializer=MurmurRPC__pb2.ACL.TemporaryGroup.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.AuthenticatorStream = channel.stream_stream(
+        '/MurmurRPC.V1/AuthenticatorStream',
+        request_serializer=MurmurRPC__pb2.Authenticator.Response.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Authenticator.Request.FromString,
+        )
+    self.DatabaseUserQuery = channel.unary_unary(
+        '/MurmurRPC.V1/DatabaseUserQuery',
+        request_serializer=MurmurRPC__pb2.DatabaseUser.Query.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.DatabaseUser.List.FromString,
+        )
+    self.DatabaseUserGet = channel.unary_unary(
+        '/MurmurRPC.V1/DatabaseUserGet',
+        request_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+        )
+    self.DatabaseUserUpdate = channel.unary_unary(
+        '/MurmurRPC.V1/DatabaseUserUpdate',
+        request_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.DatabaseUserRegister = channel.unary_unary(
+        '/MurmurRPC.V1/DatabaseUserRegister',
+        request_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+        )
+    self.DatabaseUserDeregister = channel.unary_unary(
+        '/MurmurRPC.V1/DatabaseUserDeregister',
+        request_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.DatabaseUserVerify = channel.unary_unary(
+        '/MurmurRPC.V1/DatabaseUserVerify',
+        request_serializer=MurmurRPC__pb2.DatabaseUser.Verify.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+        )
+    self.RedirectWhisperGroupAdd = channel.unary_unary(
+        '/MurmurRPC.V1/RedirectWhisperGroupAdd',
+        request_serializer=MurmurRPC__pb2.RedirectWhisperGroup.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+    self.RedirectWhisperGroupRemove = channel.unary_unary(
+        '/MurmurRPC.V1/RedirectWhisperGroupRemove',
+        request_serializer=MurmurRPC__pb2.RedirectWhisperGroup.SerializeToString,
+        response_deserializer=MurmurRPC__pb2.Void.FromString,
+        )
+
+
+class V1Servicer(object):
+  """
+  Meta
+
+  """
+
+  def GetUptime(self, request, context):
+    """GetUptime returns murmur's uptime.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def GetVersion(self, request, context):
+    """GetVersion returns murmur's version.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def Events(self, request, context):
+    """Events returns a stream of murmur events.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerCreate(self, request, context):
+    """
+    Servers
+
+
+    ServerCreate creates a new virtual server. The returned server object
+    contains the newly created server's ID.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerQuery(self, request, context):
+    """ServerQuery returns a list of servers that match the given query.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerGet(self, request, context):
+    """ServerGet returns information about the given server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerStart(self, request, context):
+    """ServerStart starts the given stopped server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerStop(self, request, context):
+    """ServerStop stops the given virtual server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerRemove(self, request, context):
+    """ServerRemove removes the given virtual server and its configuration.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ServerEvents(self, request, context):
+    """ServerEvents returns a stream of events that happen on the given server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ContextActionAdd(self, request, context):
+    """
+    ContextActions
+
+
+    ContextActionAdd adds a context action to the given user's client. The
+    following ContextAction fields must be set:
+    context, action, text, and user.
+
+    Added context actions are valid until:
+    - The context action is removed with ContextActionRemove, or
+    - The user disconnects from the server, or
+    - The server stops.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ContextActionRemove(self, request, context):
+    """ContextActionRemove removes a context action from the given user's client.
+    The following ContextAction must be set:
+    action
+    If no user is given, the context action is removed from all users.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ContextActionEvents(self, request, context):
+    """ContextActionEvents returns a stream of context action events that are
+    triggered by users.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def TextMessageSend(self, request, context):
+    """
+    TextMessage
+
+
+    TextMessageSend sends the given TextMessage to the server.
+
+    If no users, channels, or trees are added to the TextMessage, the message
+    will be broadcast the entire server. Otherwise, the message will be
+    targeted to the specified users, channels, and trees.
+    TextMessageFilter filters text messages on a given server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def TextMessageFilter(self, request_iterator, context):
+    """TextMessageFilter filters text messages for a given server.
+
+    When a filter stream is active, text messages sent from users to the
+    server are sent over the stream. The RPC client then sends a message back
+    on the same stream, containing an action: whether the message should be
+    accepted, rejected, or dropped.
+
+    To activate the filter stream, an initial TextMessage.Filter message must
+    be sent that contains the server on which the filter will be active.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def LogQuery(self, request, context):
+    """
+    Logs
+
+
+    LogQuery returns a list of log entries from the given server.
+
+    To get the total number of log entries, omit min and/or max from the
+    query.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ConfigGet(self, request, context):
+    """
+    Config
+
+
+    ConfigGet returns the explicitly set configuration for the given server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ConfigGetField(self, request, context):
+    """ConfigGetField returns the configuration value for the given key.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ConfigSetField(self, request, context):
+    """ConfigSetField sets the configuration value to the given value.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ConfigGetDefault(self, request, context):
+    """ConfigGetDefault returns the default server configuration.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ChannelQuery(self, request, context):
+    """
+    Channels
+
+
+    ChannelQuery returns a list of channels that match the given query.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ChannelGet(self, request, context):
+    """ChannelGet returns the channel with the given ID.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ChannelAdd(self, request, context):
+    """ChannelAdd adds the channel to the given server. The parent and name of
+    the channel must be set.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ChannelRemove(self, request, context):
+    """ChannelRemove removes the given channel from the server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ChannelUpdate(self, request, context):
+    """ChannelUpdate updates the given channel's attributes. Only the fields that
+    are set will be updated.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def UserQuery(self, request, context):
+    """
+    Users
+
+
+    UserQuery returns a list of connected users who match the given query.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def UserGet(self, request, context):
+    """UserGet returns information on the connected user, given by the user's
+    session or name.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def UserUpdate(self, request, context):
+    """UserUpdate changes the given user's state. Only the following fields can
+    be changed:
+    name, mute, deaf, suppress, priority_speaker, channel, comment.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def UserKick(self, request, context):
+    """UserKick kicks the user from the server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def TreeQuery(self, request, context):
+    """
+    Tree
+
+
+    TreeQuery returns a representation of the given server's channel/user
+    tree.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def BansGet(self, request, context):
+    """
+    Bans
+
+
+    BansGet returns a list of bans for the given server.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def BansSet(self, request, context):
+    """BansSet replaces the server's ban list with the given list.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ACLGet(self, request, context):
+    """
+    ACL
+
+
+    ACLGet returns the ACL for the given channel.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ACLSet(self, request, context):
+    """ACLSet overrides the ACL of the given channel to what is provided.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ACLGetEffectivePermissions(self, request, context):
+    """ACLGetEffectivePermissions returns the effective permissions for the given
+    user in the given channel.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ACLAddTemporaryGroup(self, request, context):
+    """ACLAddTemporaryGroup adds a user to a temporary group.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def ACLRemoveTemporaryGroup(self, request, context):
+    """ACLRemoveTemporaryGroup removes a user from a temporary group.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def AuthenticatorStream(self, request_iterator, context):
+    """
+    Authenticator
+
+
+    AuthenticatorStream opens an authentication stream to the server.
+
+    There can only be one RPC client with an open Stream. If a new
+    authenticator connects, the open connected will be closed.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def DatabaseUserQuery(self, request, context):
+    """
+    Database
+
+
+    DatabaseUserQuery returns a list of registered users who match given
+    query.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def DatabaseUserGet(self, request, context):
+    """DatabaseUserGet returns the database user with the given ID.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def DatabaseUserUpdate(self, request, context):
+    """DatabaseUserUpdate updates the given database user.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def DatabaseUserRegister(self, request, context):
+    """DatabaseUserRegister registers a user with the given information on the
+    server. The returned DatabaseUser will contain the newly registered user's
+    ID.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def DatabaseUserDeregister(self, request, context):
+    """DatabaseUserDeregister deregisters the given user.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def DatabaseUserVerify(self, request, context):
+    """DatabaseUserVerify verifies the that the given user-password pair is
+    correct.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def RedirectWhisperGroupAdd(self, request, context):
+    """
+    Audio
+
+
+    AddRedirectWhisperGroup add a whisper targets redirection for the given
+    user. Whenever a user whispers to group "source", the whisper will be
+    redirected to group "target".
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+  def RedirectWhisperGroupRemove(self, request, context):
+    """RemoveRedirectWhisperGroup removes a whisper target redirection for
+    the the given user.
+    """
+    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+    context.set_details('Method not implemented!')
+    raise NotImplementedError('Method not implemented!')
+
+
+def add_V1Servicer_to_server(servicer, server):
+  rpc_method_handlers = {
+      'GetUptime': grpc.unary_unary_rpc_method_handler(
+          servicer.GetUptime,
+          request_deserializer=MurmurRPC__pb2.Void.FromString,
+          response_serializer=MurmurRPC__pb2.Uptime.SerializeToString,
+      ),
+      'GetVersion': grpc.unary_unary_rpc_method_handler(
+          servicer.GetVersion,
+          request_deserializer=MurmurRPC__pb2.Void.FromString,
+          response_serializer=MurmurRPC__pb2.Version.SerializeToString,
+      ),
+      'Events': grpc.unary_stream_rpc_method_handler(
+          servicer.Events,
+          request_deserializer=MurmurRPC__pb2.Void.FromString,
+          response_serializer=MurmurRPC__pb2.Event.SerializeToString,
+      ),
+      'ServerCreate': grpc.unary_unary_rpc_method_handler(
+          servicer.ServerCreate,
+          request_deserializer=MurmurRPC__pb2.Void.FromString,
+          response_serializer=MurmurRPC__pb2.Server.SerializeToString,
+      ),
+      'ServerQuery': grpc.unary_unary_rpc_method_handler(
+          servicer.ServerQuery,
+          request_deserializer=MurmurRPC__pb2.Server.Query.FromString,
+          response_serializer=MurmurRPC__pb2.Server.List.SerializeToString,
+      ),
+      'ServerGet': grpc.unary_unary_rpc_method_handler(
+          servicer.ServerGet,
+          request_deserializer=MurmurRPC__pb2.Server.FromString,
+          response_serializer=MurmurRPC__pb2.Server.SerializeToString,
+      ),
+      'ServerStart': grpc.unary_unary_rpc_method_handler(
+          servicer.ServerStart,
+          request_deserializer=MurmurRPC__pb2.Server.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ServerStop': grpc.unary_unary_rpc_method_handler(
+          servicer.ServerStop,
+          request_deserializer=MurmurRPC__pb2.Server.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ServerRemove': grpc.unary_unary_rpc_method_handler(
+          servicer.ServerRemove,
+          request_deserializer=MurmurRPC__pb2.Server.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ServerEvents': grpc.unary_stream_rpc_method_handler(
+          servicer.ServerEvents,
+          request_deserializer=MurmurRPC__pb2.Server.FromString,
+          response_serializer=MurmurRPC__pb2.Server.Event.SerializeToString,
+      ),
+      'ContextActionAdd': grpc.unary_unary_rpc_method_handler(
+          servicer.ContextActionAdd,
+          request_deserializer=MurmurRPC__pb2.ContextAction.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ContextActionRemove': grpc.unary_unary_rpc_method_handler(
+          servicer.ContextActionRemove,
+          request_deserializer=MurmurRPC__pb2.ContextAction.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ContextActionEvents': grpc.unary_stream_rpc_method_handler(
+          servicer.ContextActionEvents,
+          request_deserializer=MurmurRPC__pb2.ContextAction.FromString,
+          response_serializer=MurmurRPC__pb2.ContextAction.SerializeToString,
+      ),
+      'TextMessageSend': grpc.unary_unary_rpc_method_handler(
+          servicer.TextMessageSend,
+          request_deserializer=MurmurRPC__pb2.TextMessage.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'TextMessageFilter': grpc.stream_stream_rpc_method_handler(
+          servicer.TextMessageFilter,
+          request_deserializer=MurmurRPC__pb2.TextMessage.Filter.FromString,
+          response_serializer=MurmurRPC__pb2.TextMessage.Filter.SerializeToString,
+      ),
+      'LogQuery': grpc.unary_unary_rpc_method_handler(
+          servicer.LogQuery,
+          request_deserializer=MurmurRPC__pb2.Log.Query.FromString,
+          response_serializer=MurmurRPC__pb2.Log.List.SerializeToString,
+      ),
+      'ConfigGet': grpc.unary_unary_rpc_method_handler(
+          servicer.ConfigGet,
+          request_deserializer=MurmurRPC__pb2.Server.FromString,
+          response_serializer=MurmurRPC__pb2.Config.SerializeToString,
+      ),
+      'ConfigGetField': grpc.unary_unary_rpc_method_handler(
+          servicer.ConfigGetField,
+          request_deserializer=MurmurRPC__pb2.Config.Field.FromString,
+          response_serializer=MurmurRPC__pb2.Config.Field.SerializeToString,
+      ),
+      'ConfigSetField': grpc.unary_unary_rpc_method_handler(
+          servicer.ConfigSetField,
+          request_deserializer=MurmurRPC__pb2.Config.Field.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ConfigGetDefault': grpc.unary_unary_rpc_method_handler(
+          servicer.ConfigGetDefault,
+          request_deserializer=MurmurRPC__pb2.Void.FromString,
+          response_serializer=MurmurRPC__pb2.Config.SerializeToString,
+      ),
+      'ChannelQuery': grpc.unary_unary_rpc_method_handler(
+          servicer.ChannelQuery,
+          request_deserializer=MurmurRPC__pb2.Channel.Query.FromString,
+          response_serializer=MurmurRPC__pb2.Channel.List.SerializeToString,
+      ),
+      'ChannelGet': grpc.unary_unary_rpc_method_handler(
+          servicer.ChannelGet,
+          request_deserializer=MurmurRPC__pb2.Channel.FromString,
+          response_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+      ),
+      'ChannelAdd': grpc.unary_unary_rpc_method_handler(
+          servicer.ChannelAdd,
+          request_deserializer=MurmurRPC__pb2.Channel.FromString,
+          response_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+      ),
+      'ChannelRemove': grpc.unary_unary_rpc_method_handler(
+          servicer.ChannelRemove,
+          request_deserializer=MurmurRPC__pb2.Channel.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ChannelUpdate': grpc.unary_unary_rpc_method_handler(
+          servicer.ChannelUpdate,
+          request_deserializer=MurmurRPC__pb2.Channel.FromString,
+          response_serializer=MurmurRPC__pb2.Channel.SerializeToString,
+      ),
+      'UserQuery': grpc.unary_unary_rpc_method_handler(
+          servicer.UserQuery,
+          request_deserializer=MurmurRPC__pb2.User.Query.FromString,
+          response_serializer=MurmurRPC__pb2.User.List.SerializeToString,
+      ),
+      'UserGet': grpc.unary_unary_rpc_method_handler(
+          servicer.UserGet,
+          request_deserializer=MurmurRPC__pb2.User.FromString,
+          response_serializer=MurmurRPC__pb2.User.SerializeToString,
+      ),
+      'UserUpdate': grpc.unary_unary_rpc_method_handler(
+          servicer.UserUpdate,
+          request_deserializer=MurmurRPC__pb2.User.FromString,
+          response_serializer=MurmurRPC__pb2.User.SerializeToString,
+      ),
+      'UserKick': grpc.unary_unary_rpc_method_handler(
+          servicer.UserKick,
+          request_deserializer=MurmurRPC__pb2.User.Kick.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'TreeQuery': grpc.unary_unary_rpc_method_handler(
+          servicer.TreeQuery,
+          request_deserializer=MurmurRPC__pb2.Tree.Query.FromString,
+          response_serializer=MurmurRPC__pb2.Tree.SerializeToString,
+      ),
+      'BansGet': grpc.unary_unary_rpc_method_handler(
+          servicer.BansGet,
+          request_deserializer=MurmurRPC__pb2.Ban.Query.FromString,
+          response_serializer=MurmurRPC__pb2.Ban.List.SerializeToString,
+      ),
+      'BansSet': grpc.unary_unary_rpc_method_handler(
+          servicer.BansSet,
+          request_deserializer=MurmurRPC__pb2.Ban.List.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ACLGet': grpc.unary_unary_rpc_method_handler(
+          servicer.ACLGet,
+          request_deserializer=MurmurRPC__pb2.Channel.FromString,
+          response_serializer=MurmurRPC__pb2.ACL.List.SerializeToString,
+      ),
+      'ACLSet': grpc.unary_unary_rpc_method_handler(
+          servicer.ACLSet,
+          request_deserializer=MurmurRPC__pb2.ACL.List.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ACLGetEffectivePermissions': grpc.unary_unary_rpc_method_handler(
+          servicer.ACLGetEffectivePermissions,
+          request_deserializer=MurmurRPC__pb2.ACL.Query.FromString,
+          response_serializer=MurmurRPC__pb2.ACL.SerializeToString,
+      ),
+      'ACLAddTemporaryGroup': grpc.unary_unary_rpc_method_handler(
+          servicer.ACLAddTemporaryGroup,
+          request_deserializer=MurmurRPC__pb2.ACL.TemporaryGroup.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'ACLRemoveTemporaryGroup': grpc.unary_unary_rpc_method_handler(
+          servicer.ACLRemoveTemporaryGroup,
+          request_deserializer=MurmurRPC__pb2.ACL.TemporaryGroup.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'AuthenticatorStream': grpc.stream_stream_rpc_method_handler(
+          servicer.AuthenticatorStream,
+          request_deserializer=MurmurRPC__pb2.Authenticator.Response.FromString,
+          response_serializer=MurmurRPC__pb2.Authenticator.Request.SerializeToString,
+      ),
+      'DatabaseUserQuery': grpc.unary_unary_rpc_method_handler(
+          servicer.DatabaseUserQuery,
+          request_deserializer=MurmurRPC__pb2.DatabaseUser.Query.FromString,
+          response_serializer=MurmurRPC__pb2.DatabaseUser.List.SerializeToString,
+      ),
+      'DatabaseUserGet': grpc.unary_unary_rpc_method_handler(
+          servicer.DatabaseUserGet,
+          request_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+          response_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+      ),
+      'DatabaseUserUpdate': grpc.unary_unary_rpc_method_handler(
+          servicer.DatabaseUserUpdate,
+          request_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'DatabaseUserRegister': grpc.unary_unary_rpc_method_handler(
+          servicer.DatabaseUserRegister,
+          request_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+          response_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+      ),
+      'DatabaseUserDeregister': grpc.unary_unary_rpc_method_handler(
+          servicer.DatabaseUserDeregister,
+          request_deserializer=MurmurRPC__pb2.DatabaseUser.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'DatabaseUserVerify': grpc.unary_unary_rpc_method_handler(
+          servicer.DatabaseUserVerify,
+          request_deserializer=MurmurRPC__pb2.DatabaseUser.Verify.FromString,
+          response_serializer=MurmurRPC__pb2.DatabaseUser.SerializeToString,
+      ),
+      'RedirectWhisperGroupAdd': grpc.unary_unary_rpc_method_handler(
+          servicer.RedirectWhisperGroupAdd,
+          request_deserializer=MurmurRPC__pb2.RedirectWhisperGroup.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+      'RedirectWhisperGroupRemove': grpc.unary_unary_rpc_method_handler(
+          servicer.RedirectWhisperGroupRemove,
+          request_deserializer=MurmurRPC__pb2.RedirectWhisperGroup.FromString,
+          response_serializer=MurmurRPC__pb2.Void.SerializeToString,
+      ),
+  }
+  generic_handler = grpc.method_handlers_generic_handler(
+      'MurmurRPC.V1', rpc_method_handlers)
+  server.add_generic_rpc_handlers((generic_handler,))

mumble/TODO

@@ -0,0 +1,8 @@
+-add lsChans()
+-lsACL? lsBans? edit these?
+-find out some way to use the ICE/GRPC interface completely
+
+-i need to learn way more about GRPC:
+https://wiki.mumble.info/wiki/GRPC
+https://github.com/mumble-voip/mumble/issues/1196
+https://grpc.io/docs/tutorials/basic/python.html

mumble/gencerthash.py

@@ -0,0 +1,242 @@
+#!/usr/bin/env python3
+
+# TODO: can we use struct instead for blobParser?
+
+import argparse
+import getpass
+import hashlib
+import re
+import sys
+import os
+from collections import defaultdict
+try:
+    import OpenSSL  # "python-pyopenssl" package on Arch
+except ImportError:
+    exit('You need to install PyOpenSSL ("pip3 install --user PyOpenSSL" if pip3 is installed)')
+
+## DEFINE SOME PRETTY STUFF ##
+class color(object):
+    # Windows doesn't support ANSI color escapes like sh does.
+    if sys.platform == 'win32':
+        # Gorram it, Windows.
+        # https://bugs.python.org/issue29059
+        # https://bugs.python.org/issue30075
+        # https://github.com/Microsoft/WSL/issues/1173
+        import subprocess
+        subprocess.call('', shell=True)
+    PURPLE = '\033[95m'
+    CYAN = '\033[96m'
+    DARKCYAN = '\033[36m'
+    BLUE = '\033[94m'
+    GREEN = '\033[92m'
+    YELLOW = '\033[93m'
+    RED = '\033[91m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'
+    END = '\033[0m'
+
+class Hasher(object):
+    def __init__(self, args):
+        self.args = args
+        self.blobGetter(self.args['cert'])
+        self.blobParser()
+
+    def getPass(self):
+        # Do we need to get the passphrase?
+        if self.args['passphrase']:
+            if self.args['passphrase'] == 'stdin':
+                self.args['passphrase'] = sys.stdin.read().replace('\n', '')
+            elif self.args['passphrase'] == 'prompt':
+                _colorargs = (color.BOLD, color.RED, self.args['cert'], color.END)
+                _repeat = True
+                while _repeat == True:
+                    _pass_in = getpass.getpass(('\n{0}What is the encryption password ' +
+                                                'for {1}{2}{0}{3}{0} ?{3} ').format(*_colorargs))
+                    if not _pass_in or _pass_in == '':
+                        print(('\n{0}Invalid passphrase for {1}{2}{0}{3}{0} ; ' +
+                               'please enter a valid passphrase!{3} ').format(*_colorargs))
+                    else:
+                        _repeat = False
+                        self.args['passphrase'] = _pass_in.replace('\n', '')
+                        print()
+            else:
+                self.args['passphrase'] = None
+        return()
+
+    def importCert(self):
+        self.getPass()
+        # Try loading the certificate
+        try:
+            self.pkcs = OpenSSL.crypto.load_pkcs12(self.cert, self.args['passphrase'])
+        except OpenSSL.crypto.Error:
+            exit('Could not load certificate! (Wrong passphrase? Wrong file?)')
+        return()
+
+    def hashCert(self):
+        self.crt_in = self.pkcs.get_certificate()
+        self.der = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_ASN1,
+                                                   self.crt_in)
+        self.hash = hashlib.sha1(self.der).hexdigest().lower()
+        return(self.hash)
+
+    def blobGetter(self, blobpath):
+        self.cert = None
+        self.blob = None
+        _blst = blobpath.split(':')
+        if len(_blst) == 2:
+            blob = _blst[1]
+            self.certtype = _blst[0].lower()
+        elif len(_blst) == 1:
+            blob = _blst[0]
+            self.certtype = 'file'
+        else:
+            raise ValueError('{0} is not a supported path'.format(blobpath))
+            self.certtype = None
+        if self.certtype:
+            _hexblob = None
+            if self.certtype in ('plist', 'ini', 'file'):
+                blob = os.path.abspath(os.path.expanduser(blob))
+                if not os.path.isfile(blob):
+                    raise FileNotFoundError('{0} does not exist'.format(blob))
+            if self.certtype == 'reg':  # Only supported on Windows machines, obviously.
+                if sys.platform == 'win32':
+                    import winreg
+                elif sys.platform == 'cygwin':
+                    # https://bitbucket.org/sfllaw/cygwinreg/issues/5/support-python3
+                    exit(('Python 3 under Cygwin does not support reading the registry. ' +
+                          'Please use native-Windows Python 3 (for now) or ' +
+                          'specify an actual PKCS #12 certificate file.'))
+                    #try:
+                    #    import cygwinreg as winreg
+                    #except ImportError:
+                    #    exit('You must install the cygwinreg python module in your cygwin environment to read the registry.')
+                _keypath = blob.split('\\')
+                _hkey = getattr(winreg, _keypath[0])
+                _skeypath = _keypath[1:-1]
+                _ckey = _keypath[-1]
+                _r = winreg.OpenKey(_hkey, '\\'.join(_skeypath))
+                _hexblob, _ = winreg.QueryValueEx(_r, _ckey)
+                winreg.CloseKey(_r)
+            elif self.certtype == 'plist':  # plistlib, however, is thankfully cross-platform.
+                import plistlib
+                with open(blob, 'rb') as f:
+                    _pdata = plistlib.loads(f.read())
+                    _hexblob = _pdata['net.certificate']
+            elif self.certtype == 'ini':
+                import configparser
+                _parser = configparser.RawConfigParser()
+                _parser.read(blob)
+                _cfg = defaultdict(dict)
+                for s in _parser.sections():
+                    _cfg[s] = {}
+                    for k in _parser.options(s):
+                        _cfg[s][k] = _parser.get(s, k)
+                self.blob = _cfg['net']['certificate']
+            else:  # It's (supposedly) a PKCS #12 file - obviously, cross-platform.
+                with open(blob, 'rb') as f:
+                    self.cert = f.read()
+        return()
+
+    def blobParser(self):
+        if not self.blob:
+            return()
+        if self.blob == '':
+            raise ValueError('We could not find an embedded certificate.')
+        # A pox upon the house of Mumble for not using base64. A POX, I SAY.
+        # So instead we need to straight up de-byte-array the mess.
+        # The below is an eldritch horror, bound to twist the mind of any sane man
+        # into the depths of madness.
+        # I probably might have been able to use a struct here, but meh.
+        blob = re.sub('^"?@ByteArray\(0(.*)\)"?$',
+                      '\g<1>',
+                      self.blob,
+                      re.MULTILINE, re.DOTALL)
+        _bytes = b'0'
+        for s in blob.split('\\x'):
+            if s == '':
+                continue
+            _chunk = list(s)
+            # Skip the first two chars for string interpolation - they're hex.
+            _start = 2
+            try:
+                _hex = ''.join(_chunk[0:2])
+                _bytes += bytes.fromhex(_hex)
+            except ValueError:
+                # We need to zero-pad, and alter the starting index
+                # because yep, you guessed it - their bytearray hex vals
+                # (in plaintext) aren't zero-padded, either.
+                _hex = ''.join(_chunk[0]).zfill(2)
+                _bytes += bytes.fromhex(_hex)
+                _start = 1
+            # And then append the rest as-is. "Mostly."
+            # Namely, we need to change the single-digit null byte notation
+            # to actual python null bytes, and then de-escape the escapes.
+            # (i.e. '\t' => '   ')
+            _str = re.sub('\\\\0([^0])',
+                          '\00\g<1>',
+                          ''.join(_chunk[_start:])).encode('utf-8').decode('unicode_escape')
+            _bytes += _str.encode('utf-8')
+            self.cert = _bytes
+        return()
+
+def parseArgs():
+    # Set the default cert path
+    _certpath = '~/Documents/MumbleAutomaticCertificateBackup.p12'
+    # This catches ALL versions of macOS/OS X.
+    if sys.platform == 'darwin':
+        _cfgpath = 'PLIST:~/Library/Preferences/net.sourceforge.mumble.Mumble.plist'
+    # ALL versions of windows, even Win10, on x86. Even 64-bit. I know.
+    # And Cygwin, which currently doesn't even suppport registry reading (see blobGetter()).
+    elif sys.platform in ('win32', 'cygwin'):
+        _cfgpath = r'REG:HKEY_CURRENT_USER\Software\Mumble\Mumble\net\certificate'
+    elif (sys.platform == 'linux') or (re.match('.*bsd.*', sys.platform)):  # duh
+        _cfgpath = 'INI:~/.config/Mumble/Mumble.conf'
+    else:
+        # WHO KNOWS what we're running on
+        _cfgpath = None
+    if not os.path.isfile(os.path.abspath(os.path.expanduser(_certpath))):
+        _defcrt = _cfgpath
+    else:
+        _defcrt = 'FILE:{0}'.format(_certpath)
+    args = argparse.ArgumentParser()
+    args.add_argument('-p',
+                      '--passphrase',
+                      choices = ['stdin', 'prompt'],
+                      dest = 'passphrase',
+                      default = None,
+                      help = ('The default is to behave as if your certificate does not have ' +
+                              'a passphrase attached (as this is Mumble\'s default); however, ' +
+                              'if you specify \'stdin\' we will expect the passphrase to be given as a stdin pipe, ' +
+                              'if you specify \'prompt\', we will prompt you for a passphrase (it will not be echoed back' +
+                              'to the console)'))
+    args.add_argument('-c', '--cert',
+                      dest = 'cert',
+                      default = _defcrt,
+                      metavar = 'path/to/mumblecert.p12',
+                      help = ('The path to your exported PKCS #12 Mumble certificate. ' +
+                              'Special prefixes are ' +
+                              '{0} (it is a PKCS #12 file, default), ' +
+                              '{1} (it is embedded in a macOS/OS X PLIST file), ' +
+                              '{2} (it is a Mumble.conf with embedded PKCS#12), or ' +
+                              '{3} (it is a path to a Windows registry object). ' +
+                              'Default: {4}').format('{0}FILE{1}'.format(color.BOLD, color.END),
+                                                     '{0}PLIST{1}'.format(color.BOLD, color.END),
+                                                     '{0}INI{1}'.format(color.BOLD, color.END),
+                                                     '{0}REG{1}'.format(color.BOLD, color.END),
+                                                     '{0}{1}{2}'.format(color.BOLD, _defcrt, color.END)))
+                                                     # this  ^ currently prints "0m" at the end of the help message,
+                                                     # all the way on the left on Windows.
+                                                     # Why? Who knows; Microsoft is a mystery even to themselves.
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    cert = Hasher(args)
+    cert.importCert()
+    h = cert.hashCert()
+    print(('\n\t{0}Your certificate\'s public hash is: ' +
+           '{1}{2}{3}\n\n\t{0}Please provide this to the Mumble server administrator ' +
+           'that has requested it.{3}').format(color.BOLD, color.BLUE, h, color.END))
+
+if __name__ == '__main__':
+    main()

mumble/getusers.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+
+import copy
+import pprint
+import usrmgmt2
+
+# NOTE: THIS IS ONLY FOR TESTING/DEVELOPMENT PURPOSES.
+# IT WILL BE REMOVED ONCE THE ACTUAL STUFF IS FINISHED.
+
+args = vars(usrmgmt2.parseArgs().parse_args())
+args['operation'] = 'ls'
+args['verbose'] = True
+args['cfgfile'] = '/home/bts/.config/optools/mumbleadmin.ini'
+#if not args['operation']:
+    #raise RuntimeError('You must specify an operation to perform. Try running with -h/--help.')
+#    exit('You must specify an operation to perform. Try running with -h/--help.')
+
+mgmt = usrmgmt2.IceMgr(args)
+
+def dictify(obj):
+    # thanks, https://github.com/alfg/murmur-rest/blob/master/app/utils.py
+    _rv = {'_type': str(type(obj))}
+    if type(obj) in (bool, int, float, str, bytes):
+        return(obj)
+    if type(obj) in (list, tuple):
+        return([dictify(i) for i in obj])
+    if type(obj) == dict:
+        return(dict((str(k), dictify(v)) for k, v in obj.items()))
+    return(dictify(obj.__dict__))
+
+
+# Here we actually print users
+#print(inspect.getmembers(Murmur.UserInfo))
+#for s in mgmt.conn['read'].getAllServers():  # iterate through all servers
+    #userattrs = [Murmur.UserInfo.Username, Murmur.UserInfo.UserEmail,
+    #             Murmur.UserInfo.UserHash, Murmur.UserInfo.UserLastActive,
+    #             Murmur.UserInfo.UserComment]
+    #print(type(s))
+    #pprint.pprint(s.getRegisteredUsers(''))  # either print a UID:username map...
+#    for uid, uname in s.getRegisteredUsers('').items():  # or let's try to get full info on them
+        #print('user: {0}\nusername: {1}\n'.format(uid, uname))
+#        _u = dictify(s.getRegistration(uid))
+#        if uid == 3:
+#            print(_u)
+
+print(mgmt.conn['read'])
+_server = mgmt.conn['read'].getServer(1)
+
+print(_server.getACL(0))
+
+#acl = _server.getACL(0)
+#print(acl[0])
+
+
+
+#pprint.pprint(dictify(acl), indent = 4)
+
+mgmt.close()

mumble/grpctest.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+
+import grpc
+from grpc.tools import protoc
+import tempfile
+
+channel = grpc.insecure_channel('sysadministrivia.com:50051')

mumble/murmur.ice

@@ -0,0 +1,888 @@
+// https://raw.githubusercontent.com/mumble-voip/mumble/master/src/murmur/Murmur.ice
+// http://mumble.sourceforge.net/slice/1.3.0/Murmur.html
+
+// Copyright 2005-2017 The Mumble Developers. All rights reserved.
+// Use of this source code is governed by a BSD-style license
+// that can be found in the LICENSE file at the root of the
+// Mumble source tree or at <https://www.mumble.info/LICENSE>.
+
+/**
+ *
+ * Information and control of the murmur server. Each server has
+ * one {@link Meta} interface that controls global information, and
+ * each virtual server has a {@link Server} interface.
+ *
+ **/
+
+#include <Ice/SliceChecksumDict.ice>
+
+module Murmur
+{
+
+	/** A network address in IPv6 format.
+	 **/
+	["python:seq:tuple"] sequence<byte> NetAddress;
+
+	/** A connected user.
+	 **/
+	struct User {
+		/** Session ID. This identifies the connection to the server. */
+		int session;
+		/** User ID. -1 if the user is anonymous. */
+		int userid;
+		/** Is user muted by the server? */
+		bool mute;
+		/** Is user deafened by the server? If true, this implies mute. */
+		bool deaf;
+		/** Is the user suppressed by the server? This means the user is not muted, but does not have speech privileges in the current channel. */
+		bool suppress;
+		/** Is the user a priority speaker? */
+		bool prioritySpeaker;
+		/** Is the user self-muted? */
+		bool selfMute;
+		/** Is the user self-deafened? If true, this implies mute. */
+		bool selfDeaf;
+		/** Is the User recording? (This flag is read-only and cannot be changed using setState().) **/
+		bool recording;
+		/** Channel ID the user is in. Matches {@link Channel.id}. */
+		int channel;
+		/** The name of the user. */
+		string name;
+		/** Seconds user has been online. */
+		int onlinesecs;
+		/** Average transmission rate in bytes per second over the last few seconds. */
+		int bytespersec;
+		/** Client version. Major version in upper 16 bits, followed by 8 bits of minor version and 8 bits of patchlevel. Version 1.2.3 = 0x010203. */
+		int version;
+		/** Client release. For official releases, this equals the version. For snapshots and git compiles, this will be something else. */
+		string release;
+		/** Client OS. */
+		string os;
+		/** Client OS Version. */
+		string osversion;
+		/** Plugin Identity. This will be the user's unique ID inside the current game. */
+		string identity;
+		/**
+		   Base64-encoded Plugin context. This is a binary blob identifying the game and team the user is on.
+
+		   The used Base64 alphabet is the one specified in RFC 2045.
+
+		   Before Mumble 1.3.0, this string was not Base64-encoded. This could cause problems for some Ice
+		   implementations, such as the .NET implementation.
+
+		   If you need the exact string that is used by Mumble, you can get it by Base64-decoding this string.
+
+		   If you simply need to detect whether two users are in the same game world, string comparisons will
+		   continue to work as before.
+		 */
+		string context;
+		/** User comment. Shown as tooltip for this user. */
+		string comment;
+		/** Client address. */
+		NetAddress address;
+		/** TCP only. True until UDP connectivity is established. */
+		bool tcponly;
+		/** Idle time. This is how many seconds it is since the user last spoke. Other activity is not counted. */
+		int idlesecs;
+		/** UDP Ping Average. This is the average ping for the user via UDP over the duration of the connection. */
+		float udpPing;
+		/** TCP Ping Average. This is the average ping for the user via TCP over the duration of the connection. */
+		float tcpPing;
+	};
+
+	sequence<int> IntList;
+
+	/** A text message between users.
+	 **/
+	struct TextMessage {
+		/** Sessions (connected users) who were sent this message. */
+		IntList sessions;
+		/** Channels who were sent this message. */
+		IntList channels;
+		/** Trees of channels who were sent this message. */
+		IntList trees;
+		/** The contents of the message. */
+		string text;
+	};
+
+	/** A channel.
+	 **/
+	struct Channel {
+		/** Channel ID. This is unique per channel, and the root channel is always id 0. */
+		int id;
+		/** Name of the channel. There can not be two channels with the same parent that has the same name. */
+		string name;
+		/** ID of parent channel, or -1 if this is the root channel. */
+		int parent;
+		/** List of id of linked channels. */
+		IntList links;
+		/** Description of channel. Shown as tooltip for this channel. */
+		string description;
+		/** Channel is temporary, and will be removed when the last user leaves it. */
+		bool temporary;
+		/** Position of the channel which is used in Client for sorting. */
+		int position;
+	};
+
+	/** A group. Groups are defined per channel, and can inherit members from parent channels.
+	 **/
+	struct Group {
+		/** Group name */
+		string name;
+		/** Is this group inherited from a parent channel? Read-only. */
+		bool inherited;
+		/** Does this group inherit members from parent channels? */
+		bool inherit;
+		/** Can subchannels inherit members from this group? */
+		bool inheritable;
+		/** List of users to add to the group. */
+		IntList add;
+		/** List of inherited users to remove from the group. */
+		IntList remove;
+		/** Current members of the group, including inherited members. Read-only. */
+		IntList members;
+	};
+
+	/** Write access to channel control. Implies all other permissions (except Speak). */
+	const int PermissionWrite = 0x01;
+	/** Traverse channel. Without this, a client cannot reach subchannels, no matter which privileges he has there. */
+	const int PermissionTraverse = 0x02;
+	/** Enter channel. */
+	const int PermissionEnter = 0x04;
+	/** Speak in channel. */
+	const int PermissionSpeak = 0x08;
+	/** Whisper to channel. This is different from Speak, so you can set up different permissions. */
+	const int PermissionWhisper = 0x100;
+	/** Mute and deafen other users in this channel. */
+	const int PermissionMuteDeafen = 0x10;
+	/** Move users from channel. You need this permission in both the source and destination channel to move another user. */
+	const int PermissionMove = 0x20;
+	/** Make new channel as a subchannel of this channel. */
+	const int PermissionMakeChannel = 0x40;
+	/** Make new temporary channel as a subchannel of this channel. */
+	const int PermissionMakeTempChannel = 0x400;
+	/** Link this channel. You need this permission in both the source and destination channel to link channels, or in either channel to unlink them. */
+	const int PermissionLinkChannel = 0x80;
+	/** Send text message to channel. */
+	const int PermissionTextMessage = 0x200;
+	/** Kick user from server. Only valid on root channel. */
+	const int PermissionKick = 0x10000;
+	/** Ban user from server. Only valid on root channel. */
+	const int PermissionBan = 0x20000;
+	/** Register and unregister users. Only valid on root channel. */
+	const int PermissionRegister = 0x40000;
+	/** Register and unregister users. Only valid on root channel. */
+	const int PermissionRegisterSelf = 0x80000;
+
+
+	/** Access Control List for a channel. ACLs are defined per channel, and can be inherited from parent channels.
+	 **/
+	struct ACL {
+		/** Does the ACL apply to this channel? */
+		bool applyHere;
+		/** Does the ACL apply to subchannels? */
+		bool applySubs;
+		/** Is this ACL inherited from a parent channel? Read-only. */
+		bool inherited;
+		/** ID of user this ACL applies to. -1 if using a group name. */
+		int userid;
+		/** Group this ACL applies to. Blank if using userid. */
+		string group;
+		/** Binary mask of privileges to allow. */
+		int allow;
+		/** Binary mask of privileges to deny. */
+		int deny;
+	};
+
+	/** A single ip mask for a ban.
+	 **/
+	struct Ban {
+		/** Address to ban. */
+		NetAddress address;
+		/** Number of bits in ban to apply. */
+		int bits;
+		/** Username associated with ban. */
+		string name;
+		/** Hash of banned user. */
+		string hash;
+		/** Reason for ban. */
+		string reason;
+		/** Date ban was applied in unix time format. */
+		int start;
+		/** Duration of ban. */
+		int duration;
+	};
+
+	/** A entry in the log.
+	 **/
+	struct LogEntry {
+		/** Timestamp in UNIX time_t */
+		int timestamp;
+		/** The log message. */
+		string txt;
+	};
+
+	class Tree;
+	sequence<Tree> TreeList;
+
+	enum ChannelInfo { ChannelDescription, ChannelPosition };
+	enum UserInfo { UserName, UserEmail, UserComment, UserHash, UserPassword, UserLastActive };
+
+	dictionary<int, User> UserMap;
+	dictionary<int, Channel> ChannelMap;
+	sequence<Channel> ChannelList;
+	sequence<User> UserList;
+	sequence<Group> GroupList;
+	sequence<ACL> ACLList;
+	sequence<LogEntry> LogList;
+	sequence<Ban> BanList;
+	sequence<int> IdList;
+	sequence<string> NameList;
+	dictionary<int, string> NameMap;
+	dictionary<string, int> IdMap;
+	sequence<byte> Texture;
+	dictionary<string, string> ConfigMap;
+	sequence<string> GroupNameList;
+	sequence<byte> CertificateDer;
+	sequence<CertificateDer> CertificateList;
+
+	/** User information map.
+	 * Older versions of ice-php can't handle enums as keys. If you are using one of these, replace 'UserInfo' with 'byte'.
+	 */
+
+	dictionary<UserInfo, string> UserInfoMap;
+
+	/** User and subchannel state. Read-only.
+	 **/
+	class Tree {
+		/** Channel definition of current channel. */
+		Channel c;
+		/** List of subchannels. */
+		TreeList children;
+		/** Users in this channel. */
+		UserList users;
+	};
+
+	exception MurmurException {};
+	/** This is thrown when you specify an invalid session. This may happen if the user has disconnected since your last call to {@link Server.getUsers}. See {@link User.session} */
+	exception InvalidSessionException extends MurmurException {};
+	/** This is thrown when you specify an invalid channel id. This may happen if the channel was removed by another provess. It can also be thrown if you try to add an invalid channel. */
+	exception InvalidChannelException extends MurmurException {};
+	/** This is thrown when you try to do an operation on a server that does not exist. This may happen if someone has removed the server. */
+	exception InvalidServerException extends MurmurException {};
+	/** This happens if you try to fetch user or channel state on a stopped server, if you try to stop an already stopped server or start an already started server. */
+	exception ServerBootedException extends MurmurException {};
+	/** This is thrown if {@link Server.start} fails, and should generally be the cause for some concern. */
+	exception ServerFailureException extends MurmurException {};
+	/** This is thrown when you specify an invalid userid. */
+	exception InvalidUserException extends MurmurException {};
+	/** This is thrown when you try to set an invalid texture. */
+	exception InvalidTextureException extends MurmurException {};
+	/** This is thrown when you supply an invalid callback. */
+	exception InvalidCallbackException extends MurmurException {};
+	/**  This is thrown when you supply the wrong secret in the calling context. */
+	exception InvalidSecretException extends MurmurException {};
+	/** This is thrown when the channel operation would excede the channel nesting limit */
+	exception NestingLimitException extends MurmurException {};
+	/**  This is thrown when you ask the server to disclose something that should be secret. */
+	exception WriteOnlyException extends MurmurException {};
+	/** This is thrown when invalid input data was specified. */
+	exception InvalidInputDataException extends MurmurException {};
+
+	/** Callback interface for servers. You can supply an implementation of this to receive notification
+	 *  messages from the server.
+	 *  If an added callback ever throws an exception or goes away, it will be automatically removed.
+	 *  Please note that all callbacks are done asynchronously; murmur does not wait for the callback to
+	 *  complete before continuing processing.
+	 *  Note that callbacks are removed when a server is stopped, so you should have a callback for
+	 *  {@link MetaCallback.started} which calls {@link Server.addCallback}.
+	 *  @see MetaCallback
+	 *  @see Server.addCallback
+	 */
+	interface ServerCallback {
+		/** Called when a user connects to the server. 
+		 *  @param state State of connected user.
+		 */
+		idempotent void userConnected(User state);
+		/** Called when a user disconnects from the server. The user has already been removed, so you can no longer use methods like {@link Server.getState}
+		 *  to retrieve the user's state.
+		 *  @param state State of disconnected user.
+		 */
+		idempotent void userDisconnected(User state);
+		/** Called when a user state changes. This is called if the user moves, is renamed, is muted, deafened etc.
+		 *  @param state New state of user.
+		 */
+		idempotent void userStateChanged(User state);
+		/** Called when user writes a text message
+		 *  @param state the User sending the message
+		 *  @param message the TextMessage the user has sent
+		 */
+		idempotent void userTextMessage(User state, TextMessage message);
+		/** Called when a new channel is created. 
+		 *  @param state State of new channel.
+		 */
+		idempotent void channelCreated(Channel state);
+		/** Called when a channel is removed. The channel has already been removed, you can no longer use methods like {@link Server.getChannelState}
+		 *  @param state State of removed channel.
+		 */
+		idempotent void channelRemoved(Channel state);
+		/** Called when a new channel state changes. This is called if the channel is moved, renamed or if new links are added.
+		 *  @param state New state of channel.
+		 */
+		idempotent void channelStateChanged(Channel state);
+	};
+
+	/** Context for actions in the Server menu. */
+	const int ContextServer = 0x01;
+	/** Context for actions in the Channel menu. */
+	const int ContextChannel = 0x02;
+	/** Context for actions in the User menu. */
+	const int ContextUser = 0x04;
+
+	/** Callback interface for context actions. You need to supply one of these for {@link Server.addContext}. 
+	 *  If an added callback ever throws an exception or goes away, it will be automatically removed.
+	 *  Please note that all callbacks are done asynchronously; murmur does not wait for the callback to
+	 *  complete before continuing processing.
+	 */
+	interface ServerContextCallback {
+		/** Called when a context action is performed.
+		 *  @param action Action to be performed.
+		 *  @param usr User which initiated the action.
+		 *  @param session If nonzero, session of target user.
+		 *  @param channelid If not -1, id of target channel.
+		 */
+		idempotent void contextAction(string action, User usr, int session, int channelid);
+	};
+
+	/** Callback interface for server authentication. You need to supply one of these for {@link Server.setAuthenticator}.
+	 *  If an added callback ever throws an exception or goes away, it will be automatically removed.
+	 *  Please note that unlike {@link ServerCallback} and {@link ServerContextCallback}, these methods are called
+	 *  synchronously. If the response lags, the entire murmur server will lag.
+	 *  Also note that, as the method calls are synchronous, making a call to {@link Server} or {@link Meta} will
+	 *  deadlock the server.
+	 */
+	interface ServerAuthenticator {
+		/** Called to authenticate a user. If you do not know the username in question, always return -2 from this
+		 *  method to fall through to normal database authentication.
+		 *  Note that if authentication succeeds, murmur will create a record of the user in it's database, reserving
+		 *  the username and id so it cannot be used for normal database authentication.
+		 *  The data in the certificate (name, email addresses etc), as well as the list of signing certificates,
+		 *  should only be trusted if certstrong is true.
+		 *
+		 *  Internally, Murmur treats usernames as case-insensitive. It is recommended
+		 *  that authenticators do the same. Murmur checks if a username is in use when
+		 *  a user connects. If the connecting user is registered, the other username is
+		 *  kicked. If the connecting user is not registered, the connecting user is not
+		 *  allowed to join the server.
+		 *
+		 *  @param name Username to authenticate.
+		 *  @param pw Password to authenticate with.
+		 *  @param certificates List of der encoded certificates the user connected with.
+		 *  @param certhash Hash of user certificate, as used by murmur internally when matching.
+		 *  @param certstrong True if certificate was valid and signed by a trusted CA.
+		 *  @param newname Set this to change the username from the supplied one.
+		 *  @param groups List of groups on the root channel that the user will be added to for the duration of the connection.
+		 *  @return UserID of authenticated user, -1 for authentication failures, -2 for unknown user (fallthrough),
+		 *          -3 for authentication failures where the data could (temporarily) not be verified.
+		 */
+		idempotent int authenticate(string name, string pw, CertificateList certificates, string certhash, bool certstrong, out string newname, out GroupNameList groups);
+
+		/** Fetch information about a user. This is used to retrieve information like email address, keyhash etc. If you
+		 *  want murmur to take care of this information itself, simply return false to fall through.
+		 *  @param id User id.
+		 *  @param info Information about user. This needs to include at least "name".
+		 *  @return true if information is present, false to fall through.
+		 */
+		idempotent bool getInfo(int id, out UserInfoMap info);
+	
+		/** Map a name to a user id.
+		 *  @param name Username to map.
+		 *  @return User id or -2 for unknown name.
+		 */
+		idempotent int nameToId(string name);
+
+		/** Map a user id to a username.
+		 *  @param id User id to map.
+		 *  @return Name of user or empty string for unknown id.
+		 */
+		idempotent string idToName(int id);
+
+		/** Map a user to a custom Texture.
+		 *  @param id User id to map.
+		 *  @return User texture or an empty texture for unknwon users or users without textures.
+		 */
+		idempotent Texture idToTexture(int id);
+	};
+
+	/** Callback interface for server authentication and registration. This allows you to support both authentication
+	 *  and account updating.
+	 *  You do not need to implement this if all you want is authentication, you only need this if other scripts
+	 *  connected to the same server calls e.g. {@link Server.setTexture}.
+	 *  Almost all of these methods support fall through, meaning murmur should continue the operation against its
+	 *  own database.
+	 */
+	interface ServerUpdatingAuthenticator extends ServerAuthenticator {
+		/** Register a new user.
+		 *  @param info Information about user to register.
+		 *  @return User id of new user, -1 for registration failure, or -2 to fall through.
+		 */
+		int registerUser(UserInfoMap info);
+
+		/** Unregister a user.
+		 *  @param id Userid to unregister.
+		 *  @return 1 for successfull unregistration, 0 for unsuccessfull unregistration, -1 to fall through.
+		 */
+		int unregisterUser(int id);
+
+		/** Get a list of registered users matching filter.
+		 *  @param filter Substring usernames must contain. If empty, return all registered users.
+		 *  @return List of matching registered users.
+		 */
+		idempotent NameMap getRegisteredUsers(string filter);
+
+		/** Set additional information for user registration.
+		 *  @param id Userid of registered user.
+		 *  @param info Information to set about user. This should be merged with existing information.
+		 *  @return 1 for successfull update, 0 for unsuccessfull update, -1 to fall through.
+		 */
+		idempotent int setInfo(int id, UserInfoMap info);
+
+		/** Set texture (now called avatar) of user registration.
+		 *  @param id registrationId of registered user.
+		 *  @param tex New texture.
+		 *  @return 1 for successfull update, 0 for unsuccessfull update, -1 to fall through.
+		 */
+		idempotent int setTexture(int id, Texture tex);
+	};
+
+	/** Per-server interface. This includes all methods for configuring and altering
+	 * the state of a single virtual server. You can retrieve a pointer to this interface
+	 * from one of the methods in {@link Meta}.
+	 **/
+	["amd"] interface Server {
+		/** Shows if the server currently running (accepting users).
+		 *
+		 * @return Run-state of server.
+		 */
+		idempotent bool isRunning() throws InvalidSecretException;
+
+		/** Start server. */
+		void start() throws ServerBootedException, ServerFailureException, InvalidSecretException;
+
+		/** Stop server.
+		 * Note: Server will be restarted on Murmur restart unless explicitly disabled
+		 *       with setConf("boot", false)
+		 */
+		void stop() throws ServerBootedException, InvalidSecretException;
+
+		/** Delete server and all it's configuration. */
+		void delete() throws ServerBootedException, InvalidSecretException;
+
+		/** Fetch the server id.
+		 *
+		 * @return Unique server id.
+		 */
+		idempotent int id() throws InvalidSecretException;
+
+		/** Add a callback. The callback will receive notifications about changes to users and channels.
+		 *
+		 * @param cb Callback interface which will receive notifications.
+		 * @see removeCallback
+		 */
+		void addCallback(ServerCallback *cb) throws ServerBootedException, InvalidCallbackException, InvalidSecretException;
+
+		/** Remove a callback.
+		 *
+		 * @param cb Callback interface to be removed.
+		 * @see addCallback
+		 */
+		void removeCallback(ServerCallback *cb) throws ServerBootedException, InvalidCallbackException, InvalidSecretException;
+
+		/** Set external authenticator. If set, all authentications from clients are forwarded to this
+		 *  proxy.
+		 *
+		 * @param auth Authenticator object to perform subsequent authentications.
+		 */
+		void setAuthenticator(ServerAuthenticator *auth) throws ServerBootedException, InvalidCallbackException, InvalidSecretException;
+
+		/** Retrieve configuration item.
+		 * @param key Configuration key.
+		 * @return Configuration value. If this is empty, see {@link Meta.getDefaultConf}
+		 */
+		idempotent string getConf(string key) throws InvalidSecretException, WriteOnlyException;
+
+		/** Retrieve all configuration items.
+		 * @return All configured values. If a value isn't set here, the value from {@link Meta.getDefaultConf} is used.
+		 */
+		idempotent ConfigMap getAllConf() throws InvalidSecretException;
+
+		/** Set a configuration item.
+		 * @param key Configuration key.
+		 * @param value Configuration value.
+		 */
+		idempotent void setConf(string key, string value) throws InvalidSecretException;
+
+		/** Set superuser password. This is just a convenience for using {@link updateRegistration} on user id 0.
+		 * @param pw Password.
+		 */
+		idempotent void setSuperuserPassword(string pw) throws InvalidSecretException;
+
+		/** Fetch log entries.
+		 * @param first Lowest numbered entry to fetch. 0 is the most recent item.
+		 * @param last Last entry to fetch.
+		 * @return List of log entries.
+		 */
+		idempotent LogList getLog(int first, int last) throws InvalidSecretException;
+
+		/** Fetch length of log
+		 * @return Number of entries in log
+		 */
+		idempotent int getLogLen() throws InvalidSecretException;
+
+		/** Fetch all users. This returns all currently connected users on the server.
+		 * @return List of connected users.
+		 * @see getState
+		 */
+		idempotent UserMap getUsers() throws ServerBootedException, InvalidSecretException;
+
+		/** Fetch all channels. This returns all defined channels on the server. The root channel is always channel 0.
+		 * @return List of defined channels.
+		 * @see getChannelState
+		 */
+		idempotent ChannelMap getChannels() throws ServerBootedException, InvalidSecretException;
+
+		/** Fetch certificate of user. This returns the complete certificate chain of a user.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @return Certificate list of user.
+		 */
+		idempotent CertificateList getCertificateList(int session) throws ServerBootedException, InvalidSessionException, InvalidSecretException;
+
+		/** Fetch all channels and connected users as a tree. This retrieves an easy-to-use representation of the server
+		 *  as a tree. This is primarily used for viewing the state of the server on a webpage.
+		 * @return Recursive tree of all channels and connected users.
+		 */
+		idempotent Tree getTree() throws ServerBootedException, InvalidSecretException;
+
+		/** Fetch all current IP bans on the server.
+		 * @return List of bans.
+		 */
+		idempotent BanList getBans() throws ServerBootedException, InvalidSecretException;
+
+		/** Set all current IP bans on the server. This will replace any bans already present, so if you want to add a ban, be sure to call {@link getBans} and then
+		 *  append to the returned list before calling this method.
+		 * @param bans List of bans.
+		 */
+		idempotent void setBans(BanList bans) throws ServerBootedException, InvalidSecretException;
+
+		/** Kick a user. The user is not banned, and is free to rejoin the server.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param reason Text message to show when user is kicked.
+		 */
+		void kickUser(int session, string reason) throws ServerBootedException, InvalidSessionException, InvalidSecretException;
+
+		/** Get state of a single connected user. 
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @return State of connected user.
+		 * @see setState
+		 * @see getUsers
+		 */
+		idempotent User getState(int session) throws ServerBootedException, InvalidSessionException, InvalidSecretException;
+
+		/** Set user state. You can use this to move, mute and deafen users.
+		 * @param state User state to set.
+		 * @see getState
+		 */
+		idempotent void setState(User state) throws ServerBootedException, InvalidSessionException, InvalidChannelException, InvalidSecretException;
+
+		/** Send text message to a single user.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param text Message to send.
+		 * @see sendMessageChannel
+		 */
+		void sendMessage(int session, string text) throws ServerBootedException, InvalidSessionException, InvalidSecretException;
+
+		/** Check if user is permitted to perform action.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param channelid ID of Channel. See {@link Channel.id}.
+		 * @param perm Permission bits to check.
+		 * @return true if any of the permissions in perm were set for the user.
+		 */
+		bool hasPermission(int session, int channelid, int perm) throws ServerBootedException, InvalidSessionException, InvalidChannelException, InvalidSecretException;
+		
+		/** Return users effective permissions
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param channelid ID of Channel. See {@link Channel.id}.
+		 * @return bitfield of allowed actions
+		 */
+		idempotent int effectivePermissions(int session, int channelid) throws ServerBootedException, InvalidSessionException, InvalidChannelException, InvalidSecretException;
+
+		/** Add a context callback. This is done per user, and will add a context menu action for the user.
+		 *
+		 * @param session Session of user which should receive context entry.
+		 * @param action Action string, a unique name to associate with the action.
+		 * @param text Name of action shown to user.
+		 * @param cb Callback interface which will receive notifications.
+		 * @param ctx Context this should be used in. Needs to be one or a combination of {@link ContextServer}, {@link ContextChannel} and {@link ContextUser}.
+		 * @see removeContextCallback
+		 */
+		void addContextCallback(int session, string action, string text, ServerContextCallback *cb, int ctx) throws ServerBootedException, InvalidCallbackException, InvalidSecretException;
+
+		/** Remove a callback.
+		 *
+		 * @param cb Callback interface to be removed. This callback will be removed from all from all users.
+		 * @see addContextCallback
+		 */
+		void removeContextCallback(ServerContextCallback *cb) throws ServerBootedException, InvalidCallbackException, InvalidSecretException;
+		
+		/** Get state of single channel.
+		 * @param channelid ID of Channel. See {@link Channel.id}.
+		 * @return State of channel.
+		 * @see setChannelState
+		 * @see getChannels
+		 */
+		idempotent Channel getChannelState(int channelid) throws ServerBootedException, InvalidChannelException, InvalidSecretException;
+
+		/** Set state of a single channel. You can use this to move or relink channels.
+		 * @param state Channel state to set.
+		 * @see getChannelState
+		 */
+		idempotent void setChannelState(Channel state) throws ServerBootedException, InvalidChannelException, InvalidSecretException, NestingLimitException;
+
+		/** Remove a channel and all its subchannels.
+		 * @param channelid ID of Channel. See {@link Channel.id}.
+		 */
+		void removeChannel(int channelid) throws ServerBootedException, InvalidChannelException, InvalidSecretException;
+
+		/** Add a new channel.
+		 * @param name Name of new channel.
+		 * @param parent Channel ID of parent channel. See {@link Channel.id}.
+		 * @return ID of newly created channel.
+		 */
+		int addChannel(string name, int parent) throws ServerBootedException, InvalidChannelException, InvalidSecretException, NestingLimitException;
+
+		/** Send text message to channel or a tree of channels.
+		 * @param channelid Channel ID of channel to send to. See {@link Channel.id}.
+		 * @param tree If true, the message will be sent to the channel and all its subchannels.
+		 * @param text Message to send.
+		 * @see sendMessage
+		 */
+		void sendMessageChannel(int channelid, bool tree, string text) throws ServerBootedException, InvalidChannelException, InvalidSecretException;
+
+		/** Retrieve ACLs and Groups on a channel.
+		 * @param channelid Channel ID of channel to fetch from. See {@link Channel.id}.
+		 * @param acls List of ACLs on the channel. This will include inherited ACLs.
+		 * @param groups List of groups on the channel. This will include inherited groups.
+		 * @param inherit Does this channel inherit ACLs from the parent channel?
+		 */
+		idempotent void getACL(int channelid, out ACLList acls, out GroupList groups, out bool inherit) throws ServerBootedException, InvalidChannelException, InvalidSecretException;
+
+		/** Set ACLs and Groups on a channel. Note that this will replace all existing ACLs and groups on the channel.
+		 * @param channelid Channel ID of channel to fetch from. See {@link Channel.id}.
+		 * @param acls List of ACLs on the channel.
+		 * @param groups List of groups on the channel.
+		 * @param inherit Should this channel inherit ACLs from the parent channel?
+		 */
+		idempotent void setACL(int channelid, ACLList acls, GroupList groups, bool inherit) throws ServerBootedException, InvalidChannelException, InvalidSecretException;
+
+		/** Temporarily add a user to a group on a channel. This state is not saved, and is intended for temporary memberships.
+		 * @param channelid Channel ID of channel to add to. See {@link Channel.id}.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param group Group name to add to.
+		 */
+		idempotent void addUserToGroup(int channelid, int session, string group) throws ServerBootedException, InvalidChannelException, InvalidSessionException, InvalidSecretException;
+
+		/** Remove a user from a temporary group membership on a channel. This state is not saved, and is intended for temporary memberships.
+		 * @param channelid Channel ID of channel to add to. See {@link Channel.id}.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param group Group name to remove from.
+		 */
+		idempotent void removeUserFromGroup(int channelid, int session, string group) throws ServerBootedException, InvalidChannelException, InvalidSessionException, InvalidSecretException;
+
+		/** Redirect whisper targets for user. If set, whenever a user tries to whisper to group "source", the whisper will be redirected to group "target".
+		 * To remove a redirect pass an empty target string. This is intended for context groups.
+		 * @param session Connection ID of user. See {@link User.session}.
+		 * @param source Group name to redirect from.
+		 * @param target Group name to redirect to.
+		 */
+		idempotent void redirectWhisperGroup(int session, string source, string target) throws ServerBootedException, InvalidSessionException, InvalidSecretException;
+
+		/** Map a list of {@link User.userid} to a matching name.
+		 * @param List of ids.
+		 * @return Matching list of names, with an empty string representing invalid or unknown ids.
+		 */
+		idempotent NameMap getUserNames(IdList ids) throws ServerBootedException, InvalidSecretException;
+
+		/** Map a list of user names to a matching id.
+		 * @param List of names.
+		 * @reuturn List of matching ids, with -1 representing invalid or unknown user names.
+		 */
+		idempotent IdMap getUserIds(NameList names) throws ServerBootedException, InvalidSecretException;
+
+		/** Register a new user.
+		 * @param info Information about new user. Must include at least "name".
+		 * @return The ID of the user. See {@link RegisteredUser.userid}.
+		 */
+		int registerUser(UserInfoMap info) throws ServerBootedException, InvalidUserException, InvalidSecretException;
+
+		/** Remove a user registration.
+		 * @param userid ID of registered user. See {@link RegisteredUser.userid}.
+		 */
+		void unregisterUser(int userid) throws ServerBootedException, InvalidUserException, InvalidSecretException;
+
+		/** Update the registration for a user. You can use this to set the email or password of a user,
+		 * and can also use it to change the user's name.
+		 * @param registration Updated registration record.
+		 */
+		idempotent void updateRegistration(int userid, UserInfoMap info) throws ServerBootedException, InvalidUserException, InvalidSecretException;
+
+		/** Fetch registration for a single user.
+		 * @param userid ID of registered user. See {@link RegisteredUser.userid}.
+		 * @return Registration record.
+		 */
+		idempotent UserInfoMap getRegistration(int userid) throws ServerBootedException, InvalidUserException, InvalidSecretException;
+
+		/** Fetch a group of registered users.
+		 * @param filter Substring of user name. If blank, will retrieve all registered users.
+		 * @return List of registration records.
+		 */
+		idempotent NameMap getRegisteredUsers(string filter) throws ServerBootedException, InvalidSecretException;
+
+		/** Verify the password of a user. You can use this to verify a user's credentials.
+		 * @param name User name. See {@link RegisteredUser.name}.
+		 * @param pw User password.
+		 * @return User ID of registered user (See {@link RegisteredUser.userid}), -1 for failed authentication or -2 for unknown usernames.
+		 */
+		idempotent int verifyPassword(string name, string pw) throws ServerBootedException, InvalidSecretException;
+
+		/** Fetch user texture. Textures are stored as zlib compress()ed 600x60 32-bit BGRA data.
+		 * @param userid ID of registered user. See {@link RegisteredUser.userid}.
+		 * @return Custom texture associated with user or an empty texture.
+		 */
+		idempotent Texture getTexture(int userid) throws ServerBootedException, InvalidUserException, InvalidSecretException;
+
+		/** Set a user texture (now called avatar).
+		 * @param userid ID of registered user. See {@link RegisteredUser.userid}.
+		 * @param tex Texture (as a Byte-Array) to set for the user, or an empty texture to remove the existing texture.
+		 */
+		idempotent void setTexture(int userid, Texture tex) throws ServerBootedException, InvalidUserException, InvalidTextureException, InvalidSecretException;
+
+		/** Get virtual server uptime.
+		 * @return Uptime of the virtual server in seconds
+		 */
+		idempotent int getUptime() throws ServerBootedException, InvalidSecretException;
+
+		/**
+		 * Update the server's certificate information.
+		 *
+		 * Reconfigure the running server's TLS socket with the given
+		 * certificate and private key.
+		 *
+		 * The certificate and and private key must be PEM formatted.
+		 *
+		 * New clients will see the new certificate.
+		 * Existing clients will continue to see the certificate the server
+		 * was using when they connected to it.
+		 *
+		 * This method throws InvalidInputDataException if any of the
+		 * following errors happen:
+		 *  - Unable to decode the PEM certificate and/or private key.
+		 *  - Unable to decrypt the private key with the given passphrase.
+		 *  - The certificate and/or private key do not contain RSA keys.
+		 *  - The certificate is not usable with the given private key.
+		 */
+		 idempotent void updateCertificate(string certificate, string privateKey, string passphrase) throws ServerBootedException, InvalidSecretException, InvalidInputDataException;
+	};
+
+	/** Callback interface for Meta. You can supply an implementation of this to receive notifications
+	 *  when servers are stopped or started.
+	 *  If an added callback ever throws an exception or goes away, it will be automatically removed.
+	 *  Please note that all callbacks are done asynchronously; murmur does not wait for the callback to
+	 *  complete before continuing processing.
+	 *  @see ServerCallback
+	 *  @see Meta.addCallback
+	 */
+	interface MetaCallback {
+		/** Called when a server is started. The server is up and running when this event is sent, so all methods that 
+		 *  need a running server will work.
+		 *  @param srv Interface for started server.
+		 */
+		void started(Server *srv);
+
+		/** Called when a server is stopped. The server is already stopped when this event is sent, so no methods that
+		 *  need a running server will work.
+		 *  @param srv Interface for started server.
+		 */
+		void stopped(Server *srv);
+	};
+
+	sequence<Server *> ServerList;
+
+	/** This is the meta interface. It is primarily used for retrieving the {@link Server} interfaces for each individual server.
+	 **/
+	["amd"] interface Meta {
+		/** Fetch interface to specific server.
+		 * @param id Server ID. See {@link Server.getId}.
+		 * @return Interface for specified server, or a null proxy if id is invalid.
+		 */
+		idempotent Server *getServer(int id) throws InvalidSecretException;
+
+		/** Create a new server. Call {@link Server.getId} on the returned interface to find it's ID.
+		 * @return Interface for new server.
+		 */
+		Server *newServer() throws InvalidSecretException;
+
+		/** Fetch list of all currently running servers.
+		 * @return List of interfaces for running servers.
+		 */
+		idempotent ServerList getBootedServers() throws InvalidSecretException;
+
+		/** Fetch list of all defined servers.
+		 * @return List of interfaces for all servers.
+		 */
+		idempotent ServerList getAllServers() throws InvalidSecretException;
+
+		/** Fetch default configuraion. This returns the configuration items that were set in the configuration file, or
+		 * the built-in default. The individual servers will use these values unless they have been overridden in the
+		 * server specific configuration. The only special case is the port, which defaults to the value defined here +
+		 * the servers ID - 1 (so that virtual server #1 uses the defined port, server #2 uses port+1 etc).
+		 * @return Default configuration of the servers.
+		 */
+		idempotent ConfigMap getDefaultConf() throws InvalidSecretException;
+
+		/** Fetch version of Murmur. 
+		 * @param major Major version.
+		 * @param minor Minor version.
+		 * @param patch Patchlevel.
+		 * @param text Textual representation of version. Note that this may not match the {@link major}, {@link minor} and {@link patch} levels, as it
+		 *   may be simply the compile date or the SVN revision. This is usually the text you want to present to users.
+		 */
+		idempotent void getVersion(out int major, out int minor, out int patch, out string text);
+
+		/** Add a callback. The callback will receive notifications when servers are started or stopped.
+		 *
+		 * @param cb Callback interface which will receive notifications.
+		 */
+		void addCallback(MetaCallback *cb) throws InvalidCallbackException, InvalidSecretException;
+
+		/** Remove a callback.
+		 *
+		 * @param cb Callback interface to be removed.
+		 */
+		void removeCallback(MetaCallback *cb) throws InvalidCallbackException, InvalidSecretException;
+		
+		/** Get murmur uptime.
+		 * @return Uptime of murmur in seconds
+		 */
+		idempotent int getUptime();
+
+		/** Get slice file.
+		 * @return Contents of the slice file server compiled with.
+		 */
+		idempotent string getSlice();
+
+		/** Returns a checksum dict for the slice file.
+		 * @return Checksum dict
+		 */
+		idempotent Ice::SliceChecksumDict getSliceChecksums();
+	};
+};

mumble/sample.mumbleadmin.ini

@@ -0,0 +1,74 @@
+[MURMUR]
+# This section controls some general settings.
+
+# The host of the Murmur server. This will be used to determine where to connect to
+# for interaction for whichever interface you choose.
+# Examples:
+#   fqdn.domain.tld
+#   127.0.0.1
+#   shorthost
+#   ::1
+host = localhost
+
+# The type of interface to use. Currently, only "ice" and "grpc" are supported.
+# "ice" is the default.
+connection = "ice"
+
+
+[GRPC]
+# The GRPC interface is intended to (potentially) replace the ICE and DBUS interfaces.
+# However, it's currently considered "experimental" - both upstream in Mumble/Murmur,
+# and in this project. It's faster and more secure than Ice, however, if you've
+# enabled TLS transport in your murmur.ini. It requires you to build murmur explicitly
+# with grpc support, however.
+
+# The port GRPC is running on.
+port = 50051
+
+# One of udp or tcp. You probably want to use tcp.
+proto = tcp
+
+# You probably will need to change this.
+# If you need a copy, you can get the most recent at:
+# https://github.com/mumble-voip/mumble/blob/master/src/murmur/MurmurRPC.proto
+# If you leave this empty ("proto = "), we will attempt to fetch the slice from the remote
+# instance ("MURMUR:host" above).
+spec = /usr/local/lib/optools/mumble/murmurRPC.proto
+
+# The maximum size for GRPC Messages (in KB)
+# You're probably fine with the default.
+max_size = 1024
+
+
+[ICE]
+# Ice is on its way out, but is currently the stable interface and most widely
+# supported across versions.
+
+# The port ICE is running on
+port = 6502
+
+# One of udp or tcp. You probably want to use tcp.
+proto = tcp
+
+# You probably will need to change this.
+# If you need a copy, you can get the most recent at:
+# https://github.com/mumble-voip/mumble/blob/master/src/murmur/Murmur.ice
+# If you leave this empty ("slice = "), we will attempt to fetch the slice from the remote
+# instance ("host" above).
+spec = /usr/local/lib/optools/mumble/murmur.ice
+
+# The maximum size for ICE Messages (in KB)
+# You're probably fine with the default.
+max_size = 1024
+
+
+[AUTH]
+# If both read and write are populated, write will be used preferentially.
+
+# The Ice secret for read-only operations.
+# Can be a blank string if you specify a write connection (see below).
+read =
+
+# The Ice secret for read+write operations.
+# Set to a blank string if you want to only make a read-only connection.
+write =

mumble/usermgmt.py

@@ -0,0 +1,453 @@
+#!/usr/bin/env python3
+
+import argparse
+import getpass
+import hashlib
+import os
+import pprint
+import sqlite3
+import subprocess
+import sys
+
+class Manager(object):
+    def __init__(self, args):
+        self.args = args
+        self.conn = self.connect()
+        self.conn.row_factory = sqlite3.Row
+        if 'interactive' not in self.args.keys():
+            self.args['interactive'] = False
+        # Key mappings/types in user_info table; thanks to DireFog in Mumble's IRC channel for help with this.
+        # src/murmur/ServerDB.h, enum UserInfo
+        # 0 = User_Name
+        # 1 = User_Email
+        # 2 = User_Comment
+        # 3 = User_Hash
+        # 4 = User_Password
+        # 5 = User_LastActive
+        # 6 = User_KDFIterations
+        self.infomap = {0: 'name',
+                        1: 'email',
+                        2: 'comment',
+                        3: 'certhash',
+                        4: 'password',
+                        5: 'last_active',
+                        6: 'kdf_iterations'}
+
+    def connect(self):
+        if not os.path.isfile(self.args['database']):
+            raise FileNotFoundError('{0} does not exist! Check your path or create the initial databse by running murmurd.')
+        conn = sqlite3.connect(self.args['database'])
+        return(conn)
+
+    def add(self):
+        # SQLDO("INSERT INTO `%1users`... in src/murmur/ServerDB.cpp
+        if not (self.args['certhash'] or self.args['password']):
+            raise RuntimeError('You must specify either a certificate hash or a method for getting the password.')
+        if self.args['certhash']:  # it's a certificate fingerprint hash
+            _e = '{0} is not a valid certificate fingerprint hash.'.format(self.args['certhash'])
+            try:
+                # Try *really hard* to mahe sure it's a SHA1.
+                # SHA1s are 160 bits in length, in hex (the string representations are
+                # 40 chars). However, we use 162 because of the prefix python3 adds
+                # automatically: "0b".
+                h = int(self.args['certhash'], 16)
+                try:
+                    assert len(bin(h)) == 162
+                except AssertionError:
+                    raise ValueError(_e)
+            except (ValueError, TypeError):
+                raise ValueError(_e)
+        if self.args['password']:  # it's a password
+            if self.args['password'] == 'stdin':
+                self.args['password'] = hashlib.sha1(sys.stdin.read().replace('\n', '').encode('utf-8')).hexdigest().lower()
+            else:
+                _repeat = True
+                while _repeat == True:
+                    _pass_in = getpass.getpass('What password should {0} have (will not echo back)? ')
+                    if not _pass_in or _pass_in == '':
+                        print('Invalid password. Please re-enter: ')
+                    else:
+                        _repeat = False
+                        self.args['password'] = hashlib.sha1(_pass_in.replace('\n', '').encode('utf-8')).hexdigest().lower()
+        # Insert into the "users" table
+        # I spit on the Mumble developers for not using https://sqlite.org/autoinc.html.
+        # Warning: this is kind of dangerous, as you can hit a race condition here.
+        _cur = self.conn.cursor()
+        _cur.execute("SELECT user_id FROM users WHERE server_id = '{0}'".format(self.args['server']))
+        _used_ids = [i[0] for i in _cur.fetchall()]
+        _used_ids2 = [x for x in range(_used_ids[0], _used_ids[-1] + 1)]
+        _avail_uids = list(set(_used_ids) ^ set(_used_ids2))
+        _qinsert = {}
+        _qinsert['lastchannel'] = '0'
+        _qinsert['last_active'] = None  # Change this to '' if it complains
+        _qinsert['texture'] = None  # Change this to '' if it complains
+        _qinsert['uid'] = _avail_uids[0]
+        for k in ('username', 'server', 'password'):
+            _qinsert[k] = self.args[k]
+        for k in _qinsert.keys():
+            if not _qinsert[k]:
+                _qinsert[k] = ''
+        _q = ("INSERT INTO users (server_id, user_id, name, pw, lastchannel, texture, last_active) " +
+              "VALUES ('{server}', '{uid}', '{username}', '{password}', '{lastchannel}', '{texture}'," +
+              "'{last_active}')").format(**_qinsert)
+        _cur.execute(_q)
+        self.conn.commit()
+        # Insert into the "user_info" table
+        for c in ('name', 'email', 'certhash', 'comment'):
+            if self.args[c]:
+                _qinsert = {}
+                _qinsert['server'] = self.args['server']
+                _qinsert['user_id'] = _avail_uids[0]
+                _qinsert['keyid'] = list(self.infomap.keys())[list(self.infomap.values()).index(c)]
+                _qinsert['value'] = self.args[c]
+                _q = ("INSERT INTO user_info (server_id, user_id, key, value) " +
+                      "VALUES ('{server}', '{user_id}', '{keyid}', '{value}')".format(**_qinsert))
+                _cur.execute(_q)
+                self.conn.commit()
+        _cur.close()
+        # Insert into the "group_members" table if we need to
+        if self.args['groups']:
+            # The groups table, thankfully, has autoincrement.
+            for g in self.args['groups']:
+                _ginfo = {}
+                _minsert = {'server': self.args['server'],
+                            'uid': _avail_uids[0],
+                            'addit': 1}
+                _ginsert = {'server': self.args['server'],
+                            'name': g,
+                            'chan_id': 0,
+                            'inherit': 1,
+                            'inheritable': 1}
+                _create = True
+                _cur = self.conn.cursor()
+                _q = "SELECT * FROM groups WHERE server_id = '{0}'".format(self.args['server'])
+                _cur.execute(_q)
+                for r in _cur.fetchall():
+                    if r['name'] == g:
+                        _create = False
+                        _ginfo = r
+                        break
+                if not _ginfo:
+                    create = True  # Just in case...
+                if _create:
+                    _q = ("INSERT INTO groups (server_id, name, channel_id, inherit, inheritable) " +
+                          "VALUES ('{server}', '{name}', '{chan_id}', '{inherit}', '{inheritable}')").format(**_ginsert)
+                    _cur.execute(_q)
+                    self.conn.commit()
+                    _lastins = _cur.lastrowid
+                    _q = ("SELECT * FROM groups WHERE group_id = '{0}' AND server_id = '{1}'").format(_lastins,
+                                                                                                      self.args['server'])
+                    _cur.execute(_q)
+                    _ginfo = _cur.fetchone()
+                _minsert['gid'] = _ginfo['group_id']
+                _q = ("INSERT INTO group_members (group_id, server_id, user_id, addit) " +
+                      "VALUES ('{gid}', '{server}', '{uid}', '{addit}')").format(**_minsert)
+                _cur.execute(_q)
+                self.conn.commit()
+                _cur.close()
+        return()
+
+    def rm(self):
+        _cur = self.conn.cursor()
+        # First we'll need the user's UID.
+        _q = "SELECT user_id FROM users WHERE server_id = '{0}' AND name = '{1}'".format(self.args['server'],
+                                                                                         self.args['username'])
+        _cur.execute(_q)
+        _uid = _cur.fetchone()[0]
+        # Then we get the groups the user's in; we'll need these in a bit.
+        _q = "SELECT group_id FROM group_members WHERE server_id = '{0}' AND user_id = '{0}'".format(self.args['server'],
+                                                                                                     _uid)
+        _cur.execute(_q)
+        _groups = [g[0] for g in _cur.fetchall()]
+        # Okay, now we can delete the user and their metadata...
+        _qtmpl = "DELETE FROM {0} WHERE server_id = '{1}' AND user_id = '{2}'"
+        for t in ('users', 'group_members'):
+            _q = _qtmpl.format(t, self.args['server'], _uid)
+            _cur.execute(_q)
+            self.conn.commit()
+        if not self.args['noprune']:
+            for t in ('user_info', 'acl'):
+                _q = _qtmpl.format(t, self.args['server'], _uid)
+                _cur.execute(_q)
+                self.conn.commit()
+        # Now some groups maintenance.
+        if self.args['prunegrps']:
+            for gid in _groups:
+                _q = ("SELECT COUNT(*) FROM group_members WHERE " +
+                      "server_id = '{0}' AND group_id = '{1}'").format(self.args['server'],
+                                                                       gid)
+                _cur.execute(_q)
+                if _cur.fetchone()[0] == 0:
+                    _q = ("DELETE FROM group_members WHERE " +
+                          "server_id = '{0}' AND group_id = '{1}'").format(self.args['server'],
+                                                                           gid)
+                    _cur.execute(_q)
+                    self.conn.commit()
+        _cur.close()
+        return()
+
+    def lsUsers(self):
+        users = {}
+        _fields = ('server_id', 'user_id', 'name', 'pw', 'lastchannel', 'texture', 'last_active')
+        if self.args['server']:
+            try:
+                self.args['server'] = int(self.args['server'])
+                _q = "SELECT * FROM users WHERE server_id = '{0}'".format(self.args['server'])
+            except (ValueError, TypeError):
+                pass  # It's set as None, which we'll parse to mean as "all" per the --help output.
+        else:
+            _q = 'SELECT * FROM users'
+        _cur = self.conn.cursor()
+        _cur.execute(_q)
+        for r in _cur.fetchall():
+            _usr = r['user_id']
+            users[_usr] = {}
+            for f in _fields:
+                if f != 'user_id':  # We set the dict key as this
+                    users[_usr][f] = r[f]
+            _q = "SELECT * FROM user_info WHERE server_id = '{0}' AND user_id = '{1}'".format(r['server_id'],
+                                                                                              r['user_id'])
+            _cur2 = self.conn.cursor()
+            _cur2.execute(_q)
+            for r2 in _cur2.fetchall():
+                if r2['key'] in self.infomap.keys():
+                    users[_usr][self.infomap[r2['key']]] = r2['value']
+            _cur2.close()
+            for k in self.infomap.keys():
+                if self.infomap[k] not in users[_usr].keys():
+                    users[_usr][self.infomap[k]] = None
+            if users[_usr]['comment']:
+                users[_usr]['comment'] = ('(truncated)' if len(users[_usr]['comment']) >= 32 else users[_usr]['comment'])
+        _cur.close()
+        #pprint.pprint(users)
+        # Now we print (or just return) the results. Whew.
+        if not self.args['interactive']:
+            return(users)
+        print_tmpl = ('{0:6}\t{1:3}\t{2:12} {3:24} {4:40} {5:40} {6:12} ' +
+               '{7:19}   {8:32}')
+        print(print_tmpl.format('Server','UID','Username','Email',
+                                'Password', 'Certhash', 'Last Channel',
+                                'Last Active', 'Comment'), end = '\n\n')
+        for uid in users.keys():
+            d = users[uid]
+            print(print_tmpl.format(int(d['server_id']),
+                                    int(uid),
+                                    str(d['name']),
+                                    str(d['email']),
+                                    str(d['pw']),
+                                    str(d['certhash']),
+                                    (str(d['lastchannel']) if not d['lastchannel'] else int(d['lastchannel'])),
+                                    str(d['last_active']),
+                                    str(d['comment'])))
+        return()
+
+    def lsGroups(self):
+        groups = {}
+        _cur = self.conn.cursor()
+        # First, we get the groups.
+        if self.args['server']:
+            _q = "SELECT * FROM groups WHERE server_id = '{0}'".format(self.args['server'])
+        else:
+            _q = "SELECT * FROM groups"
+        _cur.execute(_q)
+        for r in _cur.fetchall():
+            _gid = r['group_id']
+            groups[_gid] = {'server': r['server_id'],
+                            'name': r['name'],
+                            'chan_id': r['channel_id'],
+                            'inherit': r['inherit'],
+                            'inheritable': r['inheritable']}
+            groups[_gid]['members'] = {}
+            _cur2 = self.conn.cursor()
+            _q2 = "SELECT * FROM group_members WHERE group_id = '{0}' AND server_id = '{1}'".format(_gid,
+                                                                                                    groups[_gid]['server'])
+            _cur2.execute(_q2)
+            for r2 in _cur2.fetchall():
+                # True means they are a member of the group. False means they are excluded from the group.
+                # (Helps override default policies?)
+                groups[_gid]['members'][r2['user_id']] = (True if r2['addit'] else False)
+            _cur2.close()
+        _cur.close()
+        # Return if we're non-interactive...
+        if not self.args['interactive']:
+            return(groups)
+        # Print the groups
+        print('GROUPS:')
+        print_tmpl = ('{0:3}\t{1:16}\t{2:10}\t{3:35}\t{4:30}')
+        print(print_tmpl.format('GID', 'Name', 'Channel ID',
+                                'Inherit Parent Channel Permissions?', 'Allow Sub-channels to Inherit?'), end = '\n\n')
+        for g in groups.keys():
+            d = groups[g]
+            print(print_tmpl.format(g,
+                                    d['name'],
+                                    d['chan_id'],
+                                    str(True if d['inherit'] == 1 else False),
+                                    str(True if d['inheritable'] == 1 else False)))
+        print('\n\nMEMBERSHIPS:')
+        # And print the members
+        print_tmpl = ('\t\t{0:3}\t{1:>19}')  # UID, Include or Exclude?
+        for g in groups.keys():
+            d = groups[g]
+            print('{0} ({1}):'.format(d['name'], g))
+            if d['members']:
+                print(print_tmpl.format('UID', 'Include or Exclude?'), end = '\n\n')
+                for m in d['members'].keys():
+                    print(print_tmpl.format(m, ('Include' if d['members'][m] == 1 else 'Exclude')))
+            else:
+                print('\t\tNo members found; group is empty.')
+        return()
+
+    def edit(self):
+        print('Editing is not currently supported.')
+        return()
+
+    def close(self):
+        self.conn.close()
+        if self.args['operation'] in ('add', 'rm', 'edit'):
+            _cmd = ['systemctl', 'restart', 'murmur']
+            subprocess.run(_cmd)
+        return()
+
+def parseArgs():
+    _db = '/var/lib/murmur/murmur.sqlite'
+    commonargs = argparse.ArgumentParser(add_help = False)
+    reqcommon = commonargs.add_argument_group('REQUIRED common arguments')
+    reqcommon.add_argument('-u',
+                           '--user',
+                           type = str,
+                           dest = 'username',
+                           required = True,
+                           help = 'The username to perform the action for.')
+    reqcommon.add_argument('-s',
+                           '--server',
+                           type = int,
+                           dest = 'server',
+                           default = 1,
+                           help = 'The server ID. Defaults to \033[1m{0}\033[0m'.format(1))
+    commonargs.add_argument('-d',
+                            '--database',
+                            type = str,
+                            dest = 'database',
+                            metavar = '/path/to/murmur.sqlite3',
+                            default = _db,
+                            help = 'The path to the sqlite3 database for Murmur. Default: \033[1m{0}\033[0m'.format(_db))
+    args = argparse.ArgumentParser(epilog = 'This program has context-sensitive help (e.g. try "... add --help")')
+    subparsers = args.add_subparsers(help = 'Operation to perform',
+                                     dest = 'operation')
+    addargs = subparsers.add_parser('add',
+                                    parents = [commonargs],
+                                    help = 'Add a user to the Murmur database')
+    delargs = subparsers.add_parser('rm',
+                                    parents = [commonargs],
+                                    help = 'Remove a user from the Murmur database')
+    listargs = subparsers.add_parser('ls',
+                                     help = 'List users in the Murmur database')
+    editargs = subparsers.add_parser('edit',
+                                     parents = [commonargs],
+                                     help = 'Edit a user in the Murmur database')
+    # Operation-specific optional arguments
+    addargs.add_argument('-n',
+                         '--name',
+                         type = str,
+                         metavar = '"Firstname Lastname"',
+                         dest = 'name',
+                         default = None,
+                         help = 'The new user\'s (real) name')
+    addargs.add_argument('-c',
+                         '--comment',
+                         type = str,
+                         metavar = '"This comment becomes the user\'s profile."',
+                         dest = 'comment',
+                         default = None,
+                         help = 'The comment for the new user')
+    addargs.add_argument('-e',
+                         '--email',
+                         type = str,
+                         metavar = 'email@domain.tld',
+                         dest = 'email',
+                         default = None,
+                         help = 'The email address for the new user')
+    addargs.add_argument('-C',
+                         '--certhash',
+                         type = str,
+                         metavar = 'CERTIFICATE_FINGERPRINT_HASH',
+                         default = None,
+                         dest = 'certhash',
+                         help = ('The certificate fingerprint hash. See genfprhash.py. ' +
+                                 'If you do not specify this, you must specify -p/--passwordhash'))
+    addargs.add_argument('-p',
+                         '--passwordhash',
+                         type = str,
+                         dest = 'password',
+                         choices = ['stdin', 'prompt'],
+                         default = None,
+                         help = ('If not specified, you must specify -C/--certhash. Otherwise, either ' +
+                                 '\'stdin\' (the password is being piped into this program) or \'prompt\' ' +
+                                 '(a password will be asked for in a non-echoing prompt). "prompt" is much more secure and recommended.'))
+    addargs.add_argument('-g',
+                         '--groups',
+                         type = str,
+                         metavar = 'GROUP1(,GROUP2,GROUP3...)',
+                         default = None,
+                         help = ('A comma-separated list of groups the user should be added to. If a group ' +
+                                 'doesn\'t exist, it will be created'))
+    # Listing should only take the DB as the "common" arg
+    listargs.add_argument('-g',
+                          '--groups',
+                          action = 'store_true',
+                          dest = 'groups',
+                          help = 'If specified, list groups (and their members), not users')
+    listargs.add_argument('-s',
+                           '--server',
+                           type = str,
+                           dest = 'server',
+                           default = None,
+                           help = 'The server ID. Defaults to all servers. Specify one by the numerical ID.')
+    listargs.add_argument('-d',
+                          '--database',
+                          type = str,
+                          dest = 'database',
+                          metavar = '/path/to/murmur.sqlite3',
+                          default = _db,
+                          help = 'The path to the sqlite3 database for Murmur. Default: \033[1m{0}\033[0m'.format(_db))
+    # Deleting args
+    delargs.add_argument('-n',
+                         '--no-prune',
+                         dest = 'noprune',
+                         action = 'store_true',
+                         help = ('If specified, do NOT remove the ACLs and user info for the user as well (profile, ' +
+                                 'certificate fingerprint, etc.)'))
+    delargs.add_argument('-P',
+                         '--prune-groups',
+                         dest = 'prunegrps',
+                         action = 'store_true',
+                         help = 'If specified, remove any groups the user was in that are now empty (i.e. the user was the only member)')
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    if not args['operation']:
+        #raise RuntimeError('You must specify an operation to perform. Try running with -h/--help.')
+        exit('You must specify an operation to perform. Try running with -h/--help.')
+    args['interactive'] = True
+    #pprint.pprint(args)
+    mgmt = Manager(args)
+    if args['operation'] == 'add':
+        if args['groups']:
+            mgmt.args['groups'] = [g.strip() for g in args['groups'].split(',')]
+        mgmt.add()
+    elif args['operation'] == 'rm':
+        mgmt.rm()
+    elif args['operation'] == 'ls':
+        if not args['groups']:
+            mgmt.lsUsers()
+        else:
+            mgmt.lsGroups()
+    elif args['operation'] == 'edit':
+        mgmt.edit()
+    else:
+        pass  # No-op because something went SUPER wrong.
+    mgmt.close()
+
+if __name__ == '__main__':
+    main()

mumble/usrmgmt2.py

@@ -0,0 +1,382 @@
+#!/usr/bin/env python3
+# Thanks to https://github.com/alfg/murmur-rest/blob/master/app/__init__.py
+
+import argparse
+from collections import defaultdict
+import configparser
+import datetime
+import email.utils
+import getpass
+import hashlib
+import Ice  # python-zeroc-ice in AUR
+import IcePy  # python-zeroc-ice in AUR
+import getpass
+import os
+import re
+import sys
+import tempfile
+
+
+class IceMgr(object):
+    def __init__(self, args):
+        self.args = args
+        if 'interactive' not in self.args.keys():
+            self.args['interactive'] = False
+        if self.args['verbose']:
+            import pprint
+        self.getCfg()
+        if self.cfg['MURMUR']['connection'] == '':
+            self.cfg['MURMUR']['connection'] == 'ice'
+        self.connect(self.cfg['MURMUR']['connection'])
+
+    def getCfg(self):
+        _cfg = os.path.join(os.path.abspath(os.path.expanduser(self.args['cfgfile'])))
+        if not os.path.isfile(_cfg):
+            raise FileNotFoundError('{0} does not exist!'.format(_cfg))
+            return()
+        _parser = configparser.ConfigParser()
+        _parser._interpolation = configparser.ExtendedInterpolation()
+        _parser.read(_cfg)
+        self.cfg = defaultdict(dict)
+        for section in _parser.sections():
+            self.cfg[section] = {}
+            for option in _parser.options(section):
+                self.cfg[section][option] = _parser.get(section, option)
+        return()
+
+    def connect(self, ctxtype):
+        ctxtype = ctxtype.strip().upper()
+        if ctxtype.lower() not in ('ice', 'grpc'):
+            raise ValueError('You have specified an invalid connection type.')
+        _cxcfg = self.cfg[ctxtype]
+        self.cfg[ctxtype]['spec'] = os.path.join(os.path.abspath(os.path.expanduser(self.cfg[ctxtype]['spec'])))
+        # ICE START
+        _props = {'ImplicitContext': 'Shared',
+                  'Default.EncodingVersion': '1.0',
+                  'MessageSizeMax': str(self.cfg['ICE']['max_size'])}
+        _prop_data = Ice.createProperties()
+        for k, v in _props.items():
+            _prop_data.setProperty('Ice.{0}'.format(k), v)
+        _conn = Ice.InitializationData()
+        _conn.properties = _prop_data
+        self.ice = Ice.initialize(_conn)
+        _host = 'Meta:{0} -h {1} -p {2} -t 1000'.format(self.cfg['ICE']['proto'],
+                                                        self.cfg['ICE']['host'],
+                                                        self.cfg['ICE']['port'])
+        _ctx = self.ice.stringToProxy(_host)
+        # I owe a lot of neat tricks here to:
+        # https://raw.githubusercontent.com/mumble-voip/mumble-scripts/master/Helpers/mice.py
+        # Namely, the load-slice-from-server stuff especially
+        _slicedir = Ice.getSliceDir()
+        if not _slicedir:
+            _slicedir = ["-I/usr/share/Ice/slice", "-I/usr/share/slice"]
+        else:
+            _slicedir = ['-I' + _slicedir]
+        if self.cfg['ICE']['slice'] == '':
+            if IcePy.intVersion() < 30500:
+                # Old 3.4 signature with 9 parameters
+                _op = IcePy.Operation('getSlice',
+                                      Ice.OperationMode.Idempotent,
+                                      Ice.OperationMode.Idempotent,
+                                      True,
+                                      (), (), (),
+                                      IcePy._t_string, ())
+            else:
+                # New 3.5 signature with 10 parameters.
+                _op = IcePy.Operation('getSlice',
+                                      Ice.OperationMode.Idempotent,
+                                      Ice.OperationMode.Idempotent,
+                                      True,
+                                      None,
+                                      (), (), (),
+                                      ((), IcePy._t_string, False, 0),
+                                      ())
+            _slice = _op.invoke(_ctx,
+                                ((), None))
+            (_filedesc, _filepath)  = tempfile.mkstemp(suffix = '.ice')
+            _slicefile = os.fdopen(_filedesc, 'w')
+            _slicefile.write(_slice)
+            _slicefile.flush()
+            Ice.loadSlice('', _slicedir + [_filepath])
+            _slicefile.close()
+            os.remove(_filepath)
+        else:  # A .ice file was explicitly defined in the cfg
+            _slicedir.append(self.cfg[ctxtype]['spec'])
+            Ice.loadSlice('', _slicedir)
+        import Murmur
+        self.conn = {}
+        if self.cfg['AUTH']['read'] != '':
+            _secret = self.ice.getImplicitContext().put("secret",
+                                                        self.cfg['AUTH']['read'])
+            self.conn['read'] = Murmur.MetaPrx.checkedCast(_ctx)
+        else:
+            self.conn['read'] = False
+        if self.cfg['AUTH']['write'] != '':
+            _secret = self.ice.getImplicitContext().put("secret",
+                                                        self.cfg['AUTH']['write'])
+            self.conn['write'] = Murmur.MetaPrx.checkedCast(_ctx)
+        else:
+            self.conn['write'] = False
+        return()
+
+    def dictify(self, obj):
+        # Thanks to:
+        # https://github.com/alfg/murmur-rest/blob/master/app/utils.py
+        # (Modified to be python 3 compatible)
+        _rv = {'_type': str(type(obj))}
+        if type(obj) in (bool, int, float, str, bytes):
+            return(obj)
+        if type(obj) in (list, tuple):
+            return([dictify(i) for i in obj])
+        if type(obj) == dict:
+            return(dict((str(k), dictify(v)) for k, v in obj.items()))
+        return(dictify(obj.__dict__))
+
+    def add(self):
+        _userinfo = {Murmur.UserInfo.UserName: self.args['UserName']}
+        if not self.conn['write']:
+            raise PermissionError('You do not have write access configured!')
+        if not (self.args['certhash'] or self.args['password']):
+            raise RuntimeError(('You must specify either a certificate hash ' +
+                                 'or a method for getting the password.'))
+        if self.args['certhash']:  # it's a certificate fingerprint hash
+            _e = '{0} is not a valid certificate fingerprint hash.'.format(self.args['certhash'])
+            try:
+                # Try *really hard* to mahe sure it's a SHA1.
+                # SHA1s are 160 bits in length, in binary representation.
+                # (the string representations are 40 chars in hex).
+                # However, we use 161 because of the prefix python3 adds
+                # automatically: "0b". I know. "This should be 162!" Shut up, trust me.
+                # Change it to 162 and watch it break if you don't believe me.
+                h = int(self.args['certhash'], 16)
+                try:
+                    assert len(bin(h)) == 161
+                    #_userinfo[Murmur.UserInfo.UserPassword] = None
+                    _userinfo[Murmur.UserInfo.UserHash] = self.args['UserHash']
+                except AssertionError:
+                    raise ValueError(_e)
+            except (ValueError, TypeError):
+                raise ValueError(_e)
+        if self.args['UserPassword']:  # it's a password
+            if self.args['UserPassword'] == 'stdin':
+                #self.args['password'] = hashlib.sha1(sys.stdin.read().replace('\n', '').encode('utf-8')).hexdigest().lower()
+                _userinfo[Murmur.UserInfo.UserPassword] = sys.stdin.read().replace('\n', '').encode('utf-8')
+                #_userinfo[Murmur.UserInfo.UserHash] = None
+            else:
+                _repeat = True
+                while _repeat == True:
+                    _pass_in = getpass.getpass('What password should {0} have (will not echo back)? '.format(self.args['UserName']))
+                    if not _pass_in or _pass_in == '':
+                        print('Invalid password. Please re-enter: ')
+                    else:
+                        _repeat = False
+                        #self.args['password'] = hashlib.sha1(_pass_in.replace('\n', '').encode('utf-8')).hexdigest().lower()
+                        _userinfo[Murmur.UserInfo.UserPassword] = _pass_in.replace('\n', '').encode('utf-8')
+                        #_userinfo[Murmur.UserInfo.UserHash] = None
+        # Validate the email address
+        if self.args['UserEmail']:
+            _email = email.utils.parseaddr(self.args['UserEmail'])
+            # This is a stupidly simplified regex. For reasons why, see:
+            # https://stackoverflow.com/questions/8022530/python-check-for-valid-email-address
+            # http://www.regular-expressions.info/email.html
+            # TL;DR: email is really fucking hard to regex against,
+            # even (especially) if you follow RFC5322, and I don't want to have
+            # to rely on https://pypi.python.org/pypi/validate_email
+            if not re.match('[^@]+@[^@]+\.[^@]+', _email[1]):
+                raise ValueError('{0} is not a valid email address!'.format(self.args['UserEmail']))
+            else:
+                _userinfo[Murmur.UserInfo.UserEmail] = _email[1]
+        #else:
+        #    _userinfo[Murmur.UserInfo.UserEmail] = None
+        if self.args['UserComment']:
+            _userinfo[Murmur.UserInfo.UserComment] = self.args['UserComment']
+        # Set a dummy LastActive
+        _userinfo[Murmur.UserInfo.LastActive] = str(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
+        # Now we Do the Thing(TM)
+        _server = self.conn['write'].getServer(self.args['server'])
+        _regid = _server.registerUser(_userinfo)
+        # And a little more Doing the Thing(TM), add groups.
+        # This is... a little convoluted.
+        # See https://sourceforge.net/p/mumble/discussion/492607/thread/579de8f9/
+        if args['groups']:
+            # First we get the ACL listings. The groups are *actually stored in
+            # the ACLs*, which is... insane to me, sort of, but whatever.
+            _acl = _server.getACL()
+            # Then build a dict of all groups to assign.
+            _groups = {}
+            for g in self.args['groups'].split(','):
+                _g = g.strip().split(':')
+                if _g[0] not in _groups.keys():
+                    _groups[_g[0]] = [g[1]]
+                else:
+                    _groups[_g[0]].append(_g[1])
+            # Now we need to see which groups currently exist and which down't.
+        if sys.stdout.isatty():
+            print('Added user {0} (UID: {1})'.format(self.args['UserName'],
+                                                     _regid))
+            if self.args['verbose']:
+                _u = _server.getRegistration(_regid)
+                pprint.pprint(self.dictify(_u))
+
+        return()
+
+    def rm(self):
+        pass
+
+    def lsUsers(self):
+        pass
+
+    def lsGroups(self):
+        pass
+
+    def edit(self):
+        pass
+
+    def status(self):
+        # https://github.com/alfg/murmur-rest/blob/master/app/api.py#L71
+        pass
+
+    def close(self):
+        self.ice.destroy()
+        if self.cfg['TUNNEL']['enable'].lower() in ('', 'true'):
+            self.ssh.stop()
+            self.ssh.close()
+        return()
+
+def parseArgs():
+    _cfgfile = os.path.abspath(os.path.join(os.path.expanduser('~'),
+                                            '.config',
+                                            'optools',
+                                            'mumbleadmin.ini'))
+    commonargs = argparse.ArgumentParser(add_help = False)
+    reqcommon = commonargs.add_argument_group('REQUIRED common arguments')
+    optcommon = argparse.ArgumentParser(add_help = False)
+    reqcommon.add_argument('-u', '--user',
+                           type = str,
+                           dest = 'UserName',
+                           required = True,
+                           help = 'The username to perform the action for.')
+    reqcommon.add_argument('-s', '--server',
+                           type = int,
+                           dest = 'server',
+                           default = 1,
+                           help = ('The server ID. ' +
+                                   'Defaults to \033[1m{0}\033[0m').format(1))
+    optcommon.add_argument('-f', '--config',
+                           type = str,
+                           dest = 'cfgfile',
+                           metavar = '/path/to/mumbleadmin.ini',
+                           default = _cfgfile,
+                           help = ('The path to the configuration file ' +
+                                   '("mumleadmin.ini"). Default: \033[1m{0}\033[0m').format(_cfgfile))
+    optcommon.add_argument('-v', '--verbose',
+                           dest = 'verbose',
+                           action = 'store_true',
+                           help = ('If specified, print more information than normal'))
+    args = argparse.ArgumentParser(epilog = 'This program has context-sensitive help (e.g. try "... add --help")')
+    subparsers = args.add_subparsers(help = 'Operation to perform',
+                                     dest = 'operation')
+    addargs = subparsers.add_parser('add',
+                                    parents = [commonargs, optcommon],
+                                    help = 'Add a user to the Murmur database')
+    delargs = subparsers.add_parser('rm',
+                                    parents = [commonargs, optcommon],
+                                    help = 'Remove a user from the Murmur database')
+    listargs = subparsers.add_parser('ls',
+                                     parents = [optcommon],
+                                     help = 'List users in the Murmur database')
+    editargs = subparsers.add_parser('edit',
+                                     parents = [commonargs, optcommon],
+                                     help = 'Edit a user in the Murmur database')
+    # Operation-specific optional arguments
+    # Why did I even add this? It's not used *anywhere*.
+    #addargs.add_argument('-n', '--name',
+    #                     type = str,
+    #                     metavar = '"Firstname Lastname"',
+    #                     dest = 'name',
+    #                     default = None,
+    #                     help = 'The new user\'s (real) name')
+    addargs.add_argument('-c', '--comment',
+                         type = str,
+                         metavar = '"This comment becomes the user\'s profile."',
+                         dest = 'UserComment',
+                         default = None,
+                         help = 'The comment for the new user')
+    addargs.add_argument('-e', '--email',
+                         type = str,
+                         metavar = 'email@domain.tld',
+                         dest = 'UserEmail',
+                         default = None,
+                         help = 'The email address for the new user')
+    addargs.add_argument('-C', '--certhash',
+                         type = str,
+                         metavar = 'CERTIFICATE_FINGERPRINT_HASH',
+                         default = None,
+                         dest = 'UserHash',
+                         help = ('The certificate fingerprint hash. See gencerthash.py. ' +
+                                 'This is the preferred way. ' +
+                                 'If you do not specify this, you must specify -p/--passwordhash'))
+    addargs.add_argument('-p', '--password',
+                         type = str,
+                         dest = 'UserPassword',
+                         choices = ['stdin', 'prompt'],
+                         default = None,
+                         help = ('If not specified, you must specify -C/--certhash. Otherwise, either ' +
+                                 '\'stdin\' (the password is being piped into this program) or \'prompt\' ' +
+                                 '(a password will be asked for in a non-echoing prompt). "prompt" is much more secure and recommended.'))
+    addargs.add_argument('-g', '--groups',
+                         type = str,
+                         metavar = 'CHANID:GROUP1(,CHANID:GROUP2,CHANID:GROUP3...)',
+                         default = None,
+                         help = ('A comma-separated list of groups the user should be added to. If a group ' +
+                                 'doesn\'t exist, it will be created. CHANID is a ' +
+                                 'numerical ID of the channel to assign the group to. ' +
+                                 '(You can get channel IDs by doing "... ls -gv".) ' +
+                                 'If no CHANID is provided, the root channel (0) will be used.'))
+    # Listing should only take the DB as the "common" arg
+    listargs.add_argument('-g', '--groups',
+                          action = 'store_true',
+                          dest = 'groups',
+                          help = 'If specified, list groups (and their members), not users')
+    listargs.add_argument('-s', '--server',
+                          type = str,
+                          dest = 'server',
+                          default = None,
+                          help = 'The server ID. Defaults to all servers. Specify one by the numerical ID.')
+    # Deleting args
+    delargs.add_argument('-n', '--no-prune',
+                         dest = 'noprune',
+                         action = 'store_true',
+                         help = ('If specified, do NOT remove the ACLs and user info for the user as well (profile, ' +
+                                 'certificate fingerprint, etc.)'))
+    delargs.add_argument('-P', '--prune-groups',
+                         dest = 'prunegrps',
+                         action = 'store_true',
+                         help = ('If specified, remove any groups the user was in ' +
+                                 'that are now empty (i.e. the user was the only member)'))
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    if not args['operation']:
+        #raise RuntimeError('You must specify an operation to perform. Try running with -h/--help.')
+        exit('You must specify an operation to perform. Try running with -h/--help.')
+    args['interactive'] = True
+    mgmt = IceMgr(args)
+    if args['operation'] == 'add':
+        mgmt.add()
+    elif args['operation'] == 'rm':
+        mgmt.rm()
+    elif args['operation'] == 'ls':
+        if not args['groups']:
+            mgmt.lsUsers()
+        else:
+            mgmt.lsGroups()
+    elif args['operation'] == 'edit':
+        mgmt.edit()
+    else:
+        pass  # No-op because something went SUPER wrong.
+    mgmt.close()
+
+if __name__ == '__main__':
+    main()

mysql/tblinfo.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+
+import argparse
+import configparser
+import copy
+import os
+import pymysql
+
+mysql_internal = ['information_schema', 'mysql']
+
+# Not used, but could be in the future.
+stat_hdrs = ['Name', 'Engine', 'Version', 'Row_format', 'Rows', 'Avg_row_length', 'Data_length',
+             'Max_data_length', 'Index_length', 'Data_free', 'Auto_increment', 'Create_time',
+             'Update_time', 'Check_time', 'Collation', 'Checksum', 'Create_options', 'Comment']
+tblinfo_hdrs = ['Field', 'Type', 'Null', 'Key', 'Default', 'Extra']
+
+def get_info(db, internal = False):
+    dbs = {}
+    if os.path.isfile(os.path.expanduser('~/.my.cnf')):
+        _cfg = configparser.ConfigParser(allow_no_value = True)
+        _cfg.read(os.path.expanduser('~/.my.cnf'))
+        _cfg = dict(_cfg['client'])
+        _cfg['ssl'] = {}
+        if 'host' not in _cfg:
+            _cfg['host'] = 'localhost'
+        conn = pymysql.connect(**_cfg, cursorclass = pymysql.cursors.DictCursor)
+    else:
+        raise RuntimeError('Need mysql creds at ~/.my.cnf')
+    cur = conn.cursor()
+    if not db:
+        cur.execute("SHOW DATABASES")
+        db = [row['Database'] for row in cur.fetchall()]
+        if not internal:
+            for d in mysql_internal:
+                try:
+                    db.remove(d)
+                except ValueError:  # Not in the list; our user probably doesn't have access
+                    pass
+    else:
+        db = [db]
+    for d in db:
+        dbs[d] = {}
+        cur.execute("SHOW TABLES FROM `{0}`".format(d))
+        for tbl in [t['Tables_in_{0}'.format(d)] for t in cur.fetchall()]:
+            dbs[d][tbl] = {}
+            # Status
+            cur.execute("SHOW TABLE STATUS FROM `{0}` WHERE Name = %s".format(d), (tbl, ))
+            dbs[d][tbl]['_STATUS'] = copy.deepcopy(cur.fetchone())
+            # Columns
+            dbs[d][tbl]['_COLUMNS'] = {}
+            #cur.execute("DESCRIBE {0}.{1}".format(d, tbl))
+            cur.execute("SHOW COLUMNS IN `{0}` FROM `{1}`".format(tbl, d))
+            for row in cur.fetchall():
+                colNm = row['Field']
+                dbs[d][tbl]['_COLUMNS'][colNm] = {}
+                for k in [x for x in tblinfo_hdrs if x is not 'Field']:
+                    dbs[d][tbl]['_COLUMNS'][colNm][k] = row[k]
+    cur.close()
+    conn.close()
+    return(dbs)
+
+def parseArgs():
+    args = argparse.ArgumentParser()
+    args.add_argument('-i', '--internal',
+                      dest = 'internal',
+                      action = 'store_true',
+                      help = ('If specified, include the MySQL internal databases '
+                              '(mysql, information_schema, etc.); only used if -d is not specified'))
+    args.add_argument('-d', '--database',
+                      dest = 'db',
+                      default = None,
+                      help = 'If specified, only list table info for this DB')
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    dbs = get_info(args['db'], internal = args['internal'])
+    #import json
+    #print(json.dumps(dbs, indent = 4, sort_keys = True, default = str))
+    import pprint
+    pprint.pprint(dbs)
+
+if __name__ == '__main__':
+    main()

net/addr/TODO

@@ -0,0 +1 @@
+We can get more in-depth: https://danidee10.github.io/2016/09/24/flask-by-example-3.html

net/addr/app/__init__.py

@@ -0,0 +1,7 @@
+from flask import Flask
+
+app = Flask(__name__, instance_relative_config=True)
+
+from app import views
+
+app.config.from_object('config')

net/addr/app/dnsinfo.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+# https://gist.github.com/akshaybabloo/2a1df455e7643926739e934e910cbf2e
+
+import ipaddress
+import dns  # apacman -S python-dnspython
+import ipwhois  # apacman -S python-ipwhois
+import whois  # apacman -S python-ipwhois
+
+class netTarget(object):
+    def __init__(self, target):
+        self.target = target
+
+
+##!/usr/bin/env python3
+#
+#import pprint
+#import dns
+#import whois
+#import ipwhois
+#
+#d = 'sysadministrivia.com'  # A/AAAA
+#d = 'autoconfig.sysadministrivia.com'  # CNAME
+#
+#records = {'whois': None,
+#           'ptr': None,
+#           'allocation': None}
+#
+#def getWhois(domain):
+#    _w = whois.whois(d)
+#    records['whois'] = dict(_w)
+#    return()
+#
+#def getIps(domain):
+#    addrs = []
+#    for t in ('A', 'AAAA'):
+#        answers = dns.resolver.query(domain, t)
+#        for a in answers:
+#            try:
+#                addrs.append(a.address)
+#            except:
+#                pass
+#    return(addrs)
+#
+#def getPtr(addrs):
+#    for a in addrs:
+#        pass
+#
+#print(getIps(d))
+##pprint.pprint()

net/addr/app/templates/about.html

@@ -0,0 +1,8 @@
+{% extends "base.html" %}{% block title %}r00t^2 Client Info Revealer || About{% endblock %}{% block body %}<div class="jumbotron">
+<h1>About</h1></div>
+<p>This is a tool to reveal certain information about your connection that the server sees. Note that all of this information you see is <i>sent by your client</i>; there was no probing/scanning or the like done from the server this site is hosted on.</p>
+<p>If you don't like this info being available to server administrators of the websites you visit you may want to consider <a href="https://getfoxyproxy.org/">hiding your client IP address</a><sup><a href="#0">0</a></sup> and/or <a href="https://panopticlick.eff.org/self-defense">hiding your browser's metadata</a>, which can be done via browser plugins such as <a href="https://www.eff.org/privacybadger">Privacy Badger</a>, {{ '<a href="https://addons.mozilla.org/en-US/firefox/addon/modify-headers/">Modify Headers</a>, '|safe if request.user_agent.browser == 'firefox' else '' }}<a href="https://www.requestly.in/">Requestly</a>, and others.</p>
+<p>If you would like to view the <i>server</i> headers, then you can use a service such as <a href="https://securityheaders.io">SecurityHeaders.io</a> (or use the <b><code>curl -i</code></b> command in *Nix operating systems).</p>
+<br />
+<p><a name="0"></a><b>[0]</b> Disclosure: I am an engineer for this company.</p>
+{% endblock %}

net/addr/app/templates/base.html

@@ -0,0 +1,35 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <title>{% block title %}{% endblock %}</title>
+  <!-- Bootstrap core CSS -->
+  <!-- Thanks, https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xii-facelift and
+        https://scotch.io/tutorials/getting-started-with-flask-a-python-microframework -->
+  <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet">
+  <!--<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta/css/bootstrap.min.css" integrity="sha384-/Y6pD6FV/Vv2HJnA6t+vslU6fwYXjCFtcEpHbNJ0lyAFsXTsjBbfaDjzALeQsN6M" crossorigin="anonymous">-->
+  <!-- Custom styles for this template -->
+  <link href="https://getbootstrap.com/examples/jumbotron-narrow/jumbotron-narrow.css" rel="stylesheet">
+  <!--<link href="https://getbootstrap.com/docs/4.0/examples/offcanvas/offcanvas.css" rel="stylesheet">-->
+</head>
+<body>
+  <div class="container">
+    <div class="header clearfix">
+      <nav>
+        <ul class="nav nav-pills pull-right">
+          <li role="presentation"><a href="/">Home</a></li>
+          <li role="presentation"><a href="/about">About</a></li>
+          <li role="presentation"><a href="/usage">Usage</a></li>
+          <!-- the following opens in a new tab/window/whatever. the line after opens in the same tab/window/etc. -->
+          <!-- <li role="presentation"><a href="https://square-r00t.net/" target="_blank">r00t^2</a></li> -->
+          <li role="presentation"><a href="https://square-r00t.net/">r00t^2</a></li>
+        </ul>
+      </nav>
+    </div>
+    {% block body %}{% endblock %}
+    <footer class="footer">
+      <p><sub>The code for this page is released under the <a href="https://www.gnu.org/licenses/gpl-3.0.en.html#content">GPL 3.0 License</a>. It can be found <a href="https://git.square-r00t.net/OpTools/tree/net/addr/">here</a>.</sub></p>
+    </footer>
+  </div>
+  <!-- /container -->
+</body>
+</html>

net/addr/app/templates/html.html

@@ -0,0 +1,38 @@
+<h2>Client/Browser Information</h2>
+<p>This is information that your browser sends with its connection.</p>
+<p>
+  <ul>
+      <li><b>Client IP:</b> <a href="https://ipinfo.io/{{ visitor['ip'] }}">{{ visitor['ip'] }}</a></li>
+      <li><b>Browser:</b> {{ '<a href="{0}">{1}</a>'.format(browsers[visitor['client']['browser']][0],
+                                                            browsers[visitor['client']['browser']][1])|safe
+                              if visitor['client']['browser'] in browsers.keys()
+                              else visitor['client']['browser'].title()
+                                if visitor['client']['browser'] is not none
+                                else '(N/A)' }}</li>
+    <li><b>Language/Locale:</b> {{ visitor['client']['language'] or '(N/A)' }}</li>
+{%- set alt_os = alts[visitor['client']['os']] if visitor['client']['os'] in alts.keys() else '' %}
+      <li><b>Operating System:</b> {{ '<a href="{0}">{1}</a>{2}'.format(os[visitor['client']['os']][0],
+                                                                        os[visitor['client']['os']][1],
+                                                                        alt_os)|safe
+                                      if visitor['client']['os'] in os.keys()
+                                      else visitor['client']['os'].title()
+                                        if visitor['client']['os'] is not none
+                                        else '(N/A)' }}</li>
+      <li><b>User Agent:</b> {{ visitor['client']['str'] }}</li>
+      <li><b>Version:</b> {{ visitor['client']['version'] or '(N/A)' }}</li>
+  </ul>
+</p>
+<h2>Request Headers</h2>
+<p>These are headers sent along with the request your browser sends for the page's content.</p>
+<p>
+  <table>
+    <tr>
+      <th>Field</th>
+      <th>Value</th>
+    </tr>{% for k in visitor['headers'].keys()|sort(case_sensitive = True) %}
+    <tr>
+      <td>{{ k }}</td>
+      <td>{{ visitor['headers'][k] if visitor['headers'][k] != '' else '(N/A)' }}</td>
+    </tr>{% endfor %}
+  </table>
+</p>

net/addr/app/templates/index.html

@@ -0,0 +1,6 @@
+{% extends "base.html" %}{% block title %}r00t^2 Client Info Revealer{% endblock %}{% block body %}<div class="jumbotron">
+  <h1>Client Info Revealer</h1>
+  <p class="lead">A tool to reveal client-identifying data sent to webservers</p>
+</div>
+{% include 'html.html' if not params['json'] else 'json.html' %}
+{% endblock %}

net/addr/app/templates/json.html

@@ -0,0 +1 @@
+<pre>{{ json }}</pre>

net/addr/app/templates/usage.html

@@ -0,0 +1,51 @@
+{% extends "base.html" %}{% block title %}r00t^2 Client Info Revealer || Usage{% endblock %}{% block body %}<div class="jumbotron">
+<h1>Usage</h1></div>
+<h2>Parameters</h2>
+<p>You can control how this page displays/renders. By default it will try to "guess" what you want; e.g. if you access it in Chrome, it will display this page but if you fetch via Curl, you'll get raw JSON. The following parameters control this behavior.</p>
+<p><i><b>Note:</b> "Enabled" parameter values can be one of <b>y</b>, <b>yes</b>, <b>1</b>, or <b>true</b>. "Disabled" parameter values can be one of <b>n</b>, <b>no</b>, <b>0</b>, or <b>false</b>. The parameter names are case-sensitive but the values are not.</i></p>
+<p><ul>
+  <li><b>json:</b> Force rendering in JSON format
+    <ul>
+      <li>It will display it nicely if you're in a browser, otherwise it will return raw/plaintext JSON.</li>
+      <li>Use <b>raw</b> if you want to force raw plaintext JSON output.</li>
+    </ul></li>
+  <li><b>html:</b> Force rendering in HTML
+  <ul>
+    <li>It will render HTML in clients that would normally render as JSON (e.g. curl, wget).</li>
+  </ul></li>
+  <li><b>raw:</b> Force output into a raw JSON string 
+    <ul>
+      <li>Pure JSON instead of HTML or formatted JSON. This is suitable for API usages if your client is detected wrongly (or you just want to get the raw JSON).</li>
+      <li>Overrides all other tags.</li>
+      <li>Has no effect for clients that would normally render as JSON (curl, wget, etc.).</li>
+    </ul></li>
+  <li><b>tabs:</b> Indentation for JSON output
+    <ul>
+      <li>Accepts a positive integer.</li>
+      <li>Default is 4 for "desktop" browsers (if <b>json</b> is enabled), and no indentation otherwise.</li>
+    </ul></li>
+</ul></p>
+<h2>Examples</h2>{% set scheme = 'https' if request.is_secure else 'http'%}
+<p><table>
+  <tr>
+    <th>URL</th>
+    <th>Behavior</th>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/">{{ scheme }}://{{ request.headers['host'] }}/</a></td>
+    <td>Displays HTML and "Human" formatting if in a graphical browser, otherwise returns a raw, unformatted JSON string.</td>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/?raw=1">{{ scheme }}://{{ request.headers['host'] }}/?raw=1</a></td>
+    <td>Renders a raw, unformatted JSON string if in a graphical browser, otherwise no effect. All other parameters ignored (if in a graphical browser).</td>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/?html=1">{{ scheme }}://{{ request.headers['host'] }}/?html=1</a></td>
+    <td>Forces HTML rendering on non-graphical clients.</td>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/?json=1&tabs=4">{{ scheme }}://{{ request.headers['host'] }}/?json=1&tabs=4</a></td>
+    <td>Returns JSON indented by 4 spaces for each level (you can leave "json=1" off if it's in a non-graphical browser, unless you specified "html=1").</td>
+  </tr>
+</table></p>
+{% endblock %}

net/addr/app/views.py

@@ -0,0 +1,101 @@
+import json
+import re
+from flask import render_template, make_response, request
+from app import app
+
+@app.route('/', methods = ['GET'])  #@app.route('/')
+def index():
+    # First we define interactive browsers
+    _intbrowsers = {'camino': ['http://caminobrowser.org/', 'Camino'],
+                    'chrome': ['https://www.google.com/chrome/', 'Google Chrome'],
+                    'edge': ['https://www.microsoft.com/en-us/windows/microsoft-edge',
+                             'Microsoft Edge'],
+                    'firefox': ['https://www.mozilla.org/firefox/', 'Mozilla Firefox'],
+                    'galeon': ['http://galeon.sourceforge.net/', 'Galeon'],
+                    'kmeleon': ['http://kmeleonbrowser.org/', 'K-Meleon'],
+                    'konqueror': ['https://konqueror.org/', 'Konqueror'],
+                    'links': ['http://links.twibright.com/', 'Links'],
+                    'msie': ['https://en.wikipedia.org/wiki/Internet_Explorer',
+                             'Microsoft Internet Explorer'],
+                    'lynx': ['http://lynx.browser.org/', 'Lynx'],
+                    'safari': ['https://www.apple.com/safari/', 'Apple Safari']}
+    _os = {'aix': ['https://www.ibm.com/power/operating-systems/aix', 'AIX'],
+           'amiga': ['http://www.amiga.org/', 'Amiga'],
+           'android': ['https://www.android.com/', 'Android'],
+           'bsd': ['http://www.bsd.org/', 'BSD'],
+           'chromec': ['https://www.chromium.org/chromium-os', 'ChromeOS'],
+           'hpux': ['https://www.hpe.com/us/en/servers/hp-ux.html', 'HP-UX'],
+           'iphone': ['https://www.apple.com/iphone/', 'iPhone'],
+           'ipad': ['https://www.apple.com/ipad/', 'iPad'],
+           'irix': ['https://www.sgi.com/', 'IRIX'],
+           'linux': ['https://www.kernel.org/', 'GNU/Linux'],
+           'macos': ['https://www.apple.com/macos/', 'macOS'],
+           'sco': ['http://www.sco.com/products/unix/', 'SCO'],
+           'solaris': ['https://www.oracle.com/solaris/', 'Solaris'],
+           'wii': ['http://wii.com/', 'Wii'],
+           'windows': ['https://www.microsoft.com/windows/', 'Windows']}
+    _alts = {'amiga': ' (have you tried <a href="http://aros.sourceforge.net/">AROS</a> yet?)',
+             'android': ' (have you tried <a href="https://lineageos.org/">LineageOS</a> yet?)',
+             'macos': ' (have you tried <a href="https://elementary.io/">ElementaryOS</a> yet?)',
+             'sgi': ' (have you tried <a href="http://www.maxxinteractive.com">MaXX</a> yet?)',
+             'windows': ' (have you tried <a href="https://https://reactos.org/">ReactOS</a> yet?)'}
+    # And then we set some parameter options for less typing later on.
+    _yes = ('y', 'yes', 'true', '1', True)
+    _no = ('y', 'no', 'false', '0', False, 'none')
+    # http://werkzeug.pocoo.org/docs/0.12/utils/#module-werkzeug.useragents
+    visitor = {'client': {'str': request.user_agent.string,
+                          'browser': request.user_agent.browser,
+                          'os': request.user_agent.platform,
+                          'language': request.user_agent.language,
+                          'to_header': request.user_agent.to_header(),
+                          'version': request.user_agent.version},
+               'ip': re.sub('^::ffff:', '', request.remote_addr),
+               'headers': dict(request.headers)}
+    # We have to convert these to strings so we can do tuple comparisons on lower()s.
+    params = {'json': str(request.args.get('json')).lower(),
+              'html': str(request.args.get('html')).lower(),
+              'raw': str(request.args.get('raw')).lower()}
+    if visitor['client']['browser'] in _intbrowsers.keys():
+        if params['html'] == 'none':
+            params['html'] = True
+            if params['json'] == 'none':
+                params['json'] = False
+            elif params['json'] in _yes:
+                params['json'] = True
+    for k in params.keys():
+        if params[k] in _no:
+            params[k] = False
+        else:
+            params[k] = True
+    # Set the tabs for JSON
+    try:
+        params['tabs'] = int(request.args.get('tabs'))
+    except (ValueError, TypeError):
+        if visitor['client']['browser'] in _intbrowsers.keys() or params['html']:
+            params['tabs'] = 4
+        else:
+            params['tabs'] = None
+    j = json.dumps(visitor, indent = params['tabs'])
+    if (visitor['client']['browser'] in _intbrowsers.keys() and params['html'] and not params['raw']) or \
+        (visitor['client']['browser'] not in _intbrowsers.keys() and params['html']):
+        return(render_template('index.html',
+                               visitor = visitor,
+                               browsers = _intbrowsers,
+                               os = _os,
+                               alts = _alts,
+                               json = j,
+                               params = params))
+    else:
+        if visitor['client']['browser'] in _intbrowsers.keys() and not params['raw']:
+            return(render_template('json.html',
+                                   json = j,
+                                   params = params))
+        return(j)
+
+@app.route('/about', methods = ['GET'])
+def about():
+    return(render_template('about.html'))
+
+@app.route('/usage', methods = ['GET'])
+def usage():
+    return(render_template('usage.html'))

net/addr/config.py

@@ -0,0 +1,5 @@
+# config.py
+
+# Flask debugging - DISABLE FOR PRODUCTION ENVIRONMENTS
+DEBUG = True
+#DEBUG = False

net/addr/run.py

@@ -0,0 +1,4 @@
+from app import app
+
+if __name__ == '__main__':
+    app.run()

net/addr/uwsgi.ini

@@ -0,0 +1,18 @@
+[uwsgi]
+plugin = python
+py-autoreload = 1
+#uid = http
+#gid = http
+socket = /run/uwsgi/netinfo.sock
+chown-socket = http:http
+processes = 4
+master = 1
+base = /usr/local/lib/optools/net/addr
+chdir = %(base)
+#mount = /=%(base)/run.py
+wsgi-file = %(base)/run.py
+chmod-socket = 660
+callable = app
+cgi-helper =.py=python
+logto = /var/log/uwsgi/%n.log
+vacuum

net/bofh_gen.py

@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+import telnetlib
+import time
+
+counter = 8
+
+def get_excuse():
+    # http://www.blinkenlights.nl/services.html
+    # port 23 (default) is Star Wars.
+    # port 666 is BOfH excuses
+    with telnetlib.Telnet('towel.blinkenlights.nl', port = 666) as t:
+        excuse = [x.decode('utf-8').strip() \
+                                    for x in t.read_all().split(b'===\r\n')]
+    return(excuse[2])
+
+def main():
+    for i in range(counter):
+        e = get_excuse()
+        print(e)
+        time.sleep(1)
+
+if __name__ == '__main__':
+    main()

net/connchk.py

@@ -0,0 +1,2 @@
+#!/usr/bin/env python3
+

net/devices/actiontec/ActionTec/Cmd.py

@@ -0,0 +1 @@
+#!/usr/bin/env python3.6

net/devices/actiontec/ActionTec/Connector.py

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3.6
+
+# stdlib
+import re
+import socket
+
+def CheckConnection(host, port):
+    # We favor socket over telnetlib's check because it has a little better
+    # handling of exceptions.
+    try:
+        port = int(port)  # just in case we were passed a str()
+    except ValueError:
+        raise ValueError('"{0}" is not a port number'.format(port))
+        # In case they're catching the exception...
+        return(False)
+    s = socket.socket()
+    try:
+        s.connect((host, port))
+    except Exception as e:
+        raise RuntimeError(('We were unable to successfully connect to ' +
+                            '"{0}:{1}": {2}').format(host, port, e))
+        return(False)
+    finally:
+        s.close()
+    return(True)
+
+def Login(host, port, ssl, user, password):
+    user_prompt = [re.compile('^\s*user(name)?\s*:?\s*'.encode('utf-8'),
+                              re.IGNORECASE)]
+    passwd_prompt = [re.compile('^\s*passw(or)d?\s*:?\s*'.encode('utf-8'),
+                                re.IGNORECASE)]
+    # Are there any other valid chars? Will need to experiment.
+    # How is this even set? The default is "Wireless Broadband Router".
+    # I think it can't be changed, at least via the Web GUI.
+    cmd_prompt = [re.compile('[-_a-z0-9\s]*>'.encode('utf-8'),
+                             re.IGNORECASE)]
+    ctx = None
+    ctxargs = {'host': host, 'port': port}
+    try:
+        if ssl:
+            try:
+                from ssltelnet import SslTelnet as telnet
+                ctxargs['force_ssl'] = True
+            except ImportError:
+                raise ImportError(('You have enabled SSL but do not have ' +
+                                   'the ssltelnet module installed. See ' +
+                                   'the README file, footnote [1].'))
+        else:
+            from telnetlib import Telnet as telnet
+        ctx = telnet(**ctxargs)
+        ctx.expect(user_prompt, timeout = 8)
+        ctx.write((user + '\n').encode('utf-8'))
+        ctx.expect(passwd_prompt, timeout = 8)
+        ctx.write((password + '\n').encode('utf-8'))
+        ctx.expect(cmd_prompt, timeout = 15)
+    except EOFError:
+        if ctx:
+            ctx.close()
+            ctx = None
+    except Exception as e:
+        raise RuntimeError(('We encountered an error when trying to connect:' +
+                            ' {0}').format(e))
+        if ctx:
+            ctx.close()
+            ctx = None
+    return(ctx)

net/devices/actiontec/ActionTec/__init__.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3.6
+
+import .Cmd as Cmd
+import .Connector as Connector
+
+class Router(object):
+    def __init__(self, host, port, user, password, ssl = False):
+        self.host = host
+        self.port = port
+        self.user = user
+        self.password = password
+        self.ssl = ssl
+        self.ctx = None
+        # Convenient shorthand. See "help.all.txt".
+        self.cmds = {'reboot': 'system reboot',
+                     'wipe': 'conf factory restore',
+                     # this will... require an interactive session
+                     'shell': 'system shell'}
+        
+    def connect(self):
+        # We don't need to define an except, really.
+        # The function handles that for us.
+        Connector.CheckConnection(self.host, self.port)
+        self.ctx = Connector.Login(self.host, self.port, self.ssl, self.user,
+                                   self.password)
+        return()
+    
+    def execute(self):
+        pass
+    
+    def close(self):
+        if self.ctx:
+            self.ctx.close()

net/devices/actiontec/ActionTec/help.all.txt

@@ -0,0 +1,177 @@
+Command Category wanmonitor - wanmonitor commands for rtd, do not use it directly
+get    notify wanmonitor to send wan type to rtd
+exit   Exit sub menu
+help   Show help for commands within this menu
+
+Command Category conf - Read and write Wireless Broadband Router configuration data
+factory            Factory related commands
+print              Print Wireless Broadband Router configuration
+set                Set Wireless Broadband Router configuration path to value
+set_obscure        Set Wireless Broadband Router configuration path to an 
+                   obscured value
+del                Delete subtree from Wireless Broadband Router configuration
+ram_set            Set Wireless Broadband Router dynamic configuration
+ram_print          Print Wireless Broadband Router dynamic configuration
+reconf             Reconfigure the system according to the current Wireless 
+                   Broadband Router configuration
+firmware_restore   Restore to saved firmware and reboot.
+exit               Exit sub menu
+help               Show help for commands within this menu
+
+Command Category upnp - UPnP commands
+igd      IGD commands
+status   Display UPnP status
+exit     Exit sub menu
+help     Show help for commands within this menu
+
+Command Category qos - Control and display QoS data
+utilization   Connection utilization information
+exit          Exit sub menu
+help          Show help for commands within this menu
+
+Command Category wmm - wmm configuration and control
+get       get the specified entry
+set       set the specified entry
+del       del the specified entry
+get_dev   get the entries of the specified device
+exit      Exit sub menu
+help      Show help for commands within this menu
+
+Command Category cwmp - CWMP related commands
+status          Print CWMP status
+session_start   Start CWMP session to ACS
+session_stop    Stop CWMP session
+indexes         Print CWMP devices indexes
+exit            Exit sub menu
+help            Show help for commands within this menu
+
+Command Category bridge - API for managing ethernet bridge
+connection   connect separate network interfaces to form one seamless LAN
+config       Configure bridge
+info         Print bridge information
+exit         Exit sub menu
+help         Show help for commands within this menu
+
+Command Category firewall - Control and display Firewall and NAT data
+restart          Stop and start Firewall & NAT
+start            Start Firewall & NAT
+stop             Stop Firewall & NAT
+filter           Turn Firewall packet inspection on/off
+mac_cache_dump   Dump MAC cache data
+dump             Display Firewall data
+variable         Display variables of the firewall rules
+trace            Trace packet traversal via the Firewall ruleset
+fastpath         Turns firewall fastpath feature on/off (default is on)
+exit             Exit sub menu
+help             Show help for commands within this menu
+
+Command Category connection - API for managing connections
+pppoe   Configure pppoe interface
+vlan    Configure vlan interface
+exit    Exit sub menu
+help    Show help for commands within this menu
+
+Command Category inet_connection - API for managing internet connections
+pppoe   Configure pppoe internet connection
+ether   Configure ethernet internet connection
+exit    Exit sub menu
+help    Show help for commands within this menu
+
+Command Category misc - API for Wireless Broadband Router miscellaneous tasks
+print_ram         print ram consumption for each process
+vlan_add          Add VLAN interface
+top               Profiling over event loop and estream
+wbm_debug_set     Stop and start WBM debug mode
+wbm_border_set    Stop and start WBM border mode
+knet_hooks_dump   Dump to console which knet_hooks run on each device
+malloc_info       Print memory information of malloc module
+malloc_trim       Free unused allocated memory in malloc module
+exit              Exit sub menu
+help              Show help for commands within this menu
+
+Command Category firmware_update - Firmware update commands
+start    Remotely upgrade Wireless Broadband Router
+cancel   Kill running remote upgrade
+exit     Exit sub menu
+help     Show help for commands within this menu
+
+Command Category log - Controls Wireless Broadband Router logging behavior
+filter   Controls the CLI session logging behavior
+print    Print the contents of a given syslog buffer to the console
+clear    Clear the contents of a given syslog buffer
+exit     Exit sub menu
+help     Show help for commands within this menu
+
+Command Category dev - Device related commands
+mv88e60xx         Marvell MV88e60xx Ethernet Switch commands
+moca              MOCA commands
+mii_reg_get       Get Ethernet MII register value
+mii_reg_set       Set Ethernet MII register value
+mii_phy_reg_get   Get Ethernet MII register value
+mii_phy_reg_set   Set Ethernet MII register value
+exit              Exit sub menu
+help              Show help for commands within this menu
+
+Command Category kernel - Kernel related commands
+sys_ioctl      issue openrg ioctl
+meminfo        Print memory information
+top            Print Wireless Broadband Router's processes memory usage
+cpu_load_on    Periodically shows cpu usage.
+cpu_load_off   Stop showing cpu usage (triggered by cpu_load_on).
+cpu_load_avg   Shows average cpu usage of last 1, 5 and 15 minutes. 
+exit           Exit sub menu
+help           Show help for commands within this menu
+
+Command Category system - Commands to control Wireless Broadband Router execution
+http_intercept_status      Display HTTP intercept status
+diag_test                  run diagtest 0=all  or  select 1-9 for TBHR 
+                           ..TLANIPSTB
+diag_correction            run corrections -- may reboot or reset BHR
+die                        Exit from Wireless Broadband Router and return ret
+ps                         Print Wireless Broadband Router's tasks
+entity_close               Close an entity
+etask_list_dump            Dump back trace of all etasks
+restore_factory_settings   Restore factory configuration
+reboot                     Reboot the system
+ver                        Display version information
+print_config               Print compilation configuration. Search for option 
+                           if specified
+exec                       Execute program
+cat                        Print file contents to console
+shell                      Spawn busybox shell in foreground
+date                       Print the current UTC and local time
+print_page                 Print page id and name
+exit                       Exit sub menu
+help                       Show help for commands within this menu
+
+Command Category flash - Flash and loader related commands
+commit   Save Wireless Broadband Router configuration to flash
+erase    Erase a given section in the flash
+load     Load and burn image
+boot     Boot the system
+bset     Configure bootloader
+layout   Print the flash layout and content
+dump     Dump the flash content
+lock     Lock mtd region
+unlock   Unlock mtd region
+exit     Exit sub menu
+help     Show help for commands within this menu
+
+Command Category net - Network related commands
+dns_route         Dyncamic Routing according to DNS replies
+igmp              IGMP Proxy related commands
+host              Resolve host by name
+protected_setup   Network related commands
+wsc               wps related commands
+ifconfig          Configure network interface
+ping              Test network connectivity
+rg_ifconfig       List Wireless Broadband Router Network Devices
+route             Print route table
+main_wan          Print the name of the current main wan device
+intercept_state   Print interception state
+exit              Exit sub menu
+help              Show help for commands within this menu
+
+Command Category cmd - Commands related to the Command module
+exit   Exit from the current CLI session
+help   Show help for commands within this menu

net/devices/actiontec/README

@@ -0,0 +1,122 @@
+This has been confirmed to work for, at the very least, my own Verizon Fi-OS
+Actiontec MI424WR-GEN3I on firmware 40.21.24. It might work on other models as
+well, but this hasn't been tested.
+
+No non-stdlib modules are required.
+
+Place your routers credentials in ~/.config/optools/actiontec_mgmt.json
+in the following format:
+(pay close attention to the quoting)
+(minified json is OK/whitespace-insensitive):
+_______________________________________________________________________________
+                {
+                    "ip_addr": "192.168.1.1",
+                    "user": "admin",
+                    "password": "admin",
+                    "ssl": false,
+                    "port": 23
+                }
+_______________________________________________________________________________
+
+IF:
+
+- That file isn't found:
+-- A default (blank) one will be created (with secure permissions). All values
+   will be null (see below).
+   
+- "ip_addr" is null:
+-- You will be prompted for the IP address interactively. (If you don't know
+   the IP address of it, it's probably the default -- "192.168.1.1".)
+
+- "user" is null:
+-- You will be prompted for the username to log in interactively. (If you don't
+   know the username, it's probably the default -- "admin".)
+
+- "password" is null:
+-- You will be prompted for the password. When being prompted, it will NOT echo
+   back (like a sudo prompt).
+   
+- "ssl" is null:
+-- The default (false) will be used.
+
+- "port" is null:
+-- The default port (23) will be used.
+
+
+
+TIPS:
+
+- You need to ensure that you have the management interface enabled. Log into
+  your Actiontec's web interface, and:
+  1.) "Advanced" button (at the top)
+  2.) "Yes" button
+  3.) a.) Choose "Local administration" if you'll be managing the device within
+          the network it provides.[0]
+      b.) Choose "Remote administration" if you'll be managing the device
+          outside the network it provides (i.e. over the Internet).[0]
+      3.5) The "Telnet" options are what you want, ignore the "Web" settings.
+  4.) Select the protocols/ports you'll be using. SEE FOOTNOTE 0 ([0])!
+  5.) Click the "Apply" button.
+
+- "ip_addr" can also be a host/DNS name -- just make sure it resolves on your
+  local machine to your Actiontec IP address! The default, at least on mine,
+  was "wireless_broadband_router" (can be changed via Advanced > Yes > System
+  Settings > Wireless Broadband Router's Hostname):
+  
+        [bts@cylon ~]$ nslookup wireless_broadband_router 192.168.1.1
+        Server:		192.168.1.1
+        Address:	192.168.1.1#53
+
+        Name:	wireless_broadband_router
+        Address: 192.168.1.1
+        Name:	wireless_broadband_router
+        Address: <YOUR_PUBLIC_IP_ADDRESS>
+
+
+- Unfortunately it's a necessity to store the password in plaintext currently.
+  Future versions may give the option of encrypting it via GPG and using an
+  existing GPG agent session to unlock (if there's demand for such a feature).
+  Make sure your machine's files are safe (I recommend full-disk encryption).
+  
+
+[0] NOTE: ENABLING MANAGEMENT CAN BE HIGHLY INSECURE, *ESPECIALLY* IF ENABLING
+          "REMOTE ADMINISTRATION"! *ONLY* DO THIS IF YOU UNDERSTAND THE RISKS
+          AND HAVE ACCOUNTED FOR THEM. TELNET PASSES CREDENTIALS IN PLAINTEXT
+          BY DEFAULT, AND IF SOMEONE NASTY GETS THEIR HANDS ON YOUR DEVICE'S
+          CREDENTIALS THEY CAN DO *VERY* NASTY THINGS. I REFUSE ANY AND ALL
+          LIABILITY YOU OPEN YOURSELF UP TO BY ENABLING THIS. AT *LEAST* USE
+          THE "USING SECURE TELNET OVER SSL PORT"[1] OPTION.
+          YOU HAVE BEEN WARNED.
+
+[1] NOTE: Even if using SSL, it's HIGHLY insecure and not to be trusted. The
+          key has been leaked (as of 2018-04-12):
+                    https://code.google.com/archive/p/littleblackbox/
+          and it uses VERY weak ciphers, at that:
+          _____________________________________________________________________
+          | ssl-cert: Subject: commonName=ORname_Jungo: OpenRG Products Group/|
+          |           countryName=US                                          |
+          | Not valid before: 2004-06-03T11:11:43                             |
+          |_Not valid after:  2024-05-29T11:11:43                             |
+          |_ssl-date: 2018-04-12T09:42:22+00:00; -1s from scanner time.       |
+          |_ssl-known-key: Found in Little Black Box 0.1 -                    |
+          |         http://code.google.com/p/littleblackbox/                  |
+          |         (SHA-1: 4388 33c0 94f6 afc8 64c6 0e4a 6f57 e9f4 d128 1411)|
+          | sslv2:                                                            |
+          |   SSLv2 supported                                                 |
+          |   ciphers:                                                        |
+          |     SSL2_RC4_128_WITH_MD5                                         |
+          |     SSL2_RC4_64_WITH_MD5                                          |
+          |     SSL2_RC2_128_CBC_EXPORT40_WITH_MD5                            |
+          |     SSL2_RC4_128_EXPORT40_WITH_MD5                                |
+          |     SSL2_DES_192_EDE3_CBC_WITH_MD5                                |
+          |     SSL2_RC2_128_CBC_WITH_MD5                                     |
+          |_    SSL2_DES_64_CBC_WITH_MD5                                      |
+          |___________________________________________________________________|
+          
+          It's generally probably not even worth it, to be honest. You'll get
+          more security mileage out of firewalling off to select hosts/nets.
+          But, if you insist on having it and using it, you will ALSO need to
+          install the following module:
+          
+                        ssltelnet
+                        https://pypi.python.org/pypi/ssltelnet

net/devices/actiontec/actiontec_mgmt.py

@@ -0,0 +1 @@
+#!/usr/bin/env python3.6

net/dhcp/dhcpcdump.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+
+# INCOMPLETE
+
+# See RFC 2131, Figure 1 and Table 1 (section 2)
+# Much thanks to https://github.com/igordcard/dhcplease for digging into dhcpcd
+# source for the actual file structure (and providing inspiration).
+
+import argparse
+import collections
+import os
+import re
+import struct
+from io import BytesIO
+
+## DEFINE SOME PRETTY STUFF ##
+class color(object):
+    PURPLE = '\033[95m'
+    CYAN = '\033[96m'
+    DARKCYAN = '\033[36m'
+    BLUE = '\033[94m'
+    GREEN = '\033[92m'
+    YELLOW = '\033[93m'
+    RED = '\033[91m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'
+    END = '\033[0m'
+
+class packetParser(object):
+    def __init__(self, data):
+        ## Set the segment labels and struct formats
+        self.fmt = collections.OrderedDict()
+        # In the below, 'cnt' is how large (in octets) the field is.
+        # 'fmt' is a struct format string (https://docs.python.org/3/library/struct.html#format-characters)
+        # "op" through "hops" (incl.) may actually be '8B' instead of '8c'.
+        self.fmt['op'] = {'cnt': 8, 'fmt': '8c'}  # this will always be \x02
+        self.fmt['htype'] = {'cnt': 8, 'fmt': '8c'}  # this will always be \x01
+        self.fmt['hlen'] = {'cnt': 8, 'fmt': '8c'}
+        self.fmt['hops'] = {'cnt': 8, 'fmt': '8c'}
+        self.fmt['xid'] = {'cnt': 32, 'fmt': '8I'}
+        self.fmt['secs'] = {'cnt': 16, 'fmt': '8H'}
+        self.fmt['flags'] = {'cnt': 16, 'fmt': '8H'}
+        # "ciaddr" through "giaddr" (incl.) may actually be '4c' instead of '4B'.
+        self.fmt['ciaddr'] = {'cnt': 4, 'fmt': '4B'}
+        self.fmt['yiaddr'] = {'cnt': 4, 'fmt': '4B'}
+        self.fmt['siaddr'] = {'cnt': 4, 'fmt': '4B'}
+        self.fmt['giaddr'] = {'cnt': 4, 'fmt': '4B'}
+        # "chaddr" through "file" (incl.) may actually be <#>c instead of <#>B.
+        self.fmt['chaddr'] = {'cnt': 16, 'fmt': '16B'}  # first 6 bytes used for MAC addr of client
+        self.fmt['sname'] = {'cnt': 64, 'fmt': '64B'}  # server host name (via BOOTP)
+        self.fmt['file'] = {'cnt': 128, 'fmt': '128B'}  # the boot filename (for BOOTP)
+        # OPTIONS - RFC 2132
+        # Starting at octet 320 (so, f.seek(319, 0)) to the end of the message are
+        # DHCP options. It's a variable-length field so it makes things tricky
+        # for us. But it's at *least* 312 octets long per the RFC?
+        # It probably starts with a magic.
+        #self.dhcp_opts = {'idx': 324, 'cnt': 4, 'fmt': '4c'}
+        #self.dhcp_opts = {'idx': 324, 'cnt': 4, 'fmt': None}
+        self.opts = {'magic': b'\x63\x82\x53\x63',
+                     'struct': {'idx': 324, 'cnt': 4, 'fmt': '4B'},
+                     'size': 0,
+                     'bytes': b'\00'}
+        ## Convert the data into a bytes object because struct.unpack() wants a stream
+        self.buf = BytesIO(data)
+
+    def getStd(self):
+        self.reconstructed_segments = collections.OrderedDict()
+        _idx = 0  # add to this with the 'cnt' value for each iteration.
+        for k in self.fmt.keys():
+            print('Segment: ' + k)  # TODO: remove, this stuff goes in the printer
+            pkt = struct.Struct(self.fmt[k]['fmt'])
+            self.buf.seek(_idx, 0)
+            try:
+                self.reconstructed_segments[k] = pkt.unpack(self.buf.read(self.fmt[k]['cnt']))
+            except struct.error as e:
+                # Some DHCP implementations are... broken.
+                # I've noticed it mostly in Verizon Fi-OS gateways/WAPs/routers.
+                print('Warning({0}): {1}'.format(k, e))
+                self.buf.seek(_idx, 0)
+                _truesize = len(self.buf.read(self.fmt[k]['cnt']))
+                print('Length of bytes read: {0}'.format(_truesize))
+                # But sometimes it's... kind of fixable?
+                if k == 'file' and _truesize < self.fmt[k]['cnt']:
+                    self.buf.seek(_idx, 0)
+                    self.fmt[k] = {'cnt': _truesize, 'fmt': '{0}B'.format(_truesize)}
+                    pkt = struct.Struct(self.fmt[k]['fmt'])
+                    print('Struct format size automatically adjusted.')
+                    try:
+                        self.reconstructed_segments[k] = pkt.unpack(self.buf.read(self.fmt[k]['cnt']))
+                    except struct.error as e2:
+                        # yolo.
+                        print('We still couldn\'t populate {0}; filling with a nullbyte.'.format(k))
+                        print('Error (try #2): {0}'.format(e2))
+                        print('We read {0} bytes.'.format(_truesize))
+                        print('fmt: {0}'.format(self.fmt[k]['fmt']))
+                        self.reconstructed_segments[k] = b'\00'
+            _idx += self.fmt[k]['cnt']
+            self.buf.seek(_idx, 0)
+            # Finally, check for opts. If they exist, populate.
+            _optbytes = len(self.buf.read())
+            if _optbytes >= 1:
+                self.opts['size'] = _optbytes
+                self.buf.seek(_idx, 0)
+                self.opts['bytes'] = self.buf.read()  # read to the end
+        return()
+
+    def getOpts(self):
+        pass
+
+    def close(self):
+        self.buf.close()
+
+def parseArgs():
+    args = argparse.ArgumentParser()
+    _deflease = '/var/lib/dhcpcd/'
+    args.add_argument('-l', '--lease',
+                      metavar = '/path/to/lease/dir/or_file.lease',
+                      default = _deflease,
+                      dest = 'leasepath',
+                      help = ('The path to the directory of lease files or specific lease file. ' +
+                              'If a directory is provided, all lease files found within will be ' +
+                              'parsed. Default: {0}{1}{2}').format(color.BOLD,
+                                                                   _deflease,
+                                                                   color.END))
+    args.add_argument('-n', '--no-color',
+                      action = 'store_false',
+                      dest = 'color',
+                      help = ('If specified, suppress color formatting in output.'))
+    args.add_argument('-d', '--dump',
+                      metavar = '/path/to/dumpdir',
+                      default = False,
+                      dest = 'dump',
+                      help = ('If provided, dump the parsed leases to this directory (in ' +
+                              'addition to printing). It will dump with the same filename ' +
+                              'and overwrite any existing file with the same filename, so ' +
+                              'do NOT use the same directory as your dhcpcd lease files! ' +
+                              '({0}-l/--lease{1}). The directory will be created if it does ' +
+                              'not exist').format(color.BOLD,
+                                                  color.END))
+    args.add_argument('-p', '--pretty',
+                      action = 'store_true',
+                      dest = 'prettyprint',
+                      help = ('If specified, include color formatting {0}in the dump ' +
+                              'file(s){1}').format(color.BOLD, color.END))
+    return(args)
+
+def getLeaseData(fpath):
+    if not os.path.isfile(fpath):
+        raise FileNotFoundError('{0} does not exist'.format(fpath))
+    with open(fpath, 'rb') as f:
+        _data = f.read()
+    return(_data)
+
+def iterLease(args):
+    # If the lease path is a file, just operate on that.
+    # If it's a directory, iterate (recursively) through it.
+    leases = {}
+    if not os.path.lexists(args['leasepath']):
+        raise FileNotFoundError('{0} does not exist'.format(args['leasepath']))
+    if os.path.isfile(args['leasepath']):
+        _pp = packetParser(getLeaseData(args['leasepath']))
+        # TODO: convert the hex vals to their actual vals... maybe?
+        _keyname = re.sub('^(dhcpcd-)?(.*)\.lease$',
+                          '\g<2>',
+                          os.path.basename(args['leasepath']))
+        leases[_keyname] = leaseParse(_pp, args)
+    else:
+        # walk() instead of listdir() because whotf knows when some distro like
+        # *coughcoughUbuntucoughcough* will do some breaking change like creating
+        # subdirs based on iface name or something.
+        for _, _, files in os.walk(args['leasepath']):
+            if not files:
+                continue
+            files = [i for i in files if i.endswith('.lease')]  # only get .lease files
+            for i in files:
+                _args = args.copy()
+                _fpath = os.path.join(args['leasepath'], i)
+                _keyname = re.sub('^(dhcpcd-)?(.*)\.lease$', '\g<2>', os.path.basename(_fpath))
+                _dupeid = 0
+                # JUST in case there are multiple levels of dirs in the future
+                # that have files of the sama name
+                while _keyname in leases.keys():
+                    # TODO: convert the hex vals to their actual vals... maybe?
+                    _keyname = re.sub('^$',
+                                      '\g<1>.{0}'.format(_dupeid),
+                                      _keyname)
+                    _dupeid += 1
+                _pp = packetParser(getLeaseData(_fpath))
+                leases[_keyname] = leaseParse(_pp, _args, fname = _fpath)
+    return(leases)
+
+def leaseParse(pp, args, fname = False):
+    # Essentially just a wrapper function.
+    # Debugging output...
+    if fname:
+        print(fname)
+    pp.getStd()
+    pp.getOpts()
+    if args['dump']:
+        pass  # TODO: write to files, creating dump dir if needed, etc.
+    pp.close()
+    # do pretty-printing (color-coded segments, etc.) here
+    return(pp.reconstructed_segments)
+
+if __name__ == '__main__':
+    args = vars(parseArgs().parse_args())
+    args['leasepath'] = os.path.abspath(os.path.expanduser(args['leasepath']))
+    if not os.path.lexists(args['leasepath']):
+        exit('{0} does not exist!'.format(args['leasepath']))
+    leases = iterLease(args)
+    # just print for now until we write the parser/prettyprinter
+    print(list(leases.keys()))

net/dns/linode/README

@@ -0,0 +1,57 @@
+This script requires a configuration file (by default, ~/.config/ddns.xml). Please refer to example.ddns.xml for an example.
+
+The path to the configuration file can be changed with the -c/--config argument.
+
+!!! NOTE !!!
+This script as a precautionary measure does NOT create new domain names! It may create or remove A/AAAA records depending
+on whether your client has a IPv4 and/or IPv6 WAN route respectively, however.
+
+Because network DNS settings are unpredictable and we need to ensure we don't get split-brain or bogus DNS responses,
+this script uses Verisign's public DNS resolvers hardcoded in. These resolvers are recommended for privacy, speed, and
+RFC compliance. The exact resolvers used are:
+
+* 64.6.64.6
+* 64.6.65.6
+
+If you do not consent to this, do not use this script.
+!!!!!!!!!!!!
+
+!!! NOTE !!!
+This script, by *necessity*, connects to (tries to connect to) the following URLs:
+
+* https://ipv4.clientinfo.square-r00t.net/?raw=1
+* https://ipv6.clientinfo.square-r00t.net/?raw=1
+
+This is a necessity because otherwise we do not have a method of fetching the WAN IP if the client is e.g. behind NAT
+(or is using ULA addresses with a routed gateway/RFC 6296 in IPv6 networks, etc.).
+
+This is a service that the author himself has written (https://git.square-r00t.net/OpTools/tree/net/addr) and deployed.
+No personal information is sold, etc. and it only returns the headers and connection information the client sends in a
+standard HTTP(S) request.
+
+If you do not consent to this, either change the URL in Updater._getMyIP() (it is compatible with https://ipinfo.io/,
+but this service does not return split IPv4 and IPv6 records so further modifications would be required) or do not use
+this script.
+!!!!!!!!!!!!
+
+SETUP:
+
+1.)a.) Create the domain(s) you wish to use in the Linode Domains manager (https://cloud.linode.com/domains).
+   b.) Create the API token (https://cloud.linode.com/profile/tokens).
+       * It MUST have "Read/Write" access to the "Domains" scope. All other scopes can be "None".
+       * It is *HIGHLY recommended* that you generate a *unique* token for each and every client machine rather than
+         sharing a token across them.
+1.) Create a configuration file. Refer to the accompanying "example.ddns.xml" file.
+2.) Make sure the script is executable and you have all required python modules installed:
+      https://pypi.org/project/dnspython/
+      https://pypi.org/project/requests/
+      https://pypi.org/project/lxml/
+      https://pypi.org/project/systemd/ (optional; for logging to the journal)
+3.) You're ready to go! It is recommended that you either:
+   a.) Set up a cronjob (https://crontab.guru/), or
+   b.) Create a systemd timer (https://wiki.archlinux.org/index.php/Systemd/Timers) (if you're on a system with systemd).
+
+LOGGING:
+Logging is done to ~/.cache/ddns.log. Messages will also be logged to the systemd journal (if available and the systemd module is installed).
+
+Suggestions for improvement are welcome (r00t [at] square-r00t.net).

net/dns/linode/ddns.py

@@ -0,0 +1,299 @@
+#!/usr/bin/env python3
+
+import argparse
+import json
+import logging
+import logging.handlers
+import os
+import re
+import sys
+import warnings
+##
+import dns.exception
+import dns.resolver
+import requests
+##
+from lxml import etree
+try:
+    # https://www.freedesktop.org/software/systemd/python-systemd/journal.html#journalhandler-class
+    from systemd import journal
+    _has_journald = True
+except ImportError:
+    _has_journald = False
+
+
+logfile = '~/.cache/ddns.log'
+
+# Prep the log file.
+logfile = os.path.abspath(os.path.expanduser(logfile))
+os.makedirs(os.path.dirname(logfile), exist_ok = True, mode = 0o0700)
+if not os.path.isfile(logfile):
+    with open(logfile, 'w') as fh:
+        fh.write('')
+os.chmod(logfile, 0o0600)
+
+# And set up logging.
+_cfg_args = {'handlers': [],
+             'level': logging.DEBUG}
+if _has_journald:
+    # There were some weird changes somewhere along the line.
+    try:
+        # But it's *probably* this one.
+        h = journal.JournalHandler()
+    except AttributeError:
+        h = journal.JournaldLogHandler()
+    # Systemd includes times, so we don't need to.
+    h.setFormatter(logging.Formatter(style = '{',
+                                     fmt = ('{name}:{levelname}:{name}:{filename}:'
+                                            '{funcName}:{lineno}: {message}')))
+    _cfg_args['handlers'].append(h)
+h = logging.handlers.RotatingFileHandler(logfile,
+                                         encoding = 'utf8',
+                                         # Disable rotating for now.
+                                         # maxBytes = 50000000000,
+                                         # backupCount = 30
+                                         )
+h.setFormatter(logging.Formatter(style = '{',
+                                 fmt = ('{asctime}:'
+                                        '{levelname}:{name}:{filename}:'
+                                        '{funcName}:{lineno}: {message}')))
+_cfg_args['handlers'].append(h)
+logging.basicConfig(**_cfg_args)
+logger = logging.getLogger('DDNS')
+logger.info('Logging initialized.')
+
+is_tty = sys.stdin.isatty()
+if not is_tty:
+    logger.debug('Not running in an interactive invocation; disabling printing warnings')
+else:
+    logger.debug('Running in an interactive invocation; enabling printing warnings')
+
+
+class Updater(object):
+    tree = None
+    records = {}
+    api_base = None
+    session = None
+    token = None
+    my_ips = {4: None, 6: None}
+    resolver = dns.resolver.Resolver(configure = False)
+    resolver.nameservers = ['64.6.64.6', '64.6.65.6']
+
+    def __init__(self, cfg_path = '~/.config/ddns.xml', *args, **kwargs):
+        self.xml = os.path.abspath(os.path.expanduser(cfg_path))
+        logger.debug('Updater initialized with config {0}'.format(self.xml))
+        self._getConf()
+        self._getMyIP()
+        self._getSession()
+
+    def _getConf(self):
+        try:
+            with open(self.xml, 'rb') as fh:
+                self.xml = etree.fromstring(fh.read())
+        except FileNotFoundError as e:
+            logger.error('Configuration file does not exist; please create it')
+            raise e
+        self.tree = self.xml.getroottree()
+        self.token = self.xml.attrib['token']
+        self.api_base = re.sub(r'/$', '', self.xml.attrib['base'])
+        dom_xml = self.xml.findall('domain')
+        num_doms = len(dom_xml)
+        logger.debug('Found {0} domains in config'.format(num_doms))
+        for idx, d in enumerate(dom_xml):
+            domain = d.attrib['name']
+            logger.debug('Iterating domain {0} ({1}/{2})'.format(domain, (idx + 1), num_doms))
+            if domain not in self.records.keys():
+                self.records[domain] = []
+            sub_xml = d.findall('sub')
+            num_subs = len(sub_xml)
+            logger.debug('Found {0} records for domain {1}'.format(num_subs, domain))
+            for idx2, s in enumerate(sub_xml):
+                logger.debug('Adding record {0}.{1} to index ({2}/{3})'.format(s.text, domain, (idx2 + 1), num_subs))
+                self.records[domain].append(s.text)
+        return()
+
+    def _getDNS(self, record):
+        records = {}
+        for t in ('A', 'AAAA'):
+            logger.debug('Resolving {0} ({1})'.format(record, t))
+            try:
+                q = self.resolver.resolve(record, t)
+                for a in q:
+                    if t not in records.keys():
+                        records[t] = []
+                    ip = a.to_text()
+                    logger.debug('Found IP {0} for record {1} ({2})'.format(ip, record, t))
+                    records[t].append(ip)
+            except dns.exception.Timeout as e:
+                logger.error('Got a timeout when resolving {0} ({1}): {2}'.format(record, t, e))
+                continue
+            except dns.resolver.NXDOMAIN as e:
+                # This is a debug instead of an error because that record type may not exist.
+                logger.debug('Record {0} ({1}) does not exist: {2}'.format(record, t, e))
+                continue
+            except dns.resolver.YXDOMAIN as e:
+                logger.error('Record {0} ({1}) is too long: {2}'.format(record, t, e))
+                continue
+            except dns.resolver.NoAnswer as e:
+                # This is a debug instead of an error because that record type may not exist.
+                logger.debug('Record {0} ({1}) exists but has no content: {2}'.format(record, t, e))
+                continue
+            except dns.resolver.NoNameservers as e:
+                logger.error(('Could not failover to a non-broken resolver when resolving {0} ({1}): '
+                              '{2}').format(record, t, e))
+                continue
+        return(records)
+
+    def _getMyIP(self):
+        for v in self.my_ips.keys():
+            try:
+                logger.debug('Getting the client\'s WAN address for IPv{0}'.format(v))
+                r = requests.get('https://ipv{0}.clientinfo.square-r00t.net/?raw=1'.format(v))
+                if not r.ok:
+                    logger.error('Got a non-OK response from WAN IPv{0} fetch.'.format(v))
+                    raise RuntimeError('Could not get the IPv{0} address'.format(v))
+                ip = r.json()['ip']
+                logger.debug('Got WAN IP address {0} for IPv{1}'.format(ip, v))
+                self.my_ips[v] = ip
+            except requests.exceptions.ConnectionError:
+                logger.debug('Could not get WAN address for IPv{0}; likely not supported on this network'.format(v))
+        return()
+
+    def _getSession(self):
+        self.session = requests.Session()
+        self.session.headers.update({'Authorization': 'Bearer {0}'.format(self.token)})
+        return()
+
+    def update(self):
+        for d in self.records.keys():
+            d_f = json.dumps({'domain': d})
+            doms_url = '{0}/domains'.format(self.api_base)
+            logger.debug('Getting list of domains from {0} (filtered to {1})'.format(doms_url, d))
+            d_r = self.session.get(doms_url,
+                                   headers = {'X-Filter': d_f})
+            if not d_r.ok:
+                e = 'Could not get list of domains when attempting to check {0}; skipping'.format(d)
+                if is_tty:
+                    warnings.warn(e)
+                logger.warning(e)
+                continue
+            try:
+                d_id = d_r.json()['data'][0]['id']
+            except (IndexError, KeyError):
+                e = 'Could not find domain {0} in the returned domains list; skipping'.format(d)
+                if is_tty:
+                    warnings.warn(e)
+                logger.warning(e)
+                continue
+            for s in self.records[d]:
+                fqdn = '{0}.{1}'.format(s, d)
+                logger.debug('Processing {0}'.format(fqdn))
+                records = self._getDNS(fqdn)
+                for v, t in ((4, 'A'), (6, 'AAAA')):
+                    ip = self.my_ips.get(v)
+                    rrset = records.get(t)
+                    if not ip:
+                        e = 'IPv{0} disabled; skipping'.format(v)
+                        if is_tty:
+                            warnings.warn(e)
+                        logger.warning(e)
+                        continue
+                    if rrset and ip in rrset:
+                        e = 'Skipping adding {0} for {1}; already exists in DNS'.format(ip, fqdn)
+                        logger.info(e)
+                        if is_tty:
+                            print(e)
+                        continue
+                    s_f = json.dumps({'name': s,
+                                      'type': t})
+                    records_url = '{0}/domains/{1}/records'.format(self.api_base, d_id)
+                    logger.debug(('Getting list of records from {0} '
+                                  '(filtered to name {1} and type {2})').format(records_url, s, t))
+                    s_r = self.session.get(records_url,
+                                           headers = {'X-Filter': s_f})
+                    if not s_r.ok:
+                        e = 'Could not get list of records when attempting to check {0} ({1}); skipping'.format(fqdn, t)
+                        if is_tty:
+                            warnings.warn(e)
+                        logger.warning(e)
+                        continue
+                    r_ids = set()
+                    # If r_exists is:
+                    #   None, then the record exists but the current WAN IP is missing (all records replaced).
+                    #   False, then the record does not exist (record will be added).
+                    #   True, then the record exists and is current (nothing will be done).
+                    r_exists = None
+                    try:
+                        api_records = s_r.json().pop('data')
+                        for idx, r in enumerate(api_records):
+                            r_ids.add(r['id'])
+                            r_ip = r['target']
+                            if r_ip == ip:
+                                r_exists = True
+                    except (IndexError, KeyError):
+                        e = ('Could not find record {0} ({1}) in the returned records list; '
+                             'creating new record').format(fqdn, t)
+                        if is_tty:
+                            print(e)
+                        logger.info(e)
+                        r_exists = False
+                    if r_exists:
+                        # Do nothing.
+                        e = 'Skipping adding {0} for {1}; already exists in API and is correct'.format(ip, fqdn)
+                        logger.info(e)
+                        if is_tty:
+                            print(e)
+                        continue
+                    elif r_exists is None:
+                        # Remove all records and then add (at the end).
+                        # We COULD do an update:
+                        #   https://developers.linode.com/api/v4/domains-domain-id-records-record-id/#put
+                        # BUT then we break future updating since we don't know which record is the "right" one to
+                        # update.
+                        logger.debug('Record {0} ({1}) exists but does not contain {2}; replacing'.format(fqdn, t, ip))
+                        for r_id in r_ids:
+                            del_url = '{0}/{1}'.format(records_url, r_id)
+                            logger.debug(('Deleting record ID {0} for {1} ({2})').format(r_id, fqdn, t))
+                            del_r = self.session.delete(del_url)
+                            if not del_r.ok:
+                                e = 'Could not delete record ID {0} for {1} ({2}); skipping'.format(r_id, fqdn, t)
+                                if is_tty:
+                                    warnings.warn(e)
+                                logger.warning(e)
+                                continue
+                    else:
+                        # Create the record.
+                        logger.debug('Record {0} ({1}) does not exist; creating'.format(fqdn, ip))
+                    record = {'name': s,
+                              'type': t,
+                              'target': ip,
+                              'ttl_sec': 300}
+                    create_r = self.session.post(records_url,
+                                                 json = record)
+                    if not create_r.ok:
+                        e = 'Could not create record {0} ({1}); skipping'.format(fqdn, t)
+                        if is_tty:
+                            warnings.warn(e)
+                        logger.warning(e)
+                        continue
+        return()
+
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = ('Automatically update Linode DNS via their API'))
+    args.add_argument('-c', '--config',
+                      dest = 'cfg_path',
+                      default = '~/.config/ddns.xml',
+                      help = ('The path to the configuration file. Default: ~/.config/ddns.xml'))
+    return(args)
+
+
+def main():
+    args = parseArgs().parse_args()
+    u = Updater(**vars(args))
+    u.update()
+    return(None)
+
+
+if __name__ == '__main__':
+    main()

net/dns/linode/example.ddns.xml

@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!-- You very much most likely will want to leave "base" ALONE. Make sure you change "token" to your Linode API token,
+     though. -->
+<api base="https://api.linode.com/v4/"
+     token="YOUR_TOKEN_HERE">
+  <!-- Domains MUST be created first in the Linode Domains manager! -->
+  <domain name="domain1.com">
+    <!-- This would be for the A/AAAA record "foo.domain1.com". -->
+    <sub>foo</sub>
+    <!-- And obviously, this for "bar.domain1.com". -->
+    <sub>bar</sub>
+  </domain>
+  <domain name="domain2.net">
+    <!-- baz.domain2.net -->
+    <sub>baz</sub>
+    <!-- quux.domain2.net -->
+    <sub>quux</sub>
+  </domain>
+</api>

net/dns/rfc4183.py

@@ -0,0 +1,120 @@
+#!/usr/bin/env python3
+
+# https://tools.ietf.org/html/rfc2317
+# https://tools.ietf.org/html/rfc4183
+desc = 'Gets the RFC 2317/4183 PTR of given IP addresses or A/AAAA records.' 
+
+# stdlib
+import argparse
+import copy
+import ipaddress
+import os
+# pypi/pip
+#try:
+#    import ipwhois
+#except ImportError:
+#    exit('You need to install the ipwhois module.')
+try:
+    import dns.resolver
+    import dns.reversename
+except ImportError:
+    exit('You need to install the dnspython module.')
+try:
+    import fqdn
+except ImportError:
+    exit('You need to install the fqdn module.')
+
+def resolveRecord(addr):
+    r = dns.resolver.Resolver()
+    ipaddrs = {'A': [],
+               'AAAA': []}
+    for rtype in ipaddrs.keys():
+        for record in r.query(addr, 'A'):
+            ipaddrs[rtype].append(record)
+    ipaddrs['ipv4'] = sorted(list(set(copy.deepcopy(ipaddrs['A']))))
+    ipaddrs['ipv6'] = sorted(list(set(copy.deepcopy(ipaddrs['AAAA']))))
+    del(ipaddrs['A'], ipaddrs['AAAA'])
+    if ipaddrs['ipv4'] == ipaddrs['ipv6']:
+        del(ipaddrs['ipv6'])
+    return(ipaddrs)
+
+def genPTR(ipaddr, iptype):
+    _suffix = ''
+    # TODO: get the current PTR.
+    # TODO: do this more manually. We should use ipaddress and ipwhois to get
+    #       the proper return for e.g. network gateways.
+    return(dns.reversename.from_address(ipaddr))
+
+def chkInput(src):
+    # Determine the input, if we can.
+    src_out = (None, None)
+    try:
+        ipaddress.IPv4Address(src)
+        return(('ipv4', src))
+    except ipaddress.AddressValueError:
+        pass
+    try:
+        ipaddress.IPv6Address(src)
+        return(('ipv6', src))
+    except ipaddress.AddressValueError:
+        pass
+    _p = os.path.abspath(os.path.expanduser(src))
+    if os.path.isfile(_p):
+        return(('file', _p))
+    # Last shot - is it a DNS record?
+    # Not quite perfect, as it's strictly RFC and there are plenty of
+    # subdomains out there that break RFC.
+    f = fqdn.FQDN(src)
+    if f.is_valid:
+        return(('dns', src))
+    return(src_out)
+
+def parseArgs():
+    def chkArg(src):
+        src_out = chkInput(src)
+        if src_out == (None, None):
+            raise argparse.ArgumentTypeError(('"{0}" does not seem to be a ' +
+                                              'path to a file, an A/AAAA ' +
+                                              'record, or IPv4/IPv6 ' +
+                                              'address.').format(src))
+        return(src_out)
+    args = argparse.ArgumentParser(description = desc)
+    args.add_argument('data_in',
+                      type = chkArg,
+                      metavar = 'ADDRESS_OR_FILE',
+                      help = ('The path to a file containing domains and IP ' +
+                              'addresses OR a single IPv4/IPv6 address or ' +
+                              'A/AAAA record. If an A/AAAA record, your ' +
+                              'machine must be able to resolve it (and it ' +
+                              'must exist)'))
+    return(args)
+
+def main():
+    # TODO: clean this up, migrate the duplicated code into a func
+    args = vars(parseArgs().parse_args())['data_in']
+    if args[0] == 'dns':
+        r = resolveRecord(args[1])
+        for k in r.keys():
+            for ip in r[k]:
+                print('IP: {0}'.format(ip))
+                print('PTR: {0}'.format(genPTR(str(ip), k)))
+    elif args[0] in ('ipv4', 'ipv6'):
+        print('PTR: {0}'.format(genPTR(args[1], args[0])))
+    elif args[0] == 'file':
+        with open(args[1], 'r') as f:
+            recordlst = [i.strip() for i in f.readlines()]
+        for i in recordlst:
+            ltype, data = chkInput(i)
+            print('== {0} =='.format(i))
+            if ltype == 'dns':
+                r = resolveRecord(data)
+                for k in r.keys():
+                    for ip in r[k]:
+                        print('IP: {0}'.format(ip))
+                        print('PTR: {0}'.format(genPTR(str(ip), k)))
+            elif ltype in ('ipv4', 'ipv6'):
+                print('PTR: {0}'.format(genPTR(data, ltype)))
+            print()
+
+if __name__ == '__main__':
+    main()

net/get_title.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+
+import argparse
+try:
+    import requests as handler
+    has_req = True
+except ImportError:
+    from urllib.request import urlopen as handler
+    has_req = False
+try:
+    import lxml
+    parser = 'lxml'
+except ImportError:
+    parser = 'html.parser'
+from bs4 import BeautifulSoup
+
+
+def_elem = 'title'
+
+
+class InfoScraper(object):
+    def __init__(self, url, elem = def_elem, *args, **kwargs):
+        self.url = url
+        self.elem = elem
+        self.raw = None
+        self.str = None
+        self.soup = None
+        self._get_page()
+
+    def _get_page(self):
+        if has_req:
+            self.raw = handler.get(self.url).content
+        else:
+            with handler(self.url) as fh:
+                self.raw = fh.read()
+        try:
+            self.str = self.raw.decode('utf-8')
+        except Exception:
+            pass
+        self.soup = BeautifulSoup(self.str, features = parser)
+        return(None)
+
+    def find(self):
+        rtrn = [e for e in self.soup.find_all(self.elem)]
+        return(rtrn)
+        
+
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = 'Get quick information from a URL at a glance')
+    args.add_argument('-e', '--elem',
+                      dest = 'elem',
+                      default = def_elem,
+                      help = ('The element(s) you want to scrape from the page. This is likely just going to be "{0}" (the default)').format(def_elem))
+    args.add_argument('-s', '--strip',
+                      dest = 'strip',
+                      action = 'store_true',
+                      help = ('If specified, strip whitespace at the beginning/end of each element text'))
+    args.add_argument('-d', '--delineate',
+                      dest = 'delin',
+                      action = 'store_true',
+                      help = ('If specified, delineate each element instance'))
+    args.add_argument('-c', '--count',
+                      dest = 'count',
+                      action = 'store_true',
+                      help = ('If specified, provide a count of how many times -e/--elem was found'))
+    args.add_argument('url',
+                      metavar = 'URL',
+                      help = ('The URL to parse. It may need to be quoted or escaped depending on the URL and what shell you\'re using'))
+    return(args)
+
+
+def main():
+    args = parseArgs().parse_args()
+    i = InfoScraper(**vars(args))
+    rslts = i.find()
+    if args.count:
+        print('Element {0} was found {1} time(s) at {2}. Results follow:'.format(args.elem, len(rslts), args.url))
+    for i in rslts:
+        t = i.text
+        if args.strip:
+            t = t.strip()
+        if args.delin:
+            print('== {0}: =='.format(args.elem))
+        print(t)
+        if args.delin:
+            print('==\n')
+    return(None)
+
+
+if __name__ == '__main__':
+    main()
+

net/irc/.gitignore

@@ -0,0 +1 @@
+control2format.pl

net/irc/irssilogparse.py

@@ -0,0 +1,435 @@
+#!/usr/bin/env python3
+
+## REFERENCE ##
+# https://github.com/myano/jenni/wiki/IRC-String-Formatting
+# https://www.mirc.com/colors.html
+# https://en.wikipedia.org/wiki/ANSI_escape_code
+# http://invisible-island.net/xterm/ctlseqs/ctlseqs.html
+# https://github.com/shabble/irssi-docs/wiki/Formats#Colourising-Text-in-IRC-Messages
+# https://askubuntu.com/a/528938
+# <irssi.git>/themes/default.theme and ..docs/formats.txt holds SOME clues to
+# these.
+# e.g.
+# # text to insert at the beginning of each non-message line
+# line_start = "%B-%n!%B-%n ";
+#
+# # timestamp styling, nothing by default
+# timestamp = "$*";
+#
+# # any kind of text that needs hilighting, default is to bold
+# hilight = "%_$*%_";
+#####################
+# ^D = \x04
+# ^D8/ = bold dark grey (14)
+# ^D9/ = cyan (10)
+# ^Dg  = color/fmting? reset
+# ^D;/ = bold light cyan(11)
+# ^Dc  = bold
+# ^D>/ =  (incl. bell for ">"?)
+## The key seems to be /opt/dev/optools/net/irc/irssilogparse.py (& assoc. .c file)
+## see also <irssi.git>/src/core/log.c/h
+## !!! HUGE THANKS to Nei@Freenode#irssi! He pointed me to http://anti.teamidiot.de/static/nei/*/Code/Irssi/control2format.pl
+# which nicely maps those internal command/control chars to the irssi
+# templated stuff in their official docs (e.g. %b).
+
+# Sorry for the copious comments, but the majority of the Irssi log stuff isn't
+# really documented... anywhere. Just the color codes and \x03.
+# And the log2ansi.pl script is... well, perl, so minus points for that, but
+# the names are obtuse and perl's ugly af.
+
+import argparse
+import curses
+import os
+import pprint
+import re
+import sys
+try:
+    import magic
+    has_magic = True
+except ImportError:
+    print('Warning: you do not have the magic module installed (you can '
+          'install it via "pip3 install --user file-magic"). Automatic log '
+          'decompression will not work.')
+    has_magic = False
+
+# This is a map to determine which module to use to decompress,
+# if we should.
+cmprsn_map = {'text/plain': None,  # Plain ol' text
+              # Sometimes the formatting with color gives this
+              'application/octet-stream': None,
+              'application/x-bzip2': 'bz2',  # Bzip2
+              'application/x-gzip': 'gzip',  # Gzip
+              'application/x-xz': 'lzma'}  # XZ
+
+# irssi/mIRC to ANSI
+# Split into 3 maps (truecolor will be populated later, currently uses 8-bit):
+# - 8 (3/4 bit color values, 8 colors)
+# - 256 (8-bit, 256 colors)
+# - 'truecolor' (24-bit, ISO-8613-3, 16777216 colors)
+# Keys are the mIRC color value. Reference the links above for reference on
+# what the values map to.
+# Values are:
+# - 8: tuple for ANSI fg and bg values
+# - 256: single value (same number is used for fg and bg)
+# - 'truecolor': tuple of (R#, G#, B#) (same number is used for fg and bg)
+# In addition, all three have the following:
+# - ansi_wrap: the string formatter.
+#              fg: foreground color
+#              bg: background color (if present)
+#    They are concatted together in that order.
+            ## https://en.wikipedia.org/wiki/ANSI_escape_code#3/4_bit
+colormap = {8: {'0': ('97', '107'),
+                '1': ('30', '40'),
+                '2': ('34', '44'),
+                '3': ('32', '42'),
+                '4': ('91', '101'),
+                '5': ('31', '41'),
+                '6': ('35', '45'),
+                '7': ('33', '43'),
+                '8': ('93', '103'),
+                '9': ('92', '102'),
+                '10': ('36', '46'),
+                '11': ('96', '106'),
+                '12': ('94', '104'),
+                '13': ('95', '105'),
+                '14': ('90', '100'),
+                '15': ('37', '47'),
+                'ansi_wrap': {'fg': '\x1b[{0[0]}',
+                              'bg': ';{0[1]}m'}},
+            ## https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit
+            256: {'0': '15',
+                  '1': '0',
+                  '2': '19',
+                  '3': '34',
+                  '4': '196',
+                  '5': '52',
+                  '6': '90',
+                  '7': '208',
+                  '8': '226',
+                  '9': '82',
+                  '10': '37',
+                  '11': '51',
+                  '12': '21',
+                  '13': '199',
+                  '14': '241',
+                  '15': '252',
+                  'ansi_wrap': {'fg': '\x1b[38;5;{0}m',
+                                'bg': '\x1b[48;5;{0}m'}},
+            ## https://en.wikipedia.org/wiki/ANSI_escape_code#24-bit
+            # (can just use mIRC's R,G,B)
+            'truecolor': {'0': ('255', '255', '255'),
+                          '1': ('0', '0', '0'),
+                          '2': ('0', '0', '127'),
+                          '3': ('0', '147', '0'),
+                          '4': ('255', '0', '0'),
+                          '5': ('127', '0', '0'),
+                          '6': ('156', '0', '156'),
+                          '7': ('252', '127', '0'),
+                          '8': ('255', '255', '0'),
+                          '9': ('0', '252', '0'),
+                          '10': ('0', '147', '147'),
+                          '11': ('0', '255', '255'),
+                          '12': ('0', '0', '252'),
+                          '13': ('255', '0', '255'),
+                          '14': ('127', '127', '127'),
+                          '15': ('210', '210', '210'),
+                          'ansi_wrap': {'fg': '\x1b[38;2;'
+                                              '{0[0]};{0[1]};{0[2]}m',
+                                        'bg': '\x1b[48;2;'
+                                              '{0[0]};{0[1]};{0[2]}m'}}}
+# These are special "control characters" Irssi uses.
+reset_char = '\x1b[0m'
+bold_char = '\x1b[1m'
+invert_char = '\x1b[7m'
+# Used for Irssi-specific escapes/controls.
+irssi_ctrl = {'a': '\x1b[5m',  # Blink
+              'b': '\x1b[4m',  # Underline
+              'c': bold_char,  # Bold
+              'd': invert_char,  # Reverse (unused; color_inverter() is called)
+              #'e': '\t',  # Indent
+              'e': None,  # Indent
+              'f': None,  # "f" is an indent func, so no-op
+              'g': reset_char,  # Reset
+              'h': None,  # Monospace (no-op)
+              '>': bold_char,  # Undocumented? Seems to be bold/highlight.
+              ';': bold_char}  # Undocumented? Seems to be bold/highlight.
+# This is used for inversion on the colors.
+# # the foreground color (dynamic)
+# fg = '\x1b[39m'
+# # the background color (dynamic)
+# bg = '\x1b[49m'
+# the value to reset the foreground text (not changed)
+def_fg = '\x1b[39m'
+# the value to reset the background text (not changed)
+def_bg = '\x1b[49m'
+# if the state is currently inverted (dynamic)
+is_inverted = False
+
+def get_palette():
+    # Return 8, 256, or 'truecolor'
+    colorterm = os.getenv('COLORTERM', None)
+    if colorterm in ('truecolor', '24bit'):
+        # TODO: 24-bit support (16777216 colors) instead of 8-bit.
+        # See note above.
+        #return('truecolor')
+        #return(256)
+        return(8)
+    else:
+        curses.initscr()
+        curses.start_color()
+        c = curses.COLORS
+        curses.endwin()
+        return(c)
+
+def color_inverter(data = None):
+    # global fg
+    # global bg
+    global is_inverted
+    #fg, bg = bg, fg
+    if is_inverted:
+        char = '\x1b[27m'
+    else:
+        char = '\x1b[7m'
+    is_inverted = (not is_inverted)
+    return(char)
+
+
+def color_converter(data_in, palette_map):
+    # Only used if logParser().args['color'] = True
+    # Convert mIRC/Irssi color coding to ANSI color codes.
+    # A sub-function that generates the replacement characters.
+    global fg
+    global bg
+    global is_inverted
+    _colors = colormap[palette_map]
+    def _repl(ch_in):
+        ch_out = ''
+        _ch = {'stripped': re.sub('^[\x00-\x7f]', '', ch_in.group()),
+               #'ctrl': re.sub('^\x04([a-h]|[0-9]|;|>)/?.*$', '\g<1>',
+               'ctrl': re.sub('^\x04([^/])/?.*$', '\g<1>',
+                              ch_in.group())}
+        # We assign this separately as we use an existing dict entry.
+        # This is the "color code" (if it exists).
+        _ch['c'] = [re.sub('^0?([0-9]{1,2}).*',
+                           '\g<1>',
+                           i.strip()) for i in _ch['stripped'].split(',', 1)]
+        # Color-handling
+        #if _ch['ctrl'].startswith('\x03'):
+        if re.search('[\x00-\x03]', _ch['ctrl']):
+            if len(_ch['c']) == 1:
+                fg_only = True
+            elif len(_ch['c']) == 2:
+                fg_only = False
+            else:
+                raise RuntimeError('Parsing error! "{0}"'.format(
+                                                                ch_in.group()))
+            fg = _colors['ansi_wrap']['fg'].format(_colors[_ch['c'][0]])
+            ch_out = fg
+            if not fg_only:
+                bg = _colors['ansi_wrap']['bg'].format(_colors[_ch['c'][1]])
+                ch_out += bg
+            else:
+                if palette_map == 8:
+                    ch_out += 'm'
+        # Control-character handling
+        else:
+            if _ch['ctrl'] in irssi_ctrl:
+                if irssi_ctrl[_ch['ctrl']]:
+                    ch_out = irssi_ctrl[_ch['ctrl']]
+                    if _ch['ctrl'] == 'g':
+                        color_inverter()
+            elif re.search('^[0-9]', _ch['ctrl']):
+                ch_out = _colors['ansi_wrap']['fg'].format(
+                                                        _colors[_ch['c'][0]])
+                if palette_map == 8:
+                    ch_out += 'm'
+            else:
+                # _ch['ctrl'] is not found and we don't have a color number
+                # to look up, so leave ch_out as ''
+                pass
+        return(ch_out)
+    #color_ptrn = re.compile('\x03[0-9]{1,2}(,[0-9]{1,2})?')
+    catch = re.compile('(\x03[0-9]{2}(,[0-9]{1, 2})?|'
+                       '\x04([a-h]|;/|>/|[0-9]/?))')
+    # These are some non-programmatic regexes.
+    # Clean up the nick.
+    nick_re = re.compile('(\x048/\s*)')
+    # mIRC uses a different tag for reset.
+    mirc_rst = re.compile('\x0f')
+    # mIRC and Irssi tags, respectively, for inversion.
+    re_invert = re.compile('(\x16|\x04d)')
+    data = data_in.splitlines()
+    for idx, line in enumerate(data[:]):
+        # Get some preliminary replacements out of the way.
+        line = nick_re.sub(' ', line, 1)
+        line = re_invert.sub(color_inverter, line)
+        line = mirc_rst.sub(reset_char, line)
+        # This is like 90% of the magic, honestly.
+        line = catch.sub(_repl, line)
+        if not line.endswith(reset_char):
+            line += reset_char
+        # Since we clear all formatting at the end of each line
+        is_inverted = False
+        data[idx] = line
+    return('\n'.join(data))
+
+def plain_stripper(data_in):
+    # Strip to plaintext only.
+    data = data_in.splitlines()
+    ptrns = [re.compile('\x04(g|c|[389;]/?|e|>)/?'),
+             re.compile('((\x03)\d\d?,\d\d?|(\x03)\d\d?|[\x01-\x1F])')]
+    for idx, line in enumerate(data[:]):
+        # This cleans the nick field
+        l = re.sub('\x04[89]/', ' ', line, 1)
+        # And these clean the actual chat messages
+        for p in ptrns:
+            l = p.sub('', l)
+        data[idx] = l
+    return('\n'.join(data))
+
+class irssiLogParser(object):
+    def __init__(self, args, data = None):
+        # We'll need these accessible across the entire class.
+        self.args = args
+        # If specified, self.data takes precedence over self.args['logfile']
+        # (if it was specified).
+        self.data = data
+        self.raw = data
+        self.has_html = False
+        self.decompress = None
+        if 'color' in self.args and self.args['color']:
+            if not self.args['html']:
+                # Ensure that we support color output.
+                curses.initscr()
+                self.args['color'] = curses.can_change_color()
+                curses.endwin()
+                if not self.args['color'] and not self.args['raw']:
+                    raise RuntimeError('You have specified ANSI colorized '
+                                       'output but your terminal does not '
+                                       'support it. Use -fc/--force-color '
+                                       'to force.')
+                elif not self.args['color'] and self.args['raw']:
+                    self.args['color'] = True  # Force the output anyways.
+        if self.args['color']:
+            if not self.args['raw']:
+                self.colors = get_palette()
+            else:
+                self.colors = 8  # Best play it safe for maximum compatibility.
+        # The full, interpreted path.
+        if ('logfile' in self.args.keys() and
+            self.args['logfile'] is not None):
+            self.args['logfile'] = os.path.abspath(
+                                        os.path.expanduser(
+                                                self.args['logfile']))
+        if not self.data:
+            self.getlog()
+        else:
+            # Conform everything to bytes.
+            if not isinstance(self.data, bytes):
+                self.data = self.data.encode('utf-8')
+        self.decompressor()
+        self.parser()
+
+    def getlog(self):
+        # A filepath was specified
+        if self.args['logfile']:
+            if not os.path.isfile(self.args['logfile']):
+                raise FileNotFoundError('{0} does not exist'.format(
+                                                        self.args['logfile']))
+            with open(self.args['logfile'], 'rb') as f:
+                self.data = f.read()
+        # Try to get it from stdin
+        else:
+            if not sys.stdin.isatty():
+                self.data = sys.stdin.buffer.read()
+            else:
+                raise ValueError('Either a path to a logfile must be '
+                                 'specified or you must pipe a log in from '
+                                 'stdin.')
+        self.raw = self.data
+        return()
+
+    def decompressor(self):
+        # TODO: use mime module as fallback?
+        # https://docs.python.org/3/library/mimetypes.html
+        # VERY less-than-ideal since it won't work without self.args['logfile']
+        # (and has iffy detection at best, since it relies on file extensions).
+        # Determine what decompressor to use, if we need to.
+        if has_magic:
+            _mime = magic.detect_from_content(self.data).mime_type
+            self.decompress = cmprsn_map[_mime]
+            if self.decompress:
+                import importlib
+                decmp = importlib.import_module(self.decompress)
+                self.raw = decmp.decompress(self.data)
+        else:
+            # Assume that it's text and that it isn't compressed.
+            # We'll get a UnicodeDecodeError exception if it isn't.
+            pass
+        try:
+            self.raw = self.data.decode('utf-8')
+        except UnicodeDecodeError:
+            pass
+        self.data = self.raw
+        return()
+
+    def parser(self):
+        if 'color' not in self.args or not self.args['color']:
+            self.data = plain_stripper(self.data)
+        else:
+            self.data = color_converter(self.data, self.colors)
+        # Just in case...
+        self.data += '\x1b[0m'
+        return()
+
+def parseArgs():
+    args = argparse.ArgumentParser()
+    args.add_argument('-c', '--color',
+                      dest = 'color',
+                      action = 'store_true',
+                      help = ('Print the log with converted colors (ANSI)'))
+    args.add_argument('-r', '--raw',
+                      dest = 'raw',
+                      action = 'store_true',
+                      help = ('Use this switch if your terminal is detected '
+                              'as not supporting color output but wish to '
+                              'force it anyways. A string representation of '
+                              'the ANSI output will be produced instead ('
+                              'suitable for pasting elsewhere). Only used if '
+                              '-c/--color is enabled (ignored with '
+                              '-H/--html)'))
+    args.add_argument('-H', '--html',
+                      dest = 'html',
+                      action = 'store_true',
+                      help = ('Render HTML output (requires ansi2html)'))
+    args.add_argument(dest = 'logfile',
+                      default = None,
+                      nargs = '?',
+                      metavar = 'path/to/logfile',
+                      help = ('The path to the log file. It can be uncompressed ' +
+                              'or compressed with XZ/LZMA, Gzip, or Bzip2. '
+                              'If not specified, read from stdin'))
+    return(args)
+
+if __name__ == '__main__':
+    args = vars(parseArgs().parse_args())
+    l = irssiLogParser(args)
+    import shutil
+    cols = shutil.get_terminal_size().columns
+    #print('ARGS:')
+    #pprint.pprint(l.args, width = cols)
+    # print('RAW')
+    # pprint.pprint(l.raw, width = cols)
+    # with open('/tmp/freenode.formatted', 'r') as f:
+    #     print(f.read())
+    #print('DATA')
+    #pprint.pprint(l.data, width = cols)
+    #print('DATA (REPR)')
+    #pprint.pprint(repr(l.data).split('\\n'))
+    print('DATA')
+    print(l.data)
+    with open('/tmp/log.raw', 'w') as f:
+        for line in repr(l.data).split('\\n'):
+            f.write(line + '\n')
+    # l.parseLog()
+    # print(l.data.decode('utf-8'))

net/mirroring/.gitignore

@@ -0,0 +1,2 @@
+urls.csv
+/cache

net/mirroring/check.py

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+
+import argparse
+import csv
+import datetime
+import difflib
+import hashlib
+import lzma
+import os
+import pickle
+from urllib.request import urlopen
+
+# TODO: to avoid race conditions, we should probably simply ignore/remove
+# timestamps and just touch the cache file whenever checking.
+
+class website(object):
+    def __init__(self, args, csvline):
+        # Field names
+        self.fnames = ('UUID', 'url', 'checksum', 'timestamp')
+        self.args = args
+        self.parseCSV([csvline])
+        self.cache = args['cache_dir']
+        self.cacheControl()
+        self.remoteFetch()
+        return
+
+    def parseCSV(self, data):
+        _rows = csv.DictReader(data,
+                               fieldnames = self.fnames,
+                               delimiter = ',',
+                               quotechar = '"')
+        for r in _rows:
+            self.meta = r
+            break  # We only want one, so if we SOMEHOW got more than one line...
+        return()
+
+    def cacheControl(self):
+        os.makedirs(self.cache, exist_ok = True)
+        self.site = {}
+        _cachefile = os.path.join(self.cache, self.meta['UUID'])
+        if os.path.isfile(_cachefile):
+            with lzma.open(_cachefile, mode = 'rb') as f:
+                self.site['local'] = pickle.load(f)
+        else:
+            with urlopen(self.meta['url']) as _site,\
+                 lzma.open(_cachefile,
+                           mode = 'wb',
+                           check = lzma.CHECK_SHA256,
+                           preset = 9|lzma.PRESET_EXTREME) as f:
+                _data = _site.read().decode('utf-8')
+                pickle.dump(_data, f)
+                self.site['local'] = _data
+                self.meta['timestamp'] = str(int(datetime.datetime.now().timestamp()))
+                _hash = hashlib.sha256(self.site['local'].encode('utf-8'))
+                self.meta['checksum'] = str(_hash.hexdigest())
+        return()
+
+    def remoteFetch(self):
+        with urlopen(self.meta['url']) as _site:
+            self.site['remote'] = _site.read()
+            self.headers = dict(_site.info())
+        # Handle gzip encoding
+        if 'Content-Encoding' in self.headers.keys():
+            if self.headers['Content-Encoding'] == 'gzip':
+                from gzip import decompress
+                self.site['remote'] = decompress(self.site['remote']).decode('utf-8')
+        else:
+            self.site['remote'] = self.site['remote'].decode('utf-8')
+        _hash = hashlib.sha256(self.site['remote'].encode('utf-8'))
+        self.site['remotesum'] = str(_hash.hexdigest())
+        self.meta['timestamp'] = str(int(datetime.datetime.now().timestamp()))
+        return()
+
+    def compare(self):
+        # Don't even compare if the checksums match.
+        if self.site['remotesum'] == self.meta['checksum']:
+            self.diff = None
+            #print('{0}: Doing nothing'.format(self.meta['UUID']))
+            return()
+        print('{{{0}}}: "{1}":'.format(self.meta['UUID'], self.meta['url']))
+        diff = difflib.unified_diff(self.site['local'].splitlines(1),
+                                    self.site['remote'].splitlines(1))
+        self.diff = ''.join(diff)
+        print(self.diff)
+        with urlopen(self.meta['url']) as _site,\
+                 lzma.open(os.path.join(self.cache, self.meta['UUID']),
+                           mode = 'wb',
+                           check = lzma.CHECK_SHA256,
+                           preset = 9|lzma.PRESET_EXTREME) as f:
+                _data = _site.read().decode('utf-8')
+                pickle.dump(_data, f)
+        return()
+
+    def writeCSV(self):
+        #if self.diff:  # We actually WANT to write, because we're updating the last fetch timestamp.
+        _lines = []
+        with open(self.args['urls_csv'], 'r') as f:
+            _f = f.read()
+        _rows = csv.DictReader(_f.splitlines(),
+                               fieldnames = self.fnames,
+                               delimiter = ',',
+                               quotechar = '"')
+        for r in _rows:
+            _uuid = r['UUID']
+            if _uuid == self.meta['UUID']:
+                r['checksum'] = self.site['remotesum']
+                r['timestamp'] = self.meta['timestamp']
+            _lines.append(r)
+        with open(self.args['urls_csv'], 'w', newline = '') as f:
+            _w = csv.DictWriter(f,
+                                fieldnames = self.fnames,
+                                delimiter = ',',
+                                quotechar = '"',
+                                quoting = csv.QUOTE_ALL)
+            _w.writerows(_lines)
+        return()
+
+def parseArgs():
+    # Define defaults
+    _self_dir = os.path.dirname(os.path.realpath(__file__))
+    _cache_dir = os.path.join(_self_dir, 'cache')
+    _urls_csv = os.path.join(_self_dir, 'urls.csv')
+    args = argparse.ArgumentParser()
+    args.add_argument('-c',
+                      '--cache-dir',
+                      metavar = '/path/to/cache/dir/',
+                      default = _cache_dir,
+                      dest = 'cache_dir',
+                      type = str,
+                      help = ('The path to where cached versions of websites are stored. ' +
+                              'They are stored in the python binary "pickle" format. ' +
+                              'Default: \n\n\t\033[1m{0}\033[0m').format(_cache_dir))
+    args.add_argument('-u',
+                      '--urls',
+                      metavar = '/path/to/urls.csv',
+                      default = _urls_csv,
+                      dest = 'urls_csv',
+                      type = str,
+                      help = ('The path to where a CSV file of the URLs to check should be. ' +
+                              'Note that it should be writeable by whatever user the script is running as.' +
+                              'See urls.csv.spec for the specification. ' +
+                              'Default: \n\n\t\033[1m{0}\033[0m').format(_urls_csv))
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    for d in ('cache_dir', 'urls_csv'):
+        args[d] = os.path.realpath(os.path.expanduser(args[d]))
+    with open(args['urls_csv'], 'r', newline = '') as f:
+        _csv = f.read()
+    for line in _csv.splitlines():
+        w = website(args, line)
+        w.compare()
+        w.writeCSV()
+        if w.diff:
+            print(w.diff)
+
+if __name__ == '__main__':
+    main()

net/mirroring/urls.csv.spec

@@ -0,0 +1,15 @@
+"UUID","URL","SHA512_of_CONTENT","LAST_FETCHED_IN_UNIX_EPOCH"
+
+UUID can be any non-whitespace, non-slashed string suitable for filenames you want, but I recommend a UUID4.
+You can generate one at either https://www.uuidgenerator.net/ or via python:
+
+>>> import uuid
+>>> str(uuid.uuid4())
+'16728c9e-5fde-4f63-8a36-4a3db612be8d'
+
+It should be unique for every page.
+
+
+You can generate an UNIX Epoch timestamp via:
+
+  date '+%s'

net/ssh/audit.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+
+import argparse
+import paramiko
+import socket
+
+
+class SSHAuthInfo(object):
+    def __init__(self, target, port = 22, banner = True, ciphers = True, digests = True, kex = True, key_types = True,
+                 methods = True, hostkeys = True, version = True):
+        self.target = target
+        self.port = int(port)
+        self.info = {'target': self.target,
+                     'port': self.port,
+                     'banner': banner,
+                     'ciphers': ciphers,
+                     'digests': digests,
+                     'kex': kex,
+                     'key_types': key_types,
+                     'methods': methods,
+                     'hostkeys': hostkeys,
+                     'version': version}
+        self._ssh = None
+        if any((ciphers, banner, methods, digests, kex, key_types)):  # These need an SSH connection.
+            self._ssh_dummy()
+        if banner:
+            self.getBanner()
+        if hostkeys:
+            self.getHostkeys()
+        if version:
+            self.getVersion()
+        self._close()
+
+    def _ssh_dummy(self):
+        self._ssh = paramiko.Transport((self.target, self.port))
+        self._ssh.connect()
+        try:
+            self._ssh.auth_none('')
+        except paramiko.ssh_exception.BadAuthenticationType as err:
+            secopts = self._ssh.get_security_options()
+            if self.info['methods']:
+                # https://stackoverflow.com/a/1257769
+                self.info['methods'] = err.allowed_types
+            if self.info['ciphers']:
+                self.info['ciphers'] = list(secopts.ciphers)
+            if self.info['digests']:
+                self.info['digests'] = list(secopts.digests)
+            if self.info['kex']:
+                self.info['kex'] = list(secopts.kex)
+            if self.info['key_types']:
+                self.info['key_types'] = list(secopts.key_types)
+        return()
+
+    def getBanner(self):
+        self.info['banner'] = None
+        # https://github.com/paramiko/paramiko/issues/273#issuecomment-225058645 doesn't seem to work.
+        # But https://github.com/paramiko/paramiko/pull/58#issuecomment-63857078 did!
+        self.info['banner'] = self._ssh.get_banner()
+        return()
+
+    def getHostkeys(self):
+        # TODO: how the hell do I get *all* hostkeys served?
+        self.info['hostkeys'] = {}
+        k = self._ssh.get_remote_server_key()
+        self.info['hostkeys'][k.get_name()] = k.get_base64()
+        return()
+
+    def getVersion(self):
+        self.info['version'] = None
+        s = socket.socket()
+        s.connect((self.target, self.port))
+        try:
+            # 8192 bytes is kind of overkill considering most are probably going to be around 20 bytes or so.
+            self.info['version'] = s.recv(8192)
+        except Exception as e:
+            pass
+        return()
+
+    def _close(self):
+        if self._ssh:
+            self._ssh.close()
+
+def parseArgs():
+    args = argparse.ArgumentParser()
+    args.add_argument('-b', '--no-banner',
+                      action = 'store_false',
+                      dest = 'banner',
+                      help = 'Do not gather the SSH banner')
+    args.add_argument('-c', '--no-ciphers',
+                      action = 'store_false',
+                      dest = 'ciphers',
+                      help = 'Do not gather supported ciphers')
+    args.add_argument('-d', '--no-digests',
+                      action = 'store_false',
+                      dest = 'digests',
+                      help = 'Do not gather supported digests')
+    args.add_argument('-m', '--no-methods',
+                      action = 'store_false',
+                      dest = 'methods',
+                      help = 'Do not gather supported auth methods')
+    args.add_argument('-k', '--no-hostkeys',
+                      action = 'store_false',
+                      dest = 'hostkeys',
+                      help = 'Do not gather hostkeys')
+    args.add_argument('-x', '--no-kex',
+                      action = 'store_false',
+                      dest = 'kex',
+                      help = 'Do not gather supported key exchanges')
+    args.add_argument('-t', '--no-key-types',
+                      action = 'store_false',
+                      dest = 'key_types',
+                      help = 'Do not gather supported key types')
+    args.add_argument('-v', '--no-version',
+                      action = 'store_false',
+                      dest = 'version',
+                      help = 'Do not gather SSH version')
+    args.add_argument('-p', '--port',
+                      default = 22,
+                      help = 'The port on target that the SSH daemon is running on. Default is 22')
+    args.add_argument('target',
+                      help = 'The server to run the check against')
+    return(args)
+
+def main():
+    args = vars(parseArgs().parse_args())
+    i = SSHAuthInfo(**args)
+    import pprint
+    pprint.pprint(i.info)
+
+if __name__ == '__main__':
+    main()

net/ssh/hostkeymanager/app/__init__.py

@@ -0,0 +1,7 @@
+from flask import Flask
+
+app = Flask(__name__, instance_relative_config=True)
+
+from app import views
+
+app.config.from_object('config')

net/ssh/hostkeymanager/app/manage.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+
+import argparse
+import sys
+import os
+# This is ugly as fuck. TODO: can we do this more cleanly?
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir)))
+import config
+
+class DBmgr(object):
+    def __init__(self, args = None):
+        self.DB = config.DB
+        self.args = args
+
+    def keyChk(self):
+        # Is it a pubkey file?
+        if os.path.isfile(os.path.abspath(os.path.expanduser(self.args['key']))):
+            with open(os.path.abspath(os.path.expanduser(self.args['key'])), 'r') as f:
+                self.args['key'] = f.read()
+        self.args['key'] = self.args['key'].strip()
+
+
+    def add(self, key, host, role):
+        pass
+
+def argParse():
+    args = argparse.ArgumentParser()
+    args.add_argument('-k',
+                      '--key',
+                      dest = 'key',
+                      default = None,
+                      type = 'str',
+
+    return(args)
+
+def main():
+    args -
+    d = DBmgr(args)
+
+if __name__ == '__main__':
+    main()

net/ssh/hostkeymanager/app/templates/about.html

@@ -0,0 +1,4 @@
+{% extends "base.html" %}{% block title %}r00t^2 SSH Key Repository || About{% endblock %}{% block body %}<div class="jumbotron">
+<h1>About</h1></div>
+<p>This is a tool to deliver SSH public keys (or, optionally, host keys) to SSH's authentication system in a safe and secure manner.</p>
+{% endblock %}

net/ssh/hostkeymanager/app/templates/base.html

@@ -0,0 +1,35 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <title>{% block title %}{% endblock %}</title>
+  <!-- Bootstrap core CSS -->
+  <!-- Thanks, https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xii-facelift and
+        https://scotch.io/tutorials/getting-started-with-flask-a-python-microframework -->
+  <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet">
+  <!--<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta/css/bootstrap.min.css" integrity="sha384-/Y6pD6FV/Vv2HJnA6t+vslU6fwYXjCFtcEpHbNJ0lyAFsXTsjBbfaDjzALeQsN6M" crossorigin="anonymous">-->
+  <!-- Custom styles for this template -->
+  <link href="https://getbootstrap.com/examples/jumbotron-narrow/jumbotron-narrow.css" rel="stylesheet">
+  <!--<link href="https://getbootstrap.com/docs/4.0/examples/offcanvas/offcanvas.css" rel="stylesheet">-->
+</head>
+<body>
+  <div class="container">
+    <div class="header clearfix">
+      <nav>
+        <ul class="nav nav-pills pull-right">
+          <li role="presentation"><a href="/">Home</a></li>
+          <li role="presentation"><a href="/about">About</a></li>
+          <li role="presentation"><a href="/usage">Usage</a></li>
+          <!-- the following opens in a new tab/window/whatever. the line after opens in the same tab/window/etc. -->
+          <!-- <li role="presentation"><a href="https://square-r00t.net/" target="_blank">r00t^2</a></li> -->
+          <li role="presentation"><a href="https://square-r00t.net/">r00t^2</a></li>
+        </ul>
+      </nav>
+    </div>
+    {% block body %}{% endblock %}
+    <footer class="footer">
+      <p><sub>The code for this page is released under the <a href="https://www.gnu.org/licenses/gpl-3.0.en.html#content">GPL 3.0 License</a>. It can be found <a href="https://git.square-r00t.net/OpTools/tree/net">here</a>.</sub></p>
+    </footer>
+  </div>
+  <!-- /container -->
+</body>
+</html>

net/ssh/hostkeymanager/app/templates/html.html

@@ -0,0 +1,38 @@
+<h2>Client/Browser Information</h2>
+<p>This is information that your browser sends with its connection.</p>
+<p>
+  <ul>
+      <li><b>Client IP:</b> <a href="https://ipinfo.io/{{ visitor['ip'] }}">{{ visitor['ip'] }}</a></li>
+      <li><b>Browser:</b> {{ '<a href="{0}">{1}</a>'.format(browsers[visitor['client']['browser']][0],
+                                                            browsers[visitor['client']['browser']][1])|safe
+                              if visitor['client']['browser'] in browsers.keys()
+                              else visitor['client']['browser'].title()
+                                if visitor['client']['browser'] is not none
+                                else '(N/A)' }}</li>
+    <li><b>Language/Locale:</b> {{ visitor['client']['language'] or '(N/A)' }}</li>
+{%- set alt_os = alts[visitor['client']['os']] if visitor['client']['os'] in alts.keys() else '' %}
+      <li><b>Operating System:</b> {{ '<a href="{0}">{1}</a>{2}'.format(os[visitor['client']['os']][0],
+                                                                        os[visitor['client']['os']][1],
+                                                                        alt_os)|safe
+                                      if visitor['client']['os'] in os.keys()
+                                      else visitor['client']['os'].title()
+                                        if visitor['client']['os'] is not none
+                                        else '(N/A)' }}</li>
+      <li><b>User Agent:</b> {{ visitor['client']['str'] }}</li>
+      <li><b>Version:</b> {{ visitor['client']['version'] or '(N/A)' }}</li>
+  </ul>
+</p>
+<h2>Request Headers</h2>
+<p>These are headers sent along with the request your browser sends for the page's content.</p>
+<p>
+  <table>
+    <tr>
+      <th>Field</th>
+      <th>Value</th>
+    </tr>{% for k in visitor['headers'].keys()|sort(case_sensitive = True) %}
+    <tr>
+      <td>{{ k }}</td>
+      <td>{{ visitor['headers'][k] if visitor['headers'][k] != '' else '(N/A)' }}</td>
+    </tr>{% endfor %}
+  </table>
+</p>

net/ssh/hostkeymanager/app/templates/index.html

@@ -0,0 +1,6 @@
+{% extends "base.html" %}{% block title %}r00t^2 Client Info Revealer{% endblock %}{% block body %}<div class="jumbotron">
+  <h1>Client Info Revealer</h1>
+  <p class="lead">A tool to reveal client-identifying data sent to webservers</p>
+</div>
+{% include 'html.html' if not params['json'] else 'json.html' %}
+{% endblock %}

net/ssh/hostkeymanager/app/templates/json.html

@@ -0,0 +1 @@
+<pre>{{ json }}</pre>

net/ssh/hostkeymanager/app/templates/usage.html

@@ -0,0 +1,51 @@
+{% extends "base.html" %}{% block title %}r00t^2 Client Info Revealer || Usage{% endblock %}{% block body %}<div class="jumbotron">
+<h1>Usage</h1></div>
+<h2>Parameters</h2>
+<p>You can control how this page displays/renders. By default it will try to "guess" what you want; e.g. if you access it in Chrome, it will display this page but if you fetch via Curl, you'll get raw JSON. The following parameters control this behavior.</p>
+<p><i><b>Note:</b> "Enabled" parameter values can be one of <b>y</b>, <b>yes</b>, <b>1</b>, or <b>true</b>. "Disabled" parameter values can be one of <b>n</b>, <b>no</b>, <b>0</b>, or <b>false</b>. The parameter names are case-sensitive but the values are not.</i></p>
+<p><ul>
+  <li><b>json:</b> Force rendering in JSON format
+    <ul>
+      <li>It will display it nicely if you're in a browser, otherwise it will return raw/plaintext JSON.</li>
+      <li>Use <b>raw</b> if you want to force raw plaintext JSON output.</li>
+    </ul></li>
+  <li><b>html:</b> Force rendering in HTML
+  <ul>
+    <li>It will render HTML in clients that would normally render as JSON (e.g. curl, wget).</li>
+  </ul></li>
+  <li><b>raw:</b> Force output into a raw JSON string
+    <ul>
+      <li>Pure JSON instead of HTML or formatted JSON. This is suitable for API usages if your client is detected wrongly (or you just want to get the raw JSON).</li>
+      <li>Overrides all other tags.</li>
+      <li>Has no effect for clients that would normally render as JSON (curl, wget, etc.).</li>
+    </ul></li>
+  <li><b>tabs:</b> Indentation for JSON output
+    <ul>
+      <li>Accepts a positive integer.</li>
+      <li>Default is 4 for "desktop" browsers (if <b>json</b> is enabled), and no indentation otherwise.</li>
+    </ul></li>
+</ul></p>
+<h2>Examples</h2>{% set scheme = 'https' if request.is_secure else 'http'%}
+<p><table>
+  <tr>
+    <th>URL</th>
+    <th>Behavior</th>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/">{{ scheme }}://{{ request.headers['host'] }}/</a></td>
+    <td>Displays HTML and "Human" formatting if in a graphical browser, otherwise returns a raw, unformatted JSON string.</td>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/?raw=1">{{ scheme }}://{{ request.headers['host'] }}/?raw=1</a></td>
+    <td>Renders a raw, unformatted JSON string if in a graphical browser, otherwise no effect. All other parameters ignored (if in a graphical browser).</td>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/?html=1">{{ scheme }}://{{ request.headers['host'] }}/?html=1</a></td>
+    <td>Forces HTML rendering on non-graphical clients.</td>
+  </tr>
+  <tr>
+    <td><a href="{{ scheme }}://{{ request.headers['host'] }}/?json=1&tabs=4">{{ scheme }}://{{ request.headers['host'] }}/?json=1&tabs=4</a></td>
+    <td>Returns JSON indented by 4 spaces for each level (you can leave "json=1" off if it's in a non-graphical browser, unless you specified "html=1").</td>
+  </tr>
+</table></p>
+{% endblock %}

net/ssh/hostkeymanager/app/views.py

@@ -0,0 +1,57 @@
+import json
+import re
+from flask import render_template, make_response, request
+from app import app
+
+@app.route('/', methods = ['GET'])  #@app.route('/')
+def index():
+    hostkeys = None  # TODO: hostkeys go here. dict?
+    # First we define interactive browsers
+    _intbrowsers = ['camino', 'chrome', 'firefox', 'galeon',
+                    'kmeleon', 'konqueror', 'links', 'lynx']
+    # Then we set some parameter options for less typing later on.
+    _yes = ('y', 'yes', 'true', '1', True)
+    _no = ('y', 'no', 'false', '0', False, 'none')
+    # http://werkzeug.pocoo.org/docs/0.12/utils/#module-werkzeug.useragents
+    # We have to convert these to strings so we can do tuple comparisons on lower()s.
+    params = {'json': str(request.args.get('json')).lower(),
+              'html': str(request.args.get('html')).lower(),
+              'raw': str(request.args.get('raw')).lower()}
+    if request.user_agent.browser in _intbrowsers:
+        if params['html'] == 'none':
+            params['html'] = True
+            if params['json'] == 'none':
+                params['json'] = False
+            elif params['json'] in _yes:
+                params['json'] = True
+    for k in params.keys():
+        if params[k] in _no:
+            params[k] = False
+        else:
+            params[k] = True
+    # Set the tabs for JSON
+    try:
+        params['tabs'] = int(request.args.get('tabs'))
+    except (ValueError, TypeError):
+        if request.user_agent.browser in _intbrowsers or params['html']:
+            params['tabs'] = 4
+        else:
+            params['tabs'] = None
+    j = json.dumps(hostkeys, indent = params['tabs'])
+    if (request.user_agent.browser in _intbrowsers and params['html'] and not params['raw']) or \
+        (request.user_agent.browser not in _intbrowsers and params['html']):
+        return(render_template('index.html', hostkeys = hostkeys))
+    else:
+        if visitor['client']['browser'] in _intbrowsers.keys() and not params['raw']:
+            return(render_template('json.html',
+                                   json = j,
+                                   params = params))
+        return(j)
+
+@app.route('/about', methods = ['GET'])
+def about():
+    return(render_template('about.html'))
+
+@app.route('/usage', methods = ['GET'])
+def usage():
+    return(render_template('usage.html'))

net/ssh/hostkeymanager/config.py

@@ -0,0 +1,8 @@
+# config.py
+
+# Flask debugging - DISABLE FOR PRODUCTION ENVIRONMENTS
+#DEBUG = True
+DEBUG = False
+
+# Path to your Sqlite3 DB
+DB = '/var/local/db/optools/ssh_keys.sqlite3'

net/ssh/hostkeymanager/run.py

@@ -0,0 +1,4 @@
+from app import app
+
+if __name__ == '__main__':
+    app.run()

net/ssh/hostkeymanager/uwsgi.ini

@@ -0,0 +1,18 @@
+[uwsgi]
+plugin = python
+py-autoreload = 1
+#uid = http
+#gid = http
+socket = /run/uwsgi/netinfo.sock
+chown-socket = http:http
+processes = 4
+master = 1
+base = /usr/local/lib/optools/net/ssh
+chdir = %(base)
+#mount = /=%(base)/run.py
+wsgi-file = %(base)/run.py
+chmod-socket = 660
+callable = app
+cgi-helper =.py=python
+logto = /var/log/uwsgi/%n.log
+vacuum

net/ssh/keygen/keygenstructs.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+
+# Generate a struct for generating keys.
+
+class constructor(object):
+    def __init__(self, keyblob, encrypted = False):
+        self.keyblob = keyblob

net/ssh/keygen/keystructs.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+
+# Parse existing keys
+
+class constructor(object):
+    # These are various struct formats for the "new"-style OpenSSH private keys.
+    # REF1: https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL.key?annotate=1.1
+    # REF2: https://github.com/openssh/openssh-portable/blob/94bc1e7ffba3cbdea8c7dcdab8376bf29283128f/sshkey.c
+    def __init__(self, ssh_keyblob):
+        # "keyblob" is the raw binary of an extracted (i.e. "---...---"-removed) and un-base64'd private key.
+        self.keyblob = ssh_keyblob
+        # 'encrypted' if it's encrypted, 'none' if plaintext. This is determined via processing.
+        self.enctype = 'none'
+        # This is the header. It is used by both encrypted and unencrypted keys.
+        self.header = ''.join((
+                            '14cx', # "openssh-key-v1" and null byte (6f70656e7373682d6b65792d7631 00) ("magic bytes")
+                            'i'     # separator, always 00000004
+                            ))
+        # Only two cipher types that I know of that are used.
+        self.ciphertype = {
+                            'none': '4c',           # 6e6f6e65
+                            'aes256-cbc': '10c'}    # 6165733235362d636263 ("aes256-cbc")
+        # This separator is present in both.
+        self.sep1 = 'i'
+        # The name of the key encryption, if encrypted. These are the only two I know of that are used.
+        self.kdfname = {
+                            'none': '4c',       # 6e6f6e65 ("none")
+                            'bcrypt': '6c'}     # 626372797074 ("bcrypt")
+        ########################################### ENCRYPTED KEYS ONLY ################################################
+        # KDF options
+        self.kdfopts = {
+                            'none': '0i',       # zero-length
+                            'encrypted': '24i'} # 24-length int
+        # The length of the salt. Default is 16 (REF2:67)
+        self.saltlen = {
+                            'none': '0i',       # TODO: do unencrypted still have salts?
+                            'encrypted': '4i'}  # 16 bytes length?
+        # The key needs to be parsed incrementally to have these lengths adjusted.
+        self.salt = {
+                            'none': '0c',
+                            'encrypted': '4c'}  # This value may change based on self.saltlen['encrypted']'s value.
+        # The number of rounds for the key; default is 16 (REF2:69).
+        self.rounds = {
+                            'none': '0i',       # TODO: do unencrypted still have rounds?
+                            'encrypted': '16i'}
+        ################################################################################################################
+        # Number of public keys.
+        self.numpub = 'i'
+        # That's all we can populate now. The rest is handled below via functions.
+        self._chkencrypt()
+
+    def _chkencrypt(self):
+        pass
+
+
+class legacy_constructor(object):
+    # These are various struct formats for the "old"-style OpenSSH private keys.
+    def __init__(self, keyblob):
+        self.keyblob = keyblob

net/ssh/puttyconv.py

@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+
+## INCOMPLETE ##
+# TODO
+
+import argparse
+
+def parseArgs():
+    args = argparse.ArgumentParser(description = 'Convert private and public SSH keys between PuTTY/OpenSSH format')
+    args.add_argument('-l', '--legacy',
+                      dest = 'legacy_ssh',
+                      action = 'store_true',
+                      help = ('If specified, try to handle the OpenSSH key as the legacy format ("") rather than the '
+                              'newer '
+                              'more compact format ("")'))
+    ktype = args.add_mutually_exclusive_group()
+    ktype.add_argument('-s', '--ssh',
+                       dest = 'ktype',
+                       const = 'openssh',
+                       help = ('Force the conversion to OpenSSH format'))
+    ktype.add_argument('-p', '--putty',
+                       dest = 'ktype',
+                       const = 'putty',
+                       help = ('Force the conversion to PuTTY format'))

ref/ascii/ascii.adoc

@@ -0,0 +1,130 @@
+= ASCII Character Reference
+Brent Saner <r00t@square-r00t.net>
+Last updated/rendered {localdatetime}
+:doctype: book
+:data-uri:
+:imagesdir: images
+:sectlinks:
+ifeval::["{doctype}" != "article"]
+:toc: preamble
+:toc2: left
+endif::[]
+:idprefix:
+:toclevels: 7
+:docinfo: shared
+:source-highlighter: highlightjs
+
+
+This document attempts to categorize and give reference 
+for all ASCII codes and their various representations.
+I primarily used http://www.profdavis.net/ASCII_table.pdf[this PDF^] and 
+http://www.robelle.com/smugbook/ascii.html[this website^] as references.
+
+== How to Use This Document
+
+=== How to Render It
+
+The document is written in https://asciidoc.org/[AsciiDoc^]
+(and rendered with https://asciidoctor.org/[AsciiDoctor^]).
+The most up-to-date AsciiDoc code can be found at
+https://git.square-r00t.net/OpTools/tree/ref/ascii[my OpTools repo^].
+You may see some things that don't quite match the AsciiDoc spec in the source; 
+that's because they're probably AsciiDoctor-specific extensions.
+
+You can render it via `asciidoctor -n ascii.adoc`. By default it will render HTML5.
+You can also render to PDF via `asciidoctor-pdf -n ascii.adoc`.
+
+=== How to Read It
+
+Each section under <<reference_tables, Reference Tables>> has information about 
+the following tables. Each table thereafter has the following columns in order:
+
+*DEC*:: The decimal representation of the character (i.e. what would be returned in Python's `ord()`).
+*OCT*:: The octal representation of the character.
+*HEX*:: The hexadecimal representation of the character.
+*BIN*:: The binary representation of the character.
+*HTML*:: The HTML number representation of the character.
+*ESCAPE*:: The HTML escape of the character (if it has one), e.g. `&amp;quot;`.
+(Does not include the full set of https://dev.w3.org/html5/html-author/charref[HTML5 characters^], 
+only characters that should be escaped.)
+*LIT*:: The literal character (if it is unprintable, its symbol name will be given instead in _italics_).
+*DESC*:: A description of the character.
+
+If any fields are not relevant, possible, defined, printable, etc. then they will be "N/A" (without the quotes).
+
+////
+TODO
+You will also find an <<index, Index>> to help you find a certain character more quickly, as well as reverse-lookup.
+
+In this index:
+
+* the literal character is in regular style
+* the decimal is a regular style positive integer
+* the octal is in *bold* 
+* the hex is in _italics_
+* the HTML is in *_bold italics_*
+* the binary is in `fixed-width/monospace`
+* the HTML Escape (if it has one) is in `*bold fixed-width*`
+////
+
+== Reference Tables
+
+ASCII ("**A**merican **S**tandard **C**ode for **I**nformation **I**nterchange") is 
+a series of 7-bit sequences in which each single bit represents a character.
+
+The tables given provide the information in 8 bits (256 characters total) per 
+https://en.wikipedia.org/wiki/ISO/IEC_8859-1[ISO 8859-1^], commonly referred to 
+as the `Latin-1` or `latin1` character set, in order to uniformly include the
+extended ASCII set.
+
+You may see some characters link to a https://en.wikipedia.org/[Wikipedia^] article in their description.
+This is typically done because the symbol/character is known by multiple different names or they're uncommon.
+
+You can, of course, copy the character directly from this page into your clipboard (if your OS supports it).
+
+=== ASCII Control Characters
+(DEC 0-31, OCT **000**-**037**, HEX __00__-__1f__)
+
+These characters represent control codes -- characters that alter the environment. 
+They're primarily used these days for *nix (UNIX, BSD, GNU/Linux) terminals. 
+Historically they have been used for things like line printers.
+
+You may see things in `fixed-width` in the description; these are 
+https://en.wikipedia.org/wiki/Software_flow_control[flow controls^] (commonly 
+used on e.g. RS-232).
+
+.Control Characters
+include::tables/_hdr.adoc[]
+include::tables/ctrl.adoc[]
+|===
+
+=== ASCII Printable Characters
+(DEC 32-127, OCT **040**-**177**, HEX __20__-__7f__)
+
+.Printable Characters
+include::tables/_hdr.adoc[]
+include::tables/print.adoc[]
+|===
+
+=== Extended ASCII Characters
+(DEC 128-255, OCT **200**-**377**, HEX __80__-__ff__)
+
+.Extended Characters
+include::tables/_hdr.adoc[]
+include::tables/extend.adoc[]
+|===
+
+=== Combined Table (All Characters/Codes)
+(DEC 0-255, OCT **000**-**377**, HEX __00__-__ff__)
+
+The following table is a combined table of the previous three sections for ease 
+of reference.
+
+.All Characters
+include::tables/_hdr.adoc[]
+include::tables/ctrl.adoc[]
+include::tables/print.adoc[]
+include::tables/extend.adoc[]
+|===
+
+

ref/ascii/ascii_urls

@@ -0,0 +1,4 @@
+http://plato.asu.edu/MAT420/beginning_perl/3145_AppF.pdf
+http://www.profdavis.net/ASCII_table.pdf
+http://www.aboutmyip.com/AboutMyXApp/AsciiChart.jsp
+http://www.robelle.com/smugbook/ascii.html