r00t2/WireProto
SHA256
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Initial draft.

Browse Source
This commit is contained in:
brent saner
2024-07-09 18:30:10 -04:00
parent c329fc916e
commit 9a90a55aa8
7 changed files with 204 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
README.html
View File

@@ -632,7 +632,7 @@ pre.rouge .gs {
<div class="details">
<span id="author" class="author">Brent Saner</span><br>
<span id="email" class="email"><a href="mailto:bts@square-r00t.net">bts@square-r00t.net</a></span><br>
<span id="revdate">Last rendered 2024-07-07 23:58:11 -0400</span>
<span id="revdate">Last rendered 2024-07-09 18:32:40 -0400</span>
</div>
<div id="toc" class="toc2">
<div id="toctitle">Table of Contents</div>
@@ -640,8 +640,9 @@ pre.rouge .gs {
<li><a href="#license">1. License</a></li>
<li><a href="#proto">2. Protocol</a>
<ul class="sectlevel2">
<li><a href="#lib">2.1. Library</a></li>
<li><a href="#ytho">2.2. Why a Custom Message Format?</a></li>
<li><a href="#proto_reqresp">2.1. Requests/Responses</a></li>
<li><a href="#lib">2.2. Reference Library</a></li>
<li><a href="#ytho">2.3. Why Yet Another Message Format?</a></li>
</ul>
</li>
<li><a href="#msg">3. Message Format</a>
@@ -658,11 +659,7 @@ pre.rouge .gs {
<li><a href="#msg_grp_rec_kv_val">3.3.1.1.2. Field Value</a></li>
</ul>
</li>
<li><a href="#msg_grp_recresp">3.3.1.2. Copy Record (Response Copy of Request)</a>
<ul class="sectlevel5">
<li><a href="#msg_grp_rec_kvcpy">3.3.1.2.1. Field/Value Pair (Key/Value Pair) (Response Copy)</a></li>
</ul>
</li>
<li><a href="#msg_grp_recresp">3.3.1.2. Copy of Original Record</a></li>
</ul>
</li>
</ul>
@@ -672,7 +669,7 @@ pre.rouge .gs {
<li><a href="#cksum">4. Checksums</a></li>
<li><a href="#hdrs">5. Headers</a>
<ul class="sectlevel2">
<li><a href="#hdrs_respstart">5.1. <code>RESPSTART</code> Byte Sequence</a></li>
<li><a href="#hdrs_respstart">5.1. <code>RESPSTART</code> Indicator</a></li>
<li><a href="#hdrs_cksum">5.2. <code>CKSUM</code> Header Prefix</a></li>
<li><a href="#hdrs_msgstart">5.3. <code>MSGSTART</code> Header Prefix</a></li>
<li><a href="#hdrs_bodystart">5.4. <code>BODYSTART</code> Header Prefix</a></li>
@@ -1238,7 +1235,7 @@ In the event of the embedded text in this document differing from the online ver
<h2 id="proto"><a class="link" href="#proto">2. Protocol</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The WireProto data packing API is a custom wire protocol//message format designed for incredibly performant, unambiguous, predictable, platform-agnostic, client-agnostic communication. It is based heavily on the <a href="https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.key" target="_blank" rel="noopener">OpenSSH "v1" key format</a> <a href="https://git.r00t2.io/r00t2/go_sshkeys/src/branch/master/_ref/KEY_GUIDE.html#v1_plain_2">(example/details)</a> packing method.</p>
<p>The WireProto data packing API is a custom wire protocol//message format designed for incredibly performant, unambiguous, predictable, platform-agnostic, implementation-agnostic communication. It is based heavily on the <a href="https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.key" target="_blank" rel="noopener">OpenSSH "v1" key format</a> <a href="https://git.r00t2.io/r00t2/go_sshkeys/src/branch/master/_ref/KEY_GUIDE.html#v1_plain_2">(example/details)</a> packing method.</p>
</div>
<div class="paragraph">
<p>It supports arbitrary binary values, which means they can be anything according to the implementation-specific details; a common practice is to encode ("marshal") a Go struct to JSON bytes, and set that as a WireProto field&#8217;s value.</p>
@@ -1247,25 +1244,45 @@ In the event of the embedded text in this document differing from the online ver
<p>It supports both static construction/parsing/dissection and stream approaches in a single format, as well as multiple commands per request message/multiple answers per response message.</p>
</div>
<div class="paragraph">
<p><strong>All</strong> packed uint32 values are big-endian.</p>
<p><strong>All</strong> packed uint32 (<em>unsigned 32-bit integer</em>) values are a <a href="https://en.wikipedia.org/wiki/Endianness" target="_blank" rel="noopener">big-endian</a> 4-byte sequence (e.g. <code>3712599402</code> == <code>0xdd49c56a</code>, or [<code>0xdd</code>, <code>0x49</code>, <code>0xc5</code>, <code>0x6a</code>]).</p>
</div>
<div class="paragraph">
<p>This specification <a href="#proto_ver">Protocol Version</a> is <code>1</code> (<code>0x00000001</code>).</p>
<p>This specification&#8217;s <a href="#proto_ver">Protocol Version</a> is <code>1</code> (<code>0x00000001</code>).</p>
</div>
<div class="paragraph">
<p>For other releases/finalized versions of this specification, see <a href="https://git.r00t2.io/r00t2/WireProto/tags" target="_blank" rel="noopener">here</a>.</p>
</div>
<div class="paragraph">
<p>For in-development versions, drafts, etc. of this specification, see <a href="https://git.r00t2.io/r00t2/WireProto/branches" target="_blank" rel="noopener">here</a>.</p>
</div>
<div class="sect2">
<h3 id="lib"><a class="link" href="#lib">2.1. Library</a></h3>
<h3 id="proto_reqresp"><a class="link" href="#proto_reqresp">2.1. Requests/Responses</a></h3>
<div class="paragraph">
<p>This protocol specification is accompanied with a reference library for Golang, <a href="https://git.r00t2.io/r00t2/go_wireproto" target="_blank" rel="noopener">"WireProto"</a> (<a href="https://git.r00t2.io/r00t2/wireproto" target="_blank" rel="noopener"><em>source</em></a>):</p>
<p>WireProto indicates two types of Messages/communication ends: a <em>Requester</em> (<em>Requesting End</em>) and a <em>Responder</em> (<em>Responding End</em>).</p>
</div>
<div class="paragraph">
<p>This terminology is intentionally implementation-agnostic. A <em>Requester</em> is any end of a communication that is <strong>requesting data</strong>, and the <em>Responder</em> is any end of a communication that is <strong>providing that data</strong>. A Responder may not always be present (e.g. in the case of using WireProto for local disk serialization/caching, etc.), and a "client" may be a Requester, Responder, or both&#8201;&#8212;&#8201;likewise for a "server".</p>
</div>
</div>
<div class="sect2">
<h3 id="lib"><a class="link" href="#lib">2.2. Reference Library</a></h3>
<div class="paragraph">
<p>The WireProto specification is accompanied by a reference library for Golang, <a href="https://git.r00t2.io/r00t2/go_wireproto" target="_blank" rel="noopener">"WireProto"</a> (<a href="https://git.r00t2.io/r00t2/wireproto" target="_blank" rel="noopener"><em>source</em></a>):</p>
</div>
<a href="https://pkg.go.dev/go.pkg.dev/r00t2.io/wireproto">
<img src="https://pkg.go.dev/badge/go.pkg.dev/r00t2.io/wireproto.svg"
alt="Go Reference">
</a>
<br />
<br />
<div class="paragraph">
<p>Additional reference libraries may be available in the future.</p>
</div>
</div>
<div class="sect2">
<h3 id="ytho"><a class="link" href="#ytho">2.2. Why a Custom Message Format?</a></h3>
<h3 id="ytho"><a class="link" href="#ytho">2.3. Why Yet Another Message Format?</a></h3>
<div class="paragraph">
<p>Because existing ones (e.g. JSON, XML, YAML) are slow/bloaty, inaccurate, and/or inflexible. They struggle with binary or abritrary data (or in e.g. XML&#8217;s case requiring intermediate conditional encoding/decoding).</p>
<p>Because existing methods of serializing data in a structured way (e.g. JSON, XML, YAML) are slow/bloaty, inaccurate, and/or inflexible. They struggle with binary or abritrary data (or in e.g. XML&#8217;s case requiring intermediate conditional encoding/decoding).</p>
</div>
<div class="paragraph">
<p>If it can be represented as bytes (which all digital data can), WireProto can send and receive it.</p>
@@ -1276,16 +1293,16 @@ In the event of the embedded text in this document differing from the online ver
<div class="ulist">
<ul>
<li>
<p><a href="https://protobuf.dev/" target="_blank" rel="noopener"><strong>Protobuf</strong></a> has performance issues (yes, really; protobufs have large overhead) and is restrictive on data types for future-proofing.</p>
<p><a href="https://protobuf.dev/" target="_blank" rel="noopener"><strong>Protobuf</strong></a> has performance issues (yes, really; protobufs have large overhead compared to WireProto) and is restrictive on data types for future-proofing.</p>
</li>
<li>
<p><a href="https://go.dev/blog/gob" target="_blank" rel="noopener"><strong>Gob</strong></a> is very language-limiting and does not support e.g. nil pointers and cyclical values.</p>
</li>
<li>
<p><a href="https://capnproto.org/" target="_blank" rel="noopener">Cap&#8217;n Proto</a> has wide language support and excellent performance but is terribly non-idiomatic, requiring the code to be generated from the schema and not vice versa (which is only ideal if you have only one communication interface).</p>
<p><a href="https://capnproto.org/" target="_blank" rel="noopener">Cap&#8217;n Proto</a> has wide language support and excellent performance but is terribly non-idiomatic, requiring the code to be generated from the schema and not vice versa (which is only ideal if you have only one communication interface and is, in the author&#8217;s opinion, the entirely incorrect approach).</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/JSON_streaming" target="_blank" rel="noopener">JSON streams</a> have no delimiters defined, and thus this makes it an inconvenience if using a parser that does not know when the message ends/is complete, or if it is expecting a standalone JSON object.</p>
<p><a href="https://en.wikipedia.org/wiki/JSON_streaming" target="_blank" rel="noopener">JSON streams</a> have no delimiters defined which makes it an inconvenience if using a parser that does not know when the message ends/is complete, or if it is expecting a standalone JSON object (e.g. native vanilla Golang JSON parsing).</p>
</li>
</ul>
</div>
@@ -1300,7 +1317,10 @@ In the event of the embedded text in this document differing from the online ver
<p>WireProto is only used for binary packing/unpacking; this means it can be used with any e.g. <a href="https://pkg.go.dev/net#Conn" target="_blank" rel="noopener"><code>net.Conn</code></a> (and even has helper functions explicitly to facilitate this), storage on-disk, etc.</p>
</div>
<div class="paragraph">
<p>Thus it is transport/storage-agnostic, and can be used with a <a href="https://pkg.go.dev/net#Dial" target="_blank" rel="noopener">TCP socket, UDP socket, IPC (InterProcess Communication)/UDS (UNIX Domain Socket) handle,</a> <a href="https://pkg.go.dev/crypto/tls#Dial" target="_blank" rel="noopener">TLS-tunneled TCP socket</a>, etc.</p>
<p>As such it is transport/storage-agnostic, and can be used with a <a href="https://pkg.go.dev/net#Dial" target="_blank" rel="noopener">TCP socket, UDP socket, IPC (InterProcess Communication)/UDS (UNIX Domain Socket) handle,</a> <a href="https://pkg.go.dev/crypto/tls#Dial" target="_blank" rel="noopener">TLS-tunneled TCP socket</a>, etc.</p>
</div>
<div class="paragraph">
<p>See the <a href="#lib">Reference Library</a> for details.</p>
</div>
</td>
</tr>
@@ -1326,14 +1346,14 @@ In the event of the embedded text in this document differing from the online ver
<p>These refer to <em>ASCII control characters</em>. You will also see many values represented in hex.</p>
</div>
<div class="paragraph">
<p>You can find more details about this (along with a full ASCII reference) <a href="https://square-r00t.net/ascii.html" target="_blank" rel="noopener">here</a>. Note that the socket API fully supports UTF-8&#8201;&#8212;&#8201;just be sure that your <a href="#alloc_size">Size Allocator</a> are aligned to the byte count, not character count.</p>
<p>You can find more details about this (along with a full ASCII reference) <a href="https://square-r00t.net/ascii.html" target="_blank" rel="noopener">here</a>. Note that the specification fully supports UTF-8 (or any other arbitrary encoding)&#8201;&#8212;&#8201;just be sure that your <a href="#alloc_size">size allocators</a> are aligned to the <strong>byte count</strong> and not <strong>character count</strong> (as these may not be equal depending on encoding).</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Each <strong>message</strong> is generally composed of:</p>
<p>Each <strong>message</strong> is composed of:</p>
</div>
<div class="ulist">
<ul>
@@ -1341,7 +1361,7 @@ In the event of the embedded text in this document differing from the online ver
<p>The <a href="#msg_respstatus">Response Status</a><sup class="footnote" id="_footnote_responly">[<a id="_footnoteref_1" class="footnote" href="#_footnotedef_1" title="View footnote.">1</a>]</sup></p>
</li>
<li>
<p>A <a href="#cksum">Checksum</a><sup class="footnote" id="_footnote_optclient">[<a id="_footnoteref_2" class="footnote" href="#_footnotedef_2" title="View footnote.">2</a>]</sup><sup class="footnote" id="_footnote_reqsrv">[<a id="_footnoteref_3" class="footnote" href="#_footnotedef_3" title="View footnote.">3</a>]</sup></p>
<p>A <a href="#cksum">Checksum</a><sup class="footnote" id="_footnote_optreq">[<a id="_footnoteref_2" class="footnote" href="#_footnotedef_2" title="View footnote.">2</a>]</sup><sup class="footnote" id="_footnote_reqresp">[<a id="_footnoteref_3" class="footnote" href="#_footnotedef_3" title="View footnote.">3</a>]</sup></p>
</li>
<li>
<p>A <a href="#hdrs_msgstart"><code>MSGSTART</code> Header Prefix</a></p>
@@ -1350,15 +1370,15 @@ In the event of the embedded text in this document differing from the online ver
<p>A <a href="#proto_ver">Protocol Version</a></p>
</li>
<li>
<p>A <a href="#hdrs_bodystart"><code>BODYSTART</code> Header Prefix</a></p>
</li>
<li>
<p>A <a href="#msg_grp">Record Group</a> <a href="#alloc_cnt">Count Allocator</a></p>
</li>
<li>
<p>A <a href="#msg_grp">Record Group</a> <a href="#alloc_size">Size Allocator</a></p>
</li>
<li>
<p>A <a href="#hdrs_bodystart"><code>BODYSTART</code> Header Prefix</a></p>
</li>
<li>
<p>One (or more) <a href="#msg_grp">Record Group</a>(s), each of which contain:</p>
<div class="ulist">
<ul>
@@ -1376,12 +1396,12 @@ In the event of the embedded text in this document differing from the online ver
<li>
<p>A <a href="#msg_grp_rec_kv_val">Field Value</a></p>
</li>
<li>
<p>A <a href="#msg_grp_recresp">Copy Record</a><sup class="footnoteref">[<a class="footnote" href="#_footnotedef_1" title="View footnote.">1</a>]</sup></p>
</li>
</ul>
</div>
</li>
<li>
<p>A <a href="#msg_grp_recresp">copy of the original record</a><sup class="footnoteref">[<a class="footnote" href="#_footnotedef_1" title="View footnote.">1</a>]</sup></p>
</li>
</ul>
</div>
</li>
@@ -1399,8 +1419,8 @@ In the event of the embedded text in this document differing from the online ver
<div class="sect2">
<h3 id="msg_respstatus"><a class="link" href="#msg_respstatus">3.1. Response Status</a></h3>
<div class="paragraph">
<p>For responses, their messages have an additional byte prepended; a status indicator.
This allows client programs to quickly bail in the case of an error if no further parsing is desired.</p>
<p>For response messages, a speciall "summary byte" is prepended; a status indicator.
This allows requesting ends to quickly bail in the case of an error if no further parsing is desired.</p>
</div>
<div class="paragraph">
<p>The status will be indicated by one of <a href="#hdrs_respstart">two values</a>: an ASCII <code>ACK</code> (<code>0x06</code>) for all requests being returned successfully or an ASCII <code>NAK</code> (<code>0x15</code>) if one or more errors were encountered across all records.</p>
@@ -1424,7 +1444,7 @@ This allows client programs to quickly bail in the case of an error if no furthe
<div class="title">Note</div>
</td>
<td class="content">
Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>master</code> branch of this specification and should be considered experimental.
Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>master</code> branch of this specification and should be considered experimental, not conforming to any specific protocol message format version.
</td>
</tr>
</table>
@@ -1454,7 +1474,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect3">
<h4 id="msg_grp_rec"><a class="link" href="#msg_grp_rec">3.3.1. Record</a></h4>
<div class="paragraph">
<p>A record contains multiple related <a href="#msg_grp_rec_kv">Field/Value Pairs (FVP)</a>. It is typical to only have a single Record.</p>
<p>A record contains multiple related <a href="#msg_grp_rec_kv">Field/Value Pairs (FVP)</a> and, if a Response Record, a copy of the original reference Request Record it is responding to.</p>
</div>
<div class="paragraph">
<p>Its structure is:</p>
@@ -1466,25 +1486,18 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
</li>
<li>
<p><a href="#msg_grp_rec_kv">Field/Value Pair (Key/Value Pair)</a> <a href="#alloc_size">Size Allocator</a></p>
</li>
<div class="olist loweralpha">
<ol class="loweralpha" type="a">
<li>
<p>One (or more) <a href="#msg_grp_rec_kv">Field/Value Pairs</a></p>
</li>
</ol>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
<div class="paragraph">
<p>For response messages, the record&#8217;s size allocator (but NOT the count allocator) includes the <a href="#msg_grp_recresp">Copy Record</a> size for each response record copy!<sup class="footnoteref">[<a class="footnote" href="#_footnotedef_1" title="View footnote.">1</a>]</sup></p>
</div>
</td>
</tr>
</table>
</li>
<li>
<p><a href="#msg_grp_recresp">Copy of Original Record</a> <a href="#alloc_size">Size Allocator</a><sup class="footnoteref">[<a class="footnote" href="#_footnotedef_1" title="View footnote.">1</a>]</sup></p>
</li>
</ol>
</div>
<div class="sect4">
<h5 id="msg_grp_rec_kv"><a class="link" href="#msg_grp_rec_kv">3.3.1.1. Field/Value Pair (Key/Value Pair)</a></h5>
@@ -1518,7 +1531,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
</td>
<td class="content">
<div class="paragraph">
<p>Unlike most/all other <a href="#alloc">Allocators</a> for other sections/levels, the field name and value allocators are consecutive <a href="#alloc_size">Size Allocators</a>! This is because there is only one field name and value per record.</p>
<p>Unlike most/all other <a href="#alloc">Allocators</a> for other sections/levels, the field name and value allocators are consecutive <a href="#alloc_size">Size Allocators</a>! This is because there is <strong>only one</strong> field name and value per <a href="#msg_grp_rec">Record</a>.</p>
</div>
</td>
</tr>
@@ -1527,7 +1540,10 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect5">
<h6 id="msg_grp_rec_kv_nm"><a class="link" href="#msg_grp_rec_kv_nm">3.3.1.1.1. Field Name</a></h6>
<div class="paragraph">
<p>The field name is usually from a finite set of allowed names. The <a href="#msg_grp_rec_kv_val">Field Value</a>, while written as bytes, often contains a data structure defined by the field name. (A field name is closer to a "value type".) It <strong>must</strong> be a UTF-8 string.</p>
<p>The field name is usually from a finite set of allowed names. The <a href="#msg_grp_rec_kv_val">Field Value</a>, while written as bytes, often contains data defined by the field name. (That is, the parsing of <a href="#msg_grp_rec_kv_val">Field Value</a> often depends on its Field Name.) It is recommended that the field name be a UTF-8-compatible string for simplified serializing and <a href="https://www.wireshark.org/" target="_blank" rel="noopener">on-the-wire debugging</a>.</p>
</div>
<div class="paragraph">
<p>While there is no technical requirement that a field name be unique per-<a href="#msg_grp_rec">Record</a>, it is generally recommended (unless emulating/encoding arrays of data in separate <a href="#msg_grp_rec_kv">field/value pairs</a>).</p>
</div>
<div class="paragraph">
<p>Its structure is:</p>
@@ -1535,7 +1551,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="olist arabic">
<ol class="arabic">
<li>
<p>The name in bytes</p>
<p>A name/identifier in bytes</p>
</li>
</ol>
</div>
@@ -1551,72 +1567,38 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="olist arabic">
<ol class="arabic">
<li>
<p>The value in bytes</p>
<p>A value in bytes</p>
</li>
</ol>
</div>
</div>
</div>
<div class="sect4">
<h5 id="msg_grp_recresp"><a class="link" href="#msg_grp_recresp">3.3.1.2. Copy Record (Response Copy of Request)</a></h5>
<h5 id="msg_grp_recresp"><a class="link" href="#msg_grp_recresp">3.3.1.2. Copy of Original Record</a></h5>
<div class="paragraph">
<p>This contains a "copy" of the original/request&#8217;s <a href="#msg_grp_rec">Record</a> that this record is in response to.</p>
<p>This contains a "copy" of the original/request&#8217;s <a href="#msg_grp_rec">Record</a> that this record is in response to. It is only present in Response message and must not be included in Request messages.</p>
</div>
<div class="paragraph">
<p>It is a variant of a <a href="#msg_grp_rec">Record</a> used exclusively in responses, and is tied to (included in) each response&#8217;s <a href="#msg_grp_rec_kv">FVP</a>.</p>
<p>It is a complete <a href="#msg_grp_rec">Record</a> from the request embedded inside the responding Record.</p>
</div>
<div class="paragraph">
<p>Its structure is:</p>
<p>For example, if a record contains multiple <a href="#msg_grp_rec_kv">field/value pairs</a> specifying a query of some data then the response record will contain a copy of that record&#8217;s query data.</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p><a href="#msg_grp_rec_kvcpy">Field/Value Pair (Key/Value Pair) (Response Copy)</a> <a href="#alloc_cnt">Count Allocator</a></p>
</li>
<li>
<p><a href="#msg_grp_rec_kvcpy">Field/Value Pair (Key/Value Pair) (Response Copy)</a> <a href="#alloc_size">Size Allocator</a></p>
</li>
<li>
<p>One (or more) <a href="#msg_grp_rec_kvcpy">Field/Value Pairs (Response Copy)</a></p>
</li>
</ol>
</div>
<div class="sect5">
<h6 id="msg_grp_rec_kvcpy"><a class="link" href="#msg_grp_rec_kvcpy">3.3.1.2.1. Field/Value Pair (Key/Value Pair) (Response Copy)</a></h6>
<div class="paragraph">
<p>A field/value pair (also referred to as a key/value pair) contains a matched <a href="#msg_grp_rec_kv_nm">Field Name</a> and its <a href="#msg_grp_rec_kv_val">Field Value</a>.</p>
</div>
<div class="paragraph">
<p>It is a variant of a <a href="#msg_grp_rec_kv">Field/Value Pair</a> used exclusively in response copies of the original request&#8217;s FVP.</p>
</div>
<div class="paragraph">
<p>Its structure is:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p><a href="#msg_grp_rec_kv_nm">Field Name</a> <a href="#alloc_size">Size Allocator</a></p>
</li>
<li>
<p><a href="#msg_grp_rec_kv_val">Field Value</a> <a href="#alloc_size">Size Allocator</a></p>
</li>
<li>
<p>A single <a href="#msg_grp_rec_kv_nm">Field Name</a></p>
</li>
<li>
<p>A single matching <a href="#msg_grp_rec_kv_val">Field Value</a></p>
</li>
</ol>
</div>
<div class="admonitionblock important">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
<div class="title">Note</div>
</td>
<td class="content">
<div class="paragraph">
<p>Unlike most/all other <a href="#alloc">Allocators</a> for other sections/levels, the field name and value allocators are consecutive <a href="#alloc_size">Size Allocators</a>! This is because there is only one field name and value per record.</p>
<p>While <strong>not recommended</strong>, it <strong>is</strong> within specification/permissible to "alias" a request record via a session-unique identifier (e.g. <a href="https://datatracker.ietf.org/doc/html/rfc4122" target="_blank" rel="noopener">UUIDv4</a>), <strong>provided</strong> the promise that the requesting end retains an identifiable copy of/can lookup or associate its original record based on that identifying alias.</p>
</div>
<div class="paragraph">
<p>For example, a requesting end may specify <em>its own</em> provided identifier as an <a href="#msg_grp_rec_kv">field/value pair</a> (e.g. <code>identifier:f18231973d08417e877dd1a2f8e8ab74</code>) along with additional data. The returning Response Record may then include <strong>only</strong> an original/request record with an FVP of <code>identifier:f18231973d08417e877dd1a2f8e8ab74</code> along with the requested data.</p>
</div>
<div class="paragraph">
<p>Alternatively for another example, a responding end may return a Response Record with an original/request record of a single FVP such as <code>ref_id:46823da27f8749df9dee8f0bded8cce9</code> or the like. The requesting end <strong>must</strong> then be able to retrieve the full copy of the original request record as a standalone Response Record based on that <code>ref_id</code>. Responding ends <strong>may</strong> enforce lifetimes for request record lookup in this case but they must be promised.</p>
</div>
</td>
</tr>
@@ -1627,18 +1609,17 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="cksum"><a class="link" href="#cksum">4. Checksums</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Checksums are optional for the client but the server will <strong>always</strong> send them. <strong>If present</strong> in the request, the server will validate to ensure the checksum matches the message body (<a href="#hdrs_bodystart">body start</a> to <a href="#hdrs_bodyend">body end</a>, headers included). If the checksum does not match, an error will be returned.</p>
<p>Checksums are optional for the requesting end but the responding end <strong>must</strong> send them. <strong>If present</strong> in the request, the responder <strong>must</strong> validate to ensure the checksum matches the message body (<a href="#hdrs_bodystart"><code>BODYSTART</code> Header Prefix</a> to <a href="#hdrs_bodyend"><code>BODYEND</code> Sequence</a>, inclusive). If the checksum does not match, an error <strong>must</strong> be returned.</p>
</div>
<div class="paragraph">
<p>They are represented as a big-endian-packed uint32.</p>
</div>
<div class="paragraph">
<p>The checksum must be prefixed with a <a href="#hdrs_cksum"><code>CKSUM</code> Header Prefix</a>. If no checksum is provided, this prefix must <strong>not</strong> be included in the sequence.</p>
<p>The checksum must be prefixed with a <a href="#hdrs_cksum"><code>CKSUM</code> Header Prefix</a>. If no checksum is provided in a request, this prefix <strong>must not</strong> be included in the sequence.</p>
</div>
<div class="admonitionblock tip">
<table>
@@ -1648,17 +1629,20 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
</td>
<td class="content">
<div class="paragraph">
<p>You can quickly check if a checksum is present by checking the first byte in requests or the second byte in responses. If it is <code>ESC</code> (<code>0x1b</code>), a checksum is provided. If it is <code>SOH</code> (<code>0x01</code>), one was <strong>not</strong> provided.</p>
<p>A responder can quickly check if a checksum is present by checking the first byte in requests. If it is <a href="#hdrs_cksum"><code>CKSUM</code></a>, a checksum is provided. If it is <a href="#hdrs_msgstart"><code>MSGSTART</code></a>, one was <strong>not</strong> provided.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The checksum method used is the <a href="https://users.ece.cmu.edu/~koopman/crc/crc32.html" target="_blank" rel="noopener">IEEE 802.3 CRC-32</a>, which should be natively available for all/most client implementations as it is perhaps the most ubiquitous of CRC-32 variants. (Polynomial <code>0x04c11db7</code>, reversed polynomial <code>0xedb88320</code>.)</p>
<p>The checksum method used is the <a href="https://users.ece.cmu.edu/~koopman/crc/crc32.html" target="_blank" rel="noopener">IEEE 802.3 CRC-32</a>, which should be natively available for all/most implementations/languages as it is perhaps the most ubiquitous of CRC-32 variants (e.g. <a href="https://docs.python.org/3/library/zlib.html#zlib.crc32" target="_blank" rel="noopener">Python</a>, <a href="https://pkg.go.dev/hash/crc32" target="_blank" rel="noopener">Golang</a>, <a href="https://github.com/gcc-mirror/gcc/blob/master/libiberty/crc32.c" target="_blank" rel="noopener">GNU C/glibc</a>(?), <a href="https://crates.io/keywords/crc32" target="_blank" rel="noopener">Rust</a>, etc.). (Polynomial <code>0x04c11db7</code>, reversed polynomial <code>0xedb88320</code>.)</p>
</div>
<div class="paragraph">
<p>To confirm you are using the correct CRC32 implementation (as there are a <strong>ton</strong> of "CRC-32" algorithms and methods out there), use the following validations:</p>
<p>If one needs to implement the appropriate CRC32 implementation, there is extensive detail at the <a href="https://en.wikipedia.org/wiki/Cyclic_redundancy_check" target="_blank" rel="noopener">CRC Wikipedia article</a>.</p>
</div>
<div class="paragraph">
<p>To confirm the correct CRC32 implementation is being used (as there are <strong>many</strong> "CRC-32" algorithms/methods/functions/libraries), the following validations may be used:</p>
</div>
<table class="tableblock frame-all grid-all stretch">
<caption class="title">Table 1. CRC-32 Validations</caption>
@@ -1680,6 +1664,13 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
</thead>
<tbody>
<tr>
<td class="tableblock halign-center valign-middle"><p class="tableblock"><code>WireProto</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>0x5769726550726f746f</code></p></td>
<td class="tableblock halign-center valign-middle"><p class="tableblock"><code>815806352</code></p></td>
<td class="tableblock halign-center valign-middle"><p class="tableblock"><code>0x30a03790</code></p></td>
<td class="tableblock halign-center valign-middle"><p class="tableblock"><code>0x9037a030</code></p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-middle"><p class="tableblock"><code>FooBarBazQuux</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>0x466f6f42617242617a51757578</code></p></td>
<td class="tableblock halign-center valign-middle"><p class="tableblock"><code>983022564</code></p></td>
@@ -1704,7 +1695,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<p>Certain sections are wrapped with an identifying header. Those headers are included below for reference.</p>
</div>
<div class="sect2">
<h3 id="hdrs_respstart"><a class="link" href="#hdrs_respstart">5.1. <code>RESPSTART</code> Byte Sequence</a></h3>
<h3 id="hdrs_respstart"><a class="link" href="#hdrs_respstart">5.1. <code>RESPSTART</code> Indicator</a></h3>
<div class="paragraph">
<p>Responses have a <a href="#msg_respstatus">Response Status</a>.<sup class="footnoteref">[<a class="footnote" href="#_footnotedef_1" title="View footnote.">1</a>]</sup></p>
</div>
@@ -1715,13 +1706,13 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect2">
<h3 id="hdrs_cksum"><a class="link" href="#hdrs_cksum">5.2. <code>CKSUM</code> Header Prefix</a></h3>
<div class="paragraph">
<p>A checksum, if provided, will have a prefix header of <code>ESC</code> (<code>0x1b</code>).</p>
<p>A <a href="#cksum">checksum</a>, if provided<sup class="footnoteref">[<a class="footnote" href="#_footnotedef_2" title="View footnote.">2</a>]</sup><sup class="footnoteref">[<a class="footnote" href="#_footnotedef_3" title="View footnote.">3</a>]</sup>, will have a prefix header of <code>ESC</code> (<code>0x1b</code>).</p>
</div>
</div>
<div class="sect2">
<h3 id="hdrs_msgstart"><a class="link" href="#hdrs_msgstart">5.3. <code>MSGSTART</code> Header Prefix</a></h3>
<div class="paragraph">
<p>The message start header indicates a start of a message.</p>
<p>The message start header indicates a start of a "message". It is used to delineate operational headers from specification information (e.g. <a href="#proto_ver">Protocol Version</a>) and data.</p>
</div>
<div class="paragraph">
<p>It is an <code>SOH</code> (<code>0x01</code>).</p>
@@ -1730,7 +1721,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect2">
<h3 id="hdrs_bodystart"><a class="link" href="#hdrs_bodystart">5.4. <code>BODYSTART</code> Header Prefix</a></h3>
<div class="paragraph">
<p>The body start header indicates that actual data/records follows.</p>
<p>The body start header indicates that data/records follow. All bytes between <code>BODYSTART</code> and <a href="#hdrs_bodyend"><code>BODYEND</code></a> are to be assumed to be directly pertinent to the request/response rather than operational.</p>
</div>
<div class="paragraph">
<p>It is an <code>STX</code> (<code>0x02</code>).</p>
@@ -1739,7 +1730,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect2">
<h3 id="hdrs_bodyend"><a class="link" href="#hdrs_bodyend">5.5. <code>BODYEND</code> Sequence</a></h3>
<div class="paragraph">
<p>The body end prefix indicates the end of data/records.</p>
<p>The body end prefix indicates the end of data/records. All bytes between <a href="#hdrs_bodystart"><code>BODYSTART</code></a> and <code>BODYEND</code> are to be assumed to be directly pertinent to the request/response rather than operational.</p>
</div>
<div class="paragraph">
<p>It is an <code>ETX</code> (<code>0x03</code>).</p>
@@ -1748,7 +1739,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect2">
<h3 id="hdrs_msgend"><a class="link" href="#hdrs_msgend">5.6. <code>MSGEND</code> Sequence</a></h3>
<div class="paragraph">
<p>The message end prefix indicates that a message in its entirety has ended.</p>
<p>The message end prefix indicates that a message in its entirety has ended, and if no further communication is necessary per implementation the connection may be disconnected.</p>
</div>
<div class="paragraph">
<p>It is an <code>EOT</code> (<code>0x04</code>).</p>
@@ -1763,13 +1754,13 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<p>There are two type of allocators included for each following sequence of bytes: <code>count allocators</code> and <code>size allocators</code>.</p>
</div>
<div class="paragraph">
<p>They can be used by clients to determine the size of destination buffers, and are used by the server to efficiently unpack requests.</p>
<p><a href="#alloc_size">Size allocators</a> can be used by receiving ends to efficiently pre-allocate buffers and for sending ends to indicate the amount of remaining data expected.</p>
</div>
<div class="paragraph">
<p>They are usually paired together with the count allocator preceding the size allocator, but not always (e.g. <a href="#msg_grp_rec_kv">Field/Value Pair (Key/Value Pair)</a> have two <a href="#alloc_size">Size Allocator</a>).</p>
<p>They are usually preceded with a <a href="#alloc_cnt">count allocator</a> to allow for pre-allocating e.g. slice/array sizes, but not always (e.g. <a href="#msg_grp_rec_kv">field/value pairs</a> have two <a href="#alloc_size">size allocators</a>).</p>
</div>
<div class="paragraph">
<p>All allocators are unsigned 32-bit integers, little-endian-packed.</p>
<p>All allocators are unsigned 32-bit integers, big-endian-packed.</p>
</div>
<div class="sect2">
<h3 id="alloc_cnt"><a class="link" href="#alloc_cnt">6.1. Count Allocator</a></h3>
@@ -1780,7 +1771,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect2">
<h3 id="alloc_size"><a class="link" href="#alloc_size">6.2. Size Allocator</a></h3>
<div class="paragraph">
<p>Size allocators indicate <strong>how much</strong> (in bytes) all children objects are combined together. It includes e.g. separators, etc.</p>
<p>Size allocators indicate <strong>how much</strong> (in bytes) all children objects are combined as one block. They include the allocators themselves of child objects, etc. as well.</p>
</div>
</div>
</div>
@@ -1789,7 +1780,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<h2 id="ref"><a class="link" href="#ref">7. Reference Model and Examples</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>For a more visual explanation, given the following e.g. Golang structs from the <a href="https://pkg.go.dev/r00t2.io/wireproto" target="_blank" rel="noopener">Golang reference library</a> (<code>wireproto.Request{}</code> and <code>wireproto.Response{}</code>):</p>
<p>For a more visual explanation, given the following e.g. Golang structs from the <a href="#lib">Reference Library</a> (<code>wireproto.Request{}</code> and <code>wireproto.Response{}</code>):</p>
</div>
<div class="sect2">
<h3 id="ref_single"><a class="link" href="#ref_single">7.1. Single/Simple</a></h3>
@@ -1890,7 +1881,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<span class="c">// RESPONSE (Simple)</span>
<span class="n">testSimpleResp</span> <span class="o">*</span><span class="n">Response</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">Response</span><span class="p">{</span>
<span class="n">Status</span><span class="o">:</span> <span class="n">AsciiACK</span><span class="p">,</span>
<span class="n">Checksum</span><span class="o">:</span> <span class="m">4005376897</span><span class="p">,</span> <span class="c">// 0xeebd3381</span>
<span class="n">Checksum</span><span class="o">:</span> <span class="m">3472688928</span><span class="p">,</span> <span class="c">// 0xcefd0720</span>
<span class="n">ProtocolVersion</span><span class="o">:</span> <span class="n">ProtoVersion</span><span class="p">,</span>
<span class="n">RecordGroups</span><span class="o">:</span> <span class="p">[]</span><span class="o">*</span><span class="n">ResponseRecordGroup</span><span class="p">{</span>
<span class="o">&amp;</span><span class="n">ResponseRecordGroup</span><span class="p">{</span>
@@ -1924,7 +1915,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<pre class="rouge highlight"><code data-lang="text">// RESPONSE (Simple)
06 // HDR:RESPSTART (Status: OK)
1b // HDR:CKSUM
5fde82e5 // Checksum Value (1608418021)
cefd0720 // Checksum Value (3472688928)
01 // HDR:MSGSTART
00000001 // Protocol Version (1)
02 // HDR:BODYSTART
@@ -1967,7 +1958,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="text">061b5fde82e501000000010200000001000000610000000100000059000000010000001d00000030
<pre class="rouge highlight"><code data-lang="text">061bcefd072001000000010200000001000000610000000100000059000000010000001d00000030
000000050000001064617461313c61726269747261727920646174613e0000000200000028000000
06000000066669656c643176616c75653100000006000000066669656c643276616c7565320304</code></pre>
</div>
@@ -1977,7 +1968,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<div class="sect2">
<h3 id="ref_multi"><a class="link" href="#ref_multi">7.2. Multiple/Many/Complex</a></h3>
<div class="paragraph">
<p>Multiple commands, parameters, etc. can be specified in one message.</p>
<p>Multiple records, record groups, etc. can be specified in one message.</p>
</div>
<div class="sect3">
<h4 id="ref_multi_req"><a class="link" href="#ref_multi_req">7.2.1. Complex Request</a></h4>
@@ -2163,7 +2154,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<span class="c">// RESPONSE (Complex)</span>
<span class="n">testMultiResp</span> <span class="o">*</span><span class="n">Response</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">Response</span><span class="p">{</span>
<span class="n">Status</span><span class="o">:</span> <span class="n">AsciiACK</span><span class="p">,</span>
<span class="n">Checksum</span><span class="o">:</span> <span class="m">2563794802</span><span class="p">,</span> <span class="c">// 98d06772</span>
<span class="n">Checksum</span><span class="o">:</span> <span class="m">2928197330</span><span class="p">,</span> <span class="c">// 0xae88bed2</span>
<span class="n">ProtocolVersion</span><span class="o">:</span> <span class="n">ProtoVersion</span><span class="p">,</span>
<span class="n">RecordGroups</span><span class="o">:</span> <span class="p">[]</span><span class="o">*</span><span class="n">ResponseRecordGroup</span><span class="p">{</span>
<span class="o">&amp;</span><span class="n">ResponseRecordGroup</span><span class="p">{</span>
@@ -2228,7 +2219,7 @@ Version <code>0</code> is reserved for current <code>HEAD</code> of the <code>ma
<pre class="rouge highlight"><code data-lang="text">// RESPONSE (Complex)
06 // HDR:RESPSTART (Status: OK)
1b // HDR:CKSUM
d0ba719f // Checksum Value (3501879711)
ae88bed2 // Checksum Value (2928197330)
01 // HDR:MSGSTART
00000001 // Protocol Version (1)
02 // HDR:BODYSTART
@@ -2343,7 +2334,7 @@ d0ba719f // Checksum Value (3501879711)
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="text">061bd0ba719f010000000102000000020000019800000002000000c4000000010000001e00000038
<pre class="rouge highlight"><code data-lang="text">061bae88bed2010000000102000000020000019800000002000000c4000000010000001e00000038
00000006000000106461746141313c61726269747261727920646174613e00000002000000300000
0008000000086669656c6441314176616c756541314100000008000000086669656c644131427661
6c7565413142000000010000001e0000003800000006000000106461746141323c61726269747261
@@ -2375,7 +2366,7 @@ d0ba719f // Checksum Value (3501879711)
</div>
<div id="footer">
<div id="footer-text">
Last updated 2024-07-07 23:38:45 -0400
Last updated 2024-07-09 18:30:34 -0400
</div>
</div>
</body>