The Shoji Catalog Protocol

This document specifies Shoji, a JSON-based Web content catalog protocol.


1. Introduction

Shoji is a set of JSON-based document formats that describe sets of related
information known as "catalogs". Catalogs are composed of a number of items,
known as "entities", each with an extensible set of attached data known as
"values". For example, a "users" Catalog might contain zero or more "user"
Entities; each "user" Entity, for example, might possess "login_name"
and "address" Values.

The primary use case that Shoji addresses is the exposure of Web content,
such as e-commerce products and transactions, to Web sites as well as directly
to user agents.

1.1 Examples

A simple, single-entity Shoji Catalog Document:

    {"element": "shoji:catalog",
     "self": "http://example.org/users",
     "entities": ["1"]
    }

A simple Shoji Entity Document:

    {"element": "shoji:entity",
     "self": "http://example.org/users/1",
     "body": {
         "last_modified": "2003-12-13 18:30:02Z",
         "first_name": "Katsuhiro",
         "last_name": "Shoji",
         "sold_count": 387
         }
    }

A simple Shoji Value Document at http://example.org/users/1/sold_count:

    387

A more complex Shoji Catalog Document:

    {"element": "shoji:catalog",
     "self": "http://example.org/users",
     "title": "Users Catalog",
     "description": "The set of user entities for this application.",
     "updated": "#2003-12-13T18:30:02Z#",
     "catalogs": {"bills": "bills",
                  "sellers": "sellers",
                  "sellers by sold count": "sellers{?sold_count}"
                  },
     "entities": ["1", "2", "88374", "9843"],
     "views": {"Sold Counts": "sold_counts"},
    }

...and a more complex Shoji Value Document at http://example.org/users/sold_counts:

    [387, 18843, 3478, 999, 1, 18]


1.2 Namespace and Version

For convenience, this protocol may be referred to as "Shoji 1.0".
This specification uses "Shoji" internally.

1.3 Notational Conventions

This specification describes conformance in terms of three artifacts:
Shoji Category Documents, Shoji Entity Documents, and Shoji Value
Documents.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, [RFC2119],
as scoped to those conformance targets.

The grammatical rules in this document are to be interpreted as
described in ABNF [RFC4234]. Any tokens not defined herein are
the same as those in JSON [RFC4627].

1.4 Design Goals

Shoji is designed to:

 * Support delivery of arbitrary JSON payloads.
 * Declare resource relations via hyperlinks.
 * Ease client construction of hyperlinks.
 * Allow linking to, and other metadata descriptions of, other media-types.
 * Support both complete dataset exposure and sparse, restricted exposure.
 * Encourage caching.
 * Have payloads be persisted to filesystems and databases without loss of information.
 * Be independent of any particular device or implementation language.


2. Shoji Documents

This specification describes three kinds of Shoji Documents: Shoji Catalog
Documents, Shoji Entity Documents, and Shoji Value Documents.

A Shoji Catalog Document is a representation of a Shoji catalog, including
metadata about the catalog, and references to some or all of the sub-catalogs,
entities, and views associated with it.

A Shoji Entity Document represents exactly one Shoji entity, including
all attributes contained within it.

A Shoji Value Document represents a single property of a Shoji entity,
or a list of related properties from a set of Shoji entities in a catalog
obtained via one of its views.

All three kinds of Shoji Documents are specified in terms of JavaScript
Object Notation ("JSON", specified in [RFC4627]). Shoji Documents MUST be 
well-formed JSON; however, Shoji Value Documents are not "JSON texts" and
therefore may be an object, array, string, number, or literal.

Shoji allows the use of IRIs [RFC3987]. Every URI [RFC3986] is also an IRI,
so a URI may be used wherever below an IRI is named. There are two special
considerations:

 (1) when an IRI that is not also a URI is given for dereferencing, it MUST
     be mapped to a URI using the steps in Section 3.1 of [RFC3987], and
 (2) when an IRI is serving as a "self" value (see Section 3.3.1), it MUST NOT
     be so mapped, so that the comparison works as described in Section 3.2.1.

Shoji Processors MAY keep state sourced from Shoji Documents and combine them
with other Shoji Documents, in order to facilitate a contiguous view of the
contents of a catalog. The manner in which Shoji Documents are combined in
order to reconstruct a catalog (e.g., updating entities and metadata, dealing
with missing entities) is out of the scope of this specification.


3. Common Shoji Constructs

Many of Shoji's elements share a few common structures. This section defines
those structures and their requirements for convenient reference by the
appropriate element definitions.

When an element is identified as being a particular kind of construct, it
inherits the corresponding requirements from that construct's definition in
this section.

3.1 Identifiers

An Identifier is a text element whose content MUST conform to one of the
productions in the IRI specification [RFC3987], encompassed in double quotes:

    shojiIRI = DQUOTE IRI DQUOTE
    shojiIdentifier = DQUOTE IRI-reference DQUOTE
    
    shojiIdentifierArray =
        begin-array
        [ shojiIdentifier *( value-separator shojiIdentifier ) ]
        end-array

Example Identifiers:

    "http://www.example.com/listings/836759/views"
    "/listings/836759"
    "/"

3.1.1 Comparing Identifiers

Because of the risk of confusion between IRIs that would be equivalent
if they were mapped to URIs and dereferenced, the following normalization
strategy SHOULD be applied when generating shojiIdentifier elements:

    * Provide the scheme, if included, in lowercase characters.
    * Provide the host, if any, in lowercase characters.
    * Only perform percent-encoding where it is essential.
    * Use uppercase A through F characters when percent-encoding.
    * Prevent dot-segments from appearing in paths.
    * For schemes that define a default authority, use an empty authority
      if the default is desired.
    * For schemes that define an empty path to be equivalent to a path
      of "/", use "/".
    * For schemes that define a port, use an empty port if the default is
      desired.
    * Preserve empty fragment identifiers and queries.
    * Ensure that all components of the IRI are appropriately character
      normalized, e.g., by using NFC or NFKC.

Instances of shojiIdentifer elements can be compared to determine whether
an entity or catalog is the same as one seen before. Processors MUST
compare shojiIdentifer elements on a character-by-character basis
(in a case-sensitive fashion). Comparison operations MUST be based
solely on the IRI character strings and MUST NOT rely on dereferencing
the IRIs or URIs mapped from them.

As a result, two IRIs that resolve to the same resource but are not
character-for-character identical will be considered different for the
purposes of identifier comparison.

For example, these are four distinct identifiers, despite the fact that they
differ only in case:

    http://www.example.org/thing
    http://www.example.org/Thing
    http://www.EXAMPLE.org/thing
    HTTP://www.example.org/thing

Likewise, these are three distinct identifiers, because IRI %-escaping is
significant for the purposes of comparison:

    http://www.example.com/~bob
    http://www.example.com/%7ebob
    http://www.example.com/%7Ebob

3.2 IRI Patterns

In addition to IRI and IRI-reference tokens, Shoji processors MUST be prepared
to parse IRI Pattern expansions:

    op          = "/" / ";" / "?"
    varname     = 1*( iunreserved )
    required    = "!"
    vardefault  = *( iunreserved / pct-encoded )
    var         = varname [ required / ( "=" vardefault ) ]
    vars        = var *( "," var )
    expansion   = "{" [ op ] vars "}"

And just to save someone looking up [RFC3987] yet again:

    iunreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar

An IRI Pattern is a sequence of characters that contains any number of embedded
expansions. Each expansion references one or more variables whose values are used
when determining the substition value for an expansion.  An IRI Pattern becomes an
IRI when all the expansions are substituted with their values (see 3.2.3).
The generated IRI will be an IRI-reference, i.e. either an absolute IRI or a
relative reference.

    shojiIRIPattern = DQUOTE IRI-Pattern DQUOTE
    
    shojiIRIPatternArray =
        begin-array
        [ shojiIRIPattern *( value-separator shojiIRIPattern ) ]
        end-array
    
    shojiIRIPatternElement = string name-separator shojiIRIPattern
    shojiIRIPatternObject =
        begin-object
        [ shojiIRIPatternElement *( value-separator shojiIRIPatternElement ) ]
        end-object

3.2.1 IRI Pattern Object Names

When IRI Patterns are contained within a shojiIRIPatternObject, the name
portion of each shojiIRIPatternElement identifies the semantics of the link.
For example, an Entity that represents a User resource might include a single
shojiCatalog member; the identified resource returns a Catalog, to be sure,
but the IRIPattern which identifies that resource may be opaque. By supplying
the member name "orders", the Entity declares that the Catalog resource
returns Order resources related to the current User resource. In this way,
shojiIRIPatternObject names function similarly to link relation types found
in other specifications. Authors MAY use existing registries of link relation
types, but are not bound to.

3.2.2 IRI Pattern Substitution

Shoji client processors MUST NOT dereference an IRI Pattern directly;
instead, they MUST construct an IRI-reference to dereference. This section
describes, in prose, the algorithm for doing so.

For each var, if the corresponding variable is defined, substitute its value.
The value MUST be a Unicode string. Processors are free to use any data
types internally, but MUST convert them to a representation as a Unicode
string before substitution.

A given variable MAY appear in more than one expansion within a single IRI
Pattern; if so, it MUST substitute the same value each time.

If a value for a given variable is not defined, and the var is required,
the processor SHOULD raise an error and MUST NOT generate an IRI. If not
defined, and the var is not required, substitute the default value if
provided, otherwise omit the var from the production entirely.

If no operator is declared, join the substituted values within the expansion
with no intermediate characters. For example, the IRI Pattern "{a!,b,c=3}"
plus the value a=1 would produce the IRI-reference "13".

If the segment operator "/" is declared, join the substituted values within
the expansion with "/" as a prefix character for each. For example, the IRI
Pattern "foo{/a!,b,c=3}" plus the value a=1 would produce the IRI-reference
"foo/1/3".

If the parameter operator ";" is declared, join each varname within the
expansion with its substituted value using the "=" character; if the value
is equal to the empty string, the "=" character is omitted. Then join each
name-value pair with ";" as a prefix character. For example, the IRI Pattern
"foo{;a!,b,c=3}" plus the value a=1 would produce the IRI-reference
"foo;a=1;c=3".

If the querystring operator "?" is declared, join each varname within the
expansion with its substituted value using the "=" character; the "="
character is included even if the value is the empty string. Then join
each name-value pair with "&" as an intermediate character. Finally,
prefix the entire production with the "?" character. For example, the
IRI Pattern "foo{?a!,b,c=3}" plus the value a=1 would produce the IRI-
reference "foo?a=1&c=3". If no vars are substituted (i.e. if they are
all undefined and no default values are given), the "?" prefix MUST
still be output. This allows IRI Patterns of the form "a{?b}c=3" to
produce valid IRI's even where b is undefined (that is, "a?c=3", not
"ac=3").

In order to assist HTTP caches in reducing variant copies, the order
of query string variables in the expanded IRI SHOULD follow the order
of variables declared in the expansion. Shoji servers MAY require this.

3.2.3 IRI Pattern Matching

Shoji servers MAY employ IRI Patterns to match IRI's to handlers. This
section describes an algorithm (based on POSIX Extended Regular Expressions)
for doing so. All slashes ("\") are literal.

Create two empty lists for storing discovered (name, default) pairs.
Find expansions within the IRI Pattern using "\{[^}]+\}". Replace each one
with a regular expression according to the first character after the "{".
If the first character is one of the "op" characters, pop and store it.
Split the rest of the expansion by the "," character, then split each atom
on the "=" character, and store the resultant (name, default) pairs in order,
one list for "?" arguments and another for all other operators, or no operator.
If no default is given, store None or another sentinel. Depending on whether
the variable is required (i.e. ends with the "!" character) or not, replace
the expansion according to the following table:

    Operator       Required (!)    Not Required
    ---------------------------------------------
    /              (/[^/]*)        (/[^/]*)?
    ;              (;%s=?[^/;]*)   (;%s=?[^/;]*)?
    ?              Empty string    Empty string
    No operator    (.+)            (.+)?

The "%s" in either ";" expression in the above table must be replaced with
the (escaped) name of the variable.

Once all expansions have been substituted, the resultant regular expression
can be used to match the path portion of incoming Request-URI's. The query
string portion can be matched against the stored list for "?" expansions
in an order-neutral fashion. Shoji servers MAY require that query string
substitutions be performed in the order given in the IRI Pattern, in order
to reduce HTTP cache variants and to facilitate mapping substituted IRI
Pattern values to positional arguments in the matching handler.

3.3 Identifier members

Shoji objects may employ a variety of reserved object member names to
declare that a given member's value is an IRI, and IRI-reference or an
array or object of IRI-references.

3.3.1 Absolute Reference Members

3.3.1.1 self

Shoji objects may employ a reserved object member to declare the IRI of
the object itself:

    shojiSelfIRI = DQUOTE "self" DQUOTE name-separator shojiIRI

The object member name "self" is reserved for this purpose. The value MUST
be a shojiIRI. Note that the definition of "IRI" excludes relative references.

When a Shoji object is relocated, migrated, syndicated, republished,
exported, or imported, the content of its self member MUST NOT change.
Put another way, a self member pertains to all instantiations of a
particular Shoji entity or catalog; revisions retain the same content
in their self elements. It is suggested that the self element be stored
along with the associated resource.

The content of a shojiSelfIRI element MUST be created in a way that assures
uniqueness.

3.3.2 Relative Reference Members

Shoji objects may employ a variety of reserved object member names to
declare that a given member's value is an IRI-reference or an array or object
of IRI-references:

    shojiIdentifierMember =
        string name-separator
        ( shojiIdentifier / shojiIdentifierArray )
    shojiIRIPatternMember =
        string name-separator
        ( shojiIRIPattern / shojiIRIPatternArray /shojiIRIPatternObject )

Application authors should be aware of the many details of relative URI
resolution described in Section 5.2 of [RFC2396]; in particular, that
"all but the last segment of the base URI's path component is copied
to the buffer. In other words, any characters after the last (right-
most) slash character, if any, are excluded." The two URI's "/users"
and "/users/", for example, are different, may point to different
resources, and will be combined with relative URI's differently.

For Shoji documents, this implies two different approaches depending on
whether trailing slashes are preferred. If catalog resources employ
trailing slashes, then their links to subcatalogs, entities, views,
and so on are simpler. For example, the Catalog resource at "/users/" may
include a "bills" subcatalog. But if the Catalog is instead exposed at the
URI "/users" (no trailing slash), then the subcatalog IRI-reference must
be written as "users/bills" in order to be resolved correctly. Both forms
are permitted by this specification; application authors SHOULD use
HTTP redirects to help enforce one or the other as canonical.

If trailing slashes are preferred, then trailing slashes should be used
whenever the return type changes from the parent resource, and omitted when
the type does not change. For example, if the resource at "/users/" is a
Catalog for User resources, and the application designer wishes to include
subcatalogs for, on the one hand, active users, and on the other, Bill
resources, then they should expose the former without a trailing slash
("/users/active") since it will return User representations and the latter
with a trailing slash ("/users/bills/") since it will not return User
representations.

Shoji Entity resources MUST be exposed with trailing slashes so that their
body member names may serve as IRI's relative to them; that is, omitting
the trailing slash would force body member names to change unacceptably.
For example, the following entity:

   {"element": "shoji:entity",
    "self": "http://www.example.com/users/1234/",
    "body": {"first_name": "David",
             "last_name": "Robinson"}}

...if the trailing slash were omitted from "self", would be forced to change to:

   {"element": "shoji:entity",
    "self": "http://www.example.com/users/1234",
    "body": {"1234/first_name": "David",
             "1234/last_name": "Robinson"}}

Authors who can't help but think of a trailing slash as indicating an index or
collection are encouraged to view an Entity object as a collection of its values
for their own sanity.

3.3.2.1 catalogs

Shoji Catalog and Entity objects may employ a reserved object member to
declare the IRI's of related catalogs:

    shojiCatalogs = DQUOTE "catalogs" DQUOTE name-separator shojiIRIPatternObject

A GET on any of the dereferencedIRI's in the object MUST result in a Shoji
Catalog Document.

3.3.2.2 entities

Shoji Catalog objects may employ a reserved object member to declare the
IRI's of contained entities:

    shojiEntities = DQUOTE "entities" DQUOTE name-separator shojiIdentifierArray

A GET on any of the dereferencedIRI's in the array MUST result in a Shoji
Entity Document.

3.3.2.3 fragments

Shoji Entity objects may employ a reserved object member to declare the
IRI's of fragment entities:

    shojiFragments = DQUOTE "fragments" DQUOTE name-separator shojiIRIPatternObject

A GET on any of the dereferencedIRI's in the object MUST result in a Shoji
Entity Document.

Fragment entities are entities which contain a subset of the values for the
parent Entity. For example, if a single "user" Entity has public and private
attributes, a parent Entity at /users/1234/ might contain the public values,
and possess a fragment at /users/1234/private/ containing private values;
in this example, the parent Entity would possess a "fragments" member with
the value {"Priavte members": "private"}.

Parent Entities MAY include shojiValueMembers whose values are complete
fragment Entity objects, or MAY include none, any or all of the value
members of a fragment in their own set of value members. However,
implementers should be aware of the cache-invalidation problems which
arise when duplicating data in multiple resources, and trade simplicity
for convenience sparingly.

3.3.2.4 views

Shoji Catalog and Entity objects may employ a reserved object member to
declare the IRI's of related views:

    shojiViews =   DQUOTE "views" DQUOTE name-separator shojiIRIPatternObject

A GET on any of the dereferencedIRI's in the object SHOULD result in a Shoji
Value Document. Unlike body members (section 3.4), which are designed to return
mostly scalar values, views are preferred for returning tabular datasets.

3.4 Body Members

Some Shoji Documents benefit from a separation of data and metadata in order to
reduce namespace collisions and ease parsing. A reserved object member provides
this:

    shojiBodyMember = DQUOTE "body" DQUOTE name-separator value


3.5 Element Declarations

All 3 kinds of Shoji Document (Catalog, Entity, and Value) utilize the same
media-type: "application/shoji". Parsers need a means of distinguishing them.

If the document's top-level value is not a JSON object, it is a Shoji Value
Document. If the top-level value is an object, a reserved object member MUST
be used to indicate the type of the object:

    namespace = *char
    local_name = *char
    qualified_name = DQUOTE namespace ":" local_name DQUOTE
    element_member = DQUOTE "element" DQUOTE name-separator qualified_name

This is a generic, namespaced element declaration approach that we hope will be
adopted by other JSON-based media types.


4. Shoji Object Definitions

4.1 Container Objects

4.1.1 The "Catalog" Object

The "shojiCatalog" object acts as a container for metadata and data
associated with the catalog. Its members consist of a "self" attribute,
other metadata members, an optional catalogs object (with references to
its contained catalog documents), an optional entities array (with
references to its contained entity documents), and an optional views
object (with references to its contained value documents):

    shojiCatalogElement = DQUOTE "element" DQUOTE
                          name-separator "shoji:catalog"
    
    shojiCatalog = begin-object
                   shojiCatalogElement
                   shojiSelfIRI
                   *( value-separator member )
                   *1( value-separator shojiCatalogs )
                   *1( value-separator shojiEntities )
                   *1( value-separator shojiViews    )
                   end-object

Shoji catalog objects MAY follow the order of member elements given above;
however, the JSON specification defines the "object" type as an unordered set
of elements, and Shoji Processors MUST be prepared to find self, entities,
catalogs, views, or other member elements in any order.

This specification assigns no significance to the order of references within
the entities array. Applications are free to do so for their own purposes.

The "other metadata members" MAY include schemas for, or prototypes of,
Shoji Entity objects, or IRI's identifying such. Their syntax is not
mandated nor constrained by this specification.

4.1.2 The "Entity" Object

The "shojiEntity" object acts as a container for data associated with the
entity; that is, its contained Value objects, sub-catalog identifiers,
fragment identifiers, and view identifiers:

    shojiEntityElement = DQUOTE "element" DQUOTE
                         name-separator "shoji:entity"
    
    shojiEntityBody = begin-object
                      *( value-separator shojiValueMember )
                      end-object
    shojiEntityBodyMember = DQUOTE "body" DQUOTE
                            name-separator shojiEntityBody
    
    shojiEntity = begin-object
                  shojiEntityElement
                  shojiSelfIRI
                  *1( shojiEntityBodyMember )
                  *1( value-separator shojiCatalogs )
                  *1( value-separator shojiFragments )
                  *1( value-separator shojiViews )
                  end-object

Each member contained in the EntityBody SHOULD be referenceable by combining its
name with the "self" attribute of the Entity, according to the relative URI rules
in [RFC3986]. Note that, although the "self" member is part of the JSON object,
it is not a shojiValueMember and therefore is not subject to this stipulation.
If the "self" attribute is empty or not a valid IRI, the contained ValueMembers
are considered unreferenceable via distinct Identifiers.

Shoji Entity resources MUST be exposed with trailing slashes so that their
body member names may serve as IRI's relative to them. See section 3.3.2.

Note that Entity objects have no generic extension for including body
members which are not referenceable at a separate IRI. This is intentional.
Server authors MAY elect to return 404 Not Found for any Value resource (and
indeed, any code including 405 Method Not Allowed if desired to restrict
the set of HTTP methods); however, this does not nullify the expectation
that the Value IRI points to the expected resource. This promotes evolvable
structures; the Value resource can be added or modified at a later date
without changing the Entity which refers to it.

4.1.3 The "Value" Object

The "shojiValueName" object is any JSON string. The "shojiValue"
object is any JSON value:

    shojiValueName = string
    shojiValue = value

Shoji Value Documents contain a single shojiValue, although that value may
be an arbitrarily-complex object or array. They MAY contain embedded Catalog
or Entity objects; however, authors MUST still follow the "self" requirement
for these objects, and should keep in mind that Catalog and Entity member
values carry with them implied independent IRI's of their own. If an embedded
Catalog or Entity is not intended to "stand on its own" outside the Value
(that is, answer to various HTTP methods on "self" and its member values'
IRI's), authors are encouraged to emit tabular data rather than lists of
Catalog or Entity objects.

Note that documents of the media type "application/json" must be of type
"object" or "array", but Shoji Value Documents may be any valid JSON type,
including false, null, true, object, array, number, or string.

The "shojiValueMember" object is the combination of a value name
and value for inclusion within a shojiEntity:

    shojiValueMember = shojiValueName name-separator shojiValue


5. Control and Publishing

The Shoji Catalog Protocol uses HTTP methods to author Shoji Resources as follows:

 * GET is used to retrieve a representation of a known Catalog, Entity, or Value.
 * POST is used to create a new, dynamically identified, Entity.
 * PUT is used to create or edit an Entity, or edit a Value, with a known identity.
 * DELETE is used to remove a known Entity.

The Shoji Protocol only covers the creating, editing, and deleting of Entity and
Value Resources. Other Resources could be created, edited, and deleted as the result
of manipulating a Catalog, but the number of those Resources, their media types,
and effects of the Shoji Protocol operations on them are outside the scope of this
specification.

While specific HTTP status codes are shown in the interaction diagrams below, a Shoji
client should be prepared to handle any status code. For example, a PUT to an Entity
URI could result in the return of a "204 No Content" status code, which still indicates
success.

Since all aspects of client-server interaction are defined in terms of HTTP,
[RFC2616] should be consulted for any areas not covered in this specification.

5.1 Catalogs

5.1.1 Retrieving a Catalog

Client                                     Server
  |                                           |
  |  1.) GET to Catalog URI                   |
  |------------------------------------------>|
  |                                           |
  |  2.) 200 OK                               |
  |      Shoji Catalog Document               |
  |<------------------------------------------|
  |                                           |

   1. The client sends a GET request to the URI of the Catalog Document.
   2. The server responds with a Catalog Document whose entities enumerate
      the IRIs of its contained Entities and/or whose catalogs enumerate its
      sub-Catalogs and/or whose views enumerate its value queries. The content
      of this document can vary based on aspects of the client request, including,
      but not limited to, authentication credentials.

If the returned Catalog Document contains an entities member, that member MUST
refer to all accessible Entities included in the Catalog.
If the returned Catalog Document contains a catalogs member, that member MUST
refer to all accessible Catalogs included in the Catalog.
If the returned Catalog Document contains a views member, that member MUST
refer to all accessible Views included in the Catalog.

5.2 Entities

5.2.1 Creating an Entity

Client                                     Server
  |                                           |
  |  1.) POST to Catalog URI                  |
  |      Entity Representation                |
  |------------------------------------------>|
  |                                           |
  |  2.) 201 Created                          |
  |      Location: Entity URI                 |
  |<------------------------------------------|
  |                                           |

   1. The client POSTs a representation of the Entity to the URI of the Catalog.
      If the Entity includes a self member, the server MAY use it as a hint to
      construct the returned Entity URI, or it MAY ignore it.
   2. If the Entity Resource was created successfully, the server responds with
      a status code of 201 and a Location header that contains the IRI of the
      newly created Entity Resource. If the client is storing the Entity Document
      it passed in the request, it SHOULD incorporate the returned Entity IRI as
      the self member of the Entity, overwriting any existing self member.

5.2.2 Retrieving an Entity

Client                                     Server
  |                                           |
  |  1.) GET to Entity URI                    |
  |------------------------------------------>|
  |                                           |
  |  2.) 200 OK                               |
  |      Entity Representation                |
  |<------------------------------------------|
  |                                           |

   1. The client sends a GET request to the URI of an Entity Resource to retrieve
      its representation.
   2. The server responds with the representation of the Entity Resource, a Shoji
      Entity Document.

5.2.3 Editing an Entity

Client                                     Server
  |                                           |
  |  1.) PUT to Entity URI                    |
  |      Entity Representation                |
  |------------------------------------------>|
  |                                           |
  |  2.) 200 OK                               |
  |<------------------------------------------|

   1. The client sends a PUT request to store a representation of an Entity.
   2. If the request is successful, the server responds with a status code of 200.

5.2.4 Deleting an Entity

Client                                     Server
  |                                           |
  |  1.) DELETE to Entity URI                 |
  |------------------------------------------>|
  |                                           |
  |  2.) 200 OK                               |
  |<------------------------------------------|
  |                                           |

   1. The client sends a DELETE request to the URI of an Entity.
   2. If the deletion is successful, the server responds with a status code of 200.

5.3 Values

The set of Value Names of a given Entity are fixed by the server. Their values
are all considered potentially mutable, however, and may be any valid JSON value.
It is important to note that a Shoji Value document may be more than just JSON
text, which is limited to objects or arrays in part to reduce security concerns.
Because Shoji documents do not have this restriction, Shoji clients MUST NOT use
Javascript eval() or equivalent mechanisms in other languages to parse Shoji
documents.

5.3.1 Retrieving a Value

Client                                     Server
  |                                           |
  |  1.) GET to Value URI                     |
  |------------------------------------------>|
  |                                           |
  |  2.) 200 OK                               |
  |      Value Representation                 |
  |<------------------------------------------|
  |                                           |

   1. The client sends a GET request to the URI of a Value Resource to retrieve
      its representation.
   2. The server responds with the representation of the Value Resource,
      a Shoji Value Document.

5.3.2 Editing a Value

Client                                     Server
  |                                           |
  |  1.) PUT to Value URI                     |
  |      Value Representation                 |
  |------------------------------------------>|
  |                                           |
  |  2.) 200 OK                               |
  |<------------------------------------------|

   1. The client sends a PUT request to store a representation of a Value.
   2. If the request is successful, the server responds with a status code of 200.


5.4 Use of HTTP Response Codes

The Shoji Protocol uses the response status codes defined in HTTP to indicate
the success or failure of an operation. Consult the HTTP specification [RFC2616]
for detailed definitions of each status code. Implementers are asked to note
that according to the HTTP specification, HTTP 4xx and 5xx response entities
SHOULD include a human-readable explanation of the error.


6. IANA Considerations

6.1 The "application/shoji" media-type

A Shoji Document, when serialized as JSON, can be identified with the following
media type:

MIME media type name: application
MIME subtype name: shoji
Mandatory parameters: None.
Optional parameters: None.

Encoding considerations: Identical to those of "application/json" as described
    in [RFC4627], Section 3.
Security considerations: As defined in this specification.
Interoperability considerations: There are no known interoperability issues.
Published specification: This specification.
Applications that use this media type: No known applications currently use this
    media type.

Additional information:

Magic number(s): None.
File extension: .shoji
Base URI: As specified in [RFC2396], Section 6.
Macintosh File Type code: TEXT
Person and email address to contact for further information:
    Robert Brewer <fumanchu@aminus.org>
Intended usage: COMMON
Restrictions on usage: None.
Change controller: Robert Brewer <fumanchu@aminus.org>


7. Security Considerations

7.1 URIs

Shoji Processors handle URIs. See Section 7 of [RFC3986].

7.2 IRIs

Shoji Processors handle IRIs. See Section 8 of [RFC3987].

7.3 Spoofing

Shoji Processors should be aware of the potential for spoofing attacks where
the attacker publishes a Shoji document with self or entities members,
perhaps with a falsified shojiIRI element duplicating the identity of
another document. For example, a Shoji Processor could suppress display
of duplicate entries by displaying only one entry from a set of entries
with identical self values. In that situation, the Shoji Processor might
also take steps to determine whether the entries originated from the same
publisher before considering them duplicates.

7.4 Shoji Document Parsing

Shoji Processors MUST NOT use Javascript eval() or similar mechanisms to
parse Shoji Documents; doing so could expose them to malicious code from
untrusted sources or XSS vectors. JSON requires shipped documents to be
of top-most type "object" or "array", which ameliorates these vectors
somewhat. Shoji documents do not have this restriction.

8. References

8.1 Normative References
[RFC2119] Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels”,
          BCP 14, RFC 2119, March 1997.
[RFC2396] Berners-Lee, T., Fielding, R. and L. Masinter, "Uniform Resource"
          Identifiers (URI): Generic Syntax.", RFC 2396, August 1998.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P.,
          and T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1”, RFC 2616,
          June 1999.
[RFC3023] Murata, M. St.Laurent, S. Kohn, D. "XML Media Types", RFC 3023, January 2001.
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, “Uniform Resource Identifier
          (URI): Generic Syntax”, STD 66, RFC 3986, January 2005.
[RFC3987] Duerst, M. and M. Suignard, “Internationalized Resource Identifiers (IRIs)”,
          RFC 3987, January 2005.
[RFC4627] D. Crockford, "The application/json Media Type for JavaScript Object Notation
          (JSON)", RFC 4627, July 2006


8.2 Informative References
[RFC2434] Narten, T. and H. T. Alvestrand, “Guidelines for Writing an IANA
          Considerations Section in RFCs”, BCP 26, RFC 2434, October 1998.
[RFC4234] Crocker, D. and Overell, P., “Augmented BNF for Syntax Specifications: ABNF”,
          RFC4234, October 2005.


A. Contributors

The following people contributed to preliminary versions of this document:
    Robert Brewer

B. Collected ABNF

This appendix is informative.

    shojiIRI = DQUOTE IRI DQUOTE
    shojiIdentifier = DQUOTE IRI-reference DQUOTE
    shojiIdentifierArray =
        begin-array
        [ shojiIdentifier *( value-separator shojiIdentifier ) ]
        end-array
    
    iunreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar
    
    op          = "/" / ";" / "?"
    varname     = 1*( iunreserved )
    required    = "!"
    vardefault  = *( iunreserved / pct-encoded )
    var         = varname [ required / ( "=" vardefault ) ]
    vars        = var *( "," var )
    expansion   = "{" [ op ] vars "}"

    shojiIRIPattern = DQUOTE IRI-Pattern DQUOTE
    shojiIRIPatternArray =
        begin-array
        [ shojiIRIPattern *( value-separator shojiIRIPattern ) ]
        end-array
    
    shojiIRIPatternElement = string name-separator shojiIRIPattern
    shojiIRIPatternObject =
        begin-object
        [ shojiIRIPatternElement *( value-separator shojiIRIPatternElement ) ]
        end-object
    
    shojiIdentifierMember =
        string name-separator
        ( shojiIdentifier / shojiIdentifierArray )
    shojiIRIPatternMember =
        string name-separator
        ( shojiIRIPattern / shojiIRIPatternArray /shojiIRIPatternObject )
    
    shojiSelfIRI = "self" name-separator shojiIRI
    
    shojiCatalogs  = "catalogs"  name-separator shojiIRIPatternObject
    shojiEntities  = "entities"  name-separator shojiIdentifierArray
    shojiFragments = "fragments" name-separator shojiIRIPatternObject
    shojiViews     = "views"     name-separator shojiIRIPatternObject
    
    shojiValueName = string
    shojiValue = value
    shojiValueMember = shojiValueName name-separator shojiValue
    
    namespace = *char
    local_name = *char
    qualified_name = DQUOTE namespace ":" local_name DQUOTE
    element_member = DQUOTE "element" DQUOTE name-separator qualified_name
    
    shojiBodyMember = DQUOTE "body" DQUOTE name-separator value
    
    shojiEntityElement = DQUOTE "element" DQUOTE
                         name-separator "shoji:entity"
    
    shojiEntityBody = begin-object
                      *( value-separator shojiValueMember )
                      end-object
    shojiEntityBodyMember = DQUOTE "body" DQUOTE
                            name-separator shojiEntityBody

    shojiEntity = begin-object
                  shojiEntityElement
                  shojiSelfIRI
                  *1( shojiEntityBodyMember )
                  *1( value-separator shojiCatalogs )
                  *1( value-separator shojiFragments )
                  *1( value-separator shojiViews )
                  end-object
    
    shojiCatalogElement = DQUOTE "element" DQUOTE
                          name-separator "shoji:catalog"
    shojiCatalog = begin-object
                   shojiCatalogElement
                   shojiSelfIRI
                   *( value-separator member )
                   *1( value-separator shojiCatalogs )
                   *1( value-separator shojiEntities )
                   *1( value-separator shojiViews )
                   end-object