Keep an unknown JSON field for forward-compatible reporting

By Batu · English technical notes

Keep an unknown JSON field for forward-compatible reporting

To keep an unknown JSON field for forward-compatible reporting, validate the known fields and copy all remaining members into an extras mapping. Do not rebuild the outgoing record from known fields alone when the report must preserve fields introduced by a newer producer.

The fixture is the exact JSON object {"id": 1, "future_flag": true}. The strict projection validates nothing beyond selecting id, and its compact report visibly loses future_flag. validate_and_keep instead checks that the decoded value is a dictionary and that id is an integer (excluding bool, which is an int subclass in Python). It returns the validated known part alongside extras. Merging those mappings produces a JSON document that decodes back to the original synthetic record. The assertions establish that this particular field survives; they do not establish compatibility with every future schema or validate the value of unknown fields.

The standard decoder converts a JSON object to dict, and json.dumps serializes a dictionary as a JSON object; both behaviors are documented in the json module. sort_keys=True is used only to make the displayed report deterministic. This example uses dictionary unpacking, so it requires Python 3.5+.

AI assistance disclosure: this article was drafted with AI assistance and uses a fixed in-memory fixture.

import json

source = '{"id": 1, "future_flag": true}'
record = json.loads(source)

strict = {"id": record["id"]}
strict_report = json.dumps(strict, separators=(",", ":"))
assert strict_report == '{"id":1}'
print("strict report:", strict_report)


def validate_and_keep(value):
    if not isinstance(value, dict):
        raise ValueError("record must be an object")
    identifier = value.get("id")
    if isinstance(identifier, bool) or not isinstance(identifier, int):
        raise ValueError("id must be an integer")
    known = {"id": identifier}
    extras = {key: item for key, item in value.items() if key not in known}
    return known, extras


known, extras = validate_and_keep(record)
round_trip = {**known, **extras}

assert extras == {"future_flag": True}
assert json.loads(json.dumps(round_trip)) == record
print("extras:", json.dumps(extras, sort_keys=True, separators=(",", ":")))
print("report:", json.dumps(round_trip, sort_keys=True, separators=(",", ":")))

strict report: {"id":1}
extras: {"future_flag":true}
report: {"future_flag":true,"id":1}