An airport asks for your passport at check-in. Then the lounge asks for your booking reference. Then the duty free asks for your boarding pass. Then the transfer desk asks for your passport again. Every one of those partners keeps a copy of whatever you gave them, and you have no idea who holds what.
We built BCMS to invert that. The passenger holds the consent. A partner sees only the fields the passenger allowed, every access is written to a chain the passenger can read, and nobody keeps a copy of anything they were not given.
It was selected in the top five nationally at the Unisys Cloud 20/20 hackathon from over six hundred teams. It was also my first blockchain project, and I got a lot of it wrong before I got it right.
Why blockchain at all
This is the question worth asking first, because the honest answer for most blockchain projects is "we should not have."
Our answer came down to one property. The value here is not decentralization, and it is certainly not currency. It is that an audit log must be trustworthy to a party who does not trust whoever runs it.
If the airport keeps the access log in its own Postgres, a passenger has to take the airport's word that the log is complete. So does a regulator. So does a partner in a dispute. Anyone with database access can quietly delete a row.
An append-only ledger replicated across the airport, the partner and the passenger organizations solves exactly that, and nothing else. That was worth being clear about early, because it kept us from putting things on the chain that had no business being there.
Why Fabric and not Ethereum
Everyone who knew about blockchain in 2019 knew about Ethereum, so this needed justifying.
An airport is not an anonymous network. Every participant is a known, legally accountable organization. That single fact changes what you need.
Ethereum is public and permissionless. Anyone can join, so consensus has to assume some participants are hostile, which is what proof of work is for. It is expensive, it is slow, and every piece of state is visible to everyone forever. For passenger data, that last property alone rules it out.
Hyperledger Fabric is permissioned. You get an identity from a certificate authority before you can transact at all. Because membership is known, consensus does not need to be adversarial, so ordering is just a service. No mining, no gas, and transaction finality in the order of a second.
Three Fabric features decided it for us.
Channels. A channel is a private ledger between a specific set of organizations. The airport and Partner A share a channel that Partner B cannot see at all. Not encrypted from B, invisible to B.
Pluggable state database. Fabric stores current state in either LevelDB or CouchDB. Choosing CouchDB lets you store JSON and run rich queries against it, which matters enormously once you have anything more complex than a key-value pair.
Chaincode in a real language. Fabric smart contracts are ordinary Go programs, not a purpose-built language like Solidity. I could use the standard library and reason about the code the way I reason about any other code.
Our network had three peer organizations and a Raft ordering service.
What chaincode actually is
I had imagined something exotic. It is a Go program with two required methods that runs in a Docker container, which the peer starts and talks to over gRPC.
package main
import (
"encoding/json"
"fmt"
"github.com/hyperledger/fabric/core/chaincode/shim"
sc "github.com/hyperledger/fabric/protos/peer"
)
// SmartContract defines the Smart Contract structure
type SmartContract struct {
}
// Init is called once, when the chaincode is instantiated on the channel
func (s *SmartContract) Init(APIstub shim.ChaincodeStubInterface) sc.Response {
return shim.Success(nil)
}
// Invoke is called for every transaction after that
func (s *SmartContract) Invoke(APIstub shim.ChaincodeStubInterface) sc.Response {
function, args := APIstub.GetFunctionAndParameters()
if function == "queryUserData" {
return s.queryUserData(APIstub, args)
} else if function == "createUserConsent" {
return s.updateUserConsent(APIstub, args)
} else if function == "addDataAccessLog" {
return s.addDataAccessLog(APIstub, args)
}
// ...
return shim.Error("Invalid Smart Contract function name.")
}That Invoke is a manual router. Fabric hands you a function name and a slice of
strings, and dispatching them is your problem. There is no framework doing it for
you, which felt primitive at first and then felt honest.
Note that createUserConsent and updateUserConsent both call
updateUserConsent. That was intentional, because the function upserts, but it
is the kind of thing that should have had a comment saying so rather than looking
like a copy-paste mistake. I know it looks like a copy-paste mistake because I
have had to explain it three times.
The stub is the entire world
shim.ChaincodeStubInterface, which everyone calls the stub, is how chaincode
touches the ledger. It is worth understanding what it can and cannot do.
APIstub.GetState(key) // read current value
APIstub.PutState(key, valueBytes) // write, appears after the tx commits
APIstub.DelState(key) // mark deleted
APIstub.GetHistoryForKey(key) // every version ever written
APIstub.GetFunctionAndParameters() // the call being madeThe rule that took me longest to internalize: chaincode must be deterministic. Every endorsing peer runs the same transaction independently, and their results are compared. If two peers disagree, the transaction fails.
So no time.Now(). No random numbers. No network calls. No reading a file. Any
of those can differ between peers, and the transaction gets rejected with an
error message that does not tell you why.
I found this out the way everyone does, by writing a timestamp into a record and watching endorsement fail. The fix is to pass anything non-deterministic in as an argument, so the client decides the value once and every peer sees the same one.
Watching a transaction go through
This is the part of Fabric that is genuinely different, and the part I had backwards in my head for weeks.
On most chains, a transaction is ordered first and executed second. On Fabric it is the other way round. Peers execute it first, independently, against their own copy of state, and produce a signed record of what they read and what they would write. Only then does it go to the orderer. Validation at the end checks that the reads are still true.
Step through it. Then switch to the second scenario, which is the failure mode that will eventually cost you an afternoon.
The passenger grants a lounge access to their masked record. Nothing else is competing for the key.
Client
passenger app
UserOrg
peer
AirportOrg
peer
PartnerOrg
peer
Orderer
Raft
Phase 1 of 5 · Proposal
The client signs a proposal with the passenger's own certificate and sends it to the endorsing peers. Nothing is on the ledger yet.
That second one is worth sitting with. Both transactions endorsed successfully.
Both were correct at the moment they were simulated. The conflict only becomes
visible at validation, after ordering, when the first one has already moved the
version. There is nothing wrong with the code, and the error message,
MVCC_READ_CONFLICT, does not explain any of this.
The practical consequence: any key that several clients write to concurrently is a bottleneck, and the fix is usually to restructure the data so writes do not collide rather than to retry harder. A per-user consents record, as we had, is fine. A single global counter would have been a disaster.
GetHistoryForKey deserves special mention, because it is the reason the whole
project works. It returns every version a key has ever had, with the transaction
ID, the timestamp and whether it was a delete.
func GetHistory(APIstub shim.ChaincodeStubInterface, key string) []byte {
resultsIterator, _ := APIstub.GetHistoryForKey(key)
defer resultsIterator.Close()
var buffer bytes.Buffer
buffer.WriteString("[")
bArrayMemberAlreadyWritten := false
for resultsIterator.HasNext() {
response, _ := resultsIterator.Next()
if bArrayMemberAlreadyWritten == true {
buffer.WriteString(",")
}
buffer.WriteString(`{"TxId":"`)
buffer.WriteString(response.TxId)
buffer.WriteString(`", "Value":`)
// A delete has no value, so write null rather than empty bytes
if response.IsDelete {
buffer.WriteString("null")
} else {
buffer.WriteString(string(response.Value))
}
// timestamp, isDelete, ...
bArrayMemberAlreadyWritten = true
}
buffer.WriteString("]")
return buffer.Bytes()
}Building JSON by writing bytes into a buffer looks archaic next to
json.Marshal. I copied the pattern from the Fabric samples and kept it, because
the history response has a shape that does not map cleanly onto a Go struct.
Given the same problem now I would define a type and marshal it, and accept the
small amount of extra code.
That function is what lets a passenger ask "who has read my data, and when," and get an answer nobody can edit.
The decision the whole design rests on
Here is the part I would defend hardest.
No personal data is on the blockchain. Not encrypted. Not in a private data collection. Not at all.
A blockchain is append-only and replicated. Those are the properties you want for an audit trail and exactly the properties you must not have for personal data. You cannot delete it, which means you cannot honour a deletion request. It exists on every peer, which means every organization holds a copy forever. Encryption does not save you, because encryption is a bet on how long a key stays secret and "forever" is a long time to bet on.
So the ledger holds identifiers and hashes. Nothing else.
// UserData defines the userData structure
type UserData struct {
DataID string `json:"dataID"`
DataHash string `json:"dataHash"`
}
// UserConsent holds a single consent of a user
// (it can be said as a user - partner one to one relation)
type UserConsent struct {
ConsentID string `json:"consentID"`
ConsentHash string `json:"consentHash"`
PartnerID string `json:"partnerID"`
MaskedDataID string `json:"maskedDataID"`
MaskedDataHash string `json:"maskedDataHash"`
}The real record lives in a conventional database. The chain holds a hash of it. Anyone can verify the record has not been altered by hashing it and comparing. Nobody can read anything from the chain alone. And when a passenger asks to be forgotten, you delete the record from the database, and what remains on the chain is a hash of something that no longer exists anywhere.
Here is that property, working. Each block carries the hash of the one before it, which is what makes the chain a chain. Block 2841 sealed the hash of the record. Block 2842 recomputes it at read time and compares.
The record underneath is off-chain, so you can edit it. The blocks cannot be edited. Change a character and watch the link break.
Channel ledger
Consent granted
- prev
- 000000000000
- hash
- 4a1c9e07b3f2
Data hash committed
- prev
- 4a1c9e07b3f2
- hash
- ············
Partner read
- prev
- ············
- hash
- ············
Block 2842 recomputes the hash at read time and compares it with what block 2841 sealed.
Off-chain database, not on any ledger
Chain broken. This record is not what was committed.
The digests are real SHA-256, computed in your browser over the record as serialized in field order. Nothing is faked, which matters, because a demo that faked it would prove nothing about a claim I am asking you to believe. The values in the fields are placeholders for the same reason the system exists.
Notice what the blocks never contain. No passenger name, no flight. Identifiers and digests, which is enough to prove the record has not moved and useless to anyone who steals the ledger.
MaskedDataID is the other half. A partner does not get a consent that points at
the full record. They get one that points at a masked projection containing only
the fields that partner was granted. The lounge sees a name. The transfer desk
sees a flight number. Neither knows the other exists.
Hashing an object is harder than it sounds
If the chain holds a hash, the hash has to be reproducible. Hash the same record twice and get two different values, and every verification fails.
JSON does not guarantee key order. Two libraries, two language versions, or two runs of the same code can serialize the same object with keys in a different sequence, and the hashes differ completely.
So field order is fixed by declaration, not by whatever the serializer felt like.
enum FieldNames {
consentID = "consentID",
partnerID = "partnerID",
name = "name",
phone = "phone",
email = "email"
}
// IMPORTANT: should not add values in between. Only append
const userConsentOrder: FieldNames[] = [
FieldNames.consentID,
FieldNames.partnerID,
FieldNames.name,
FieldNames.phone,
FieldNames.email
];That comment is the most load-bearing line in the codebase. Insert a field into the middle of that array and every hash ever written becomes unverifiable. Append only.
The hashing itself normalizes before serializing.
const orderKeys = (obj: object, order: string[]): object => {
const sortedObj: object = {};
order.forEach(key => {
sortedObj[key] = obj[key];
});
return sortedObj;
};
// trailing nulls are dropped so an added-but-unused field
// does not change the hash of existing records
const removeTerminalNullValues = (obj: object): object => {
let lastTrueIndex = 0;
const Keys = Object.keys(obj);
Keys.forEach((key, index) => {
if (obj[key]) {
lastTrueIndex = index;
}
});
for (let i = lastTrueIndex + 1; i < Keys.length; i++) {
delete obj[Keys[i]];
}
return obj;
};
const hashObject = async (obj: object, order: string[]): Promise<string> => {
const orderedNormalizedObj = removeTerminalNullValues(orderKeys(obj, order));
const stringOfObj = JSON.stringify(orderedNormalizedObj);
return await bcrypt.hash(stringOfObj, 10);
};Order the keys, drop trailing nulls, serialize, hash. The trailing-null rule is what makes the append-only schema rule work: add a sixth field, leave it empty on old records, and their hashes do not move.
One thing I would change. This should not be bcrypt. bcrypt is a password
hashing function, deliberately slow and deliberately salted, which is exactly
right for passwords and wrong here. Because it salts, the same input gives a
different output every time, so you can only ever compare, never recompute and
match. And the slowness that protects a password database is pure cost when you
are hashing a record on every write. SHA-256 is the correct tool for a content
digest. I reached for the hashing function I had used before rather than the one
the problem called for.
CouchDB, and why it mattered
Fabric can keep world state in LevelDB or CouchDB. LevelDB is a key-value store, so you can fetch by key and nothing else. CouchDB stores the values as JSON documents and lets you query inside them.
couchdb0:
container_name: couchdb0
image: hyperledger/fabric-couchdb
ports:
- "5984:5984"
peer0.airportOrg.redmoon.live:
environment:
- CORE_LEDGER_STATE_STATEDATABASE=CouchDB
- CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb0:5984One CouchDB container per peer, because world state is per peer rather than shared.
We needed it for questions like "every consent this partner holds," which is a query over the contents of records rather than a lookup by key. With LevelDB the only way to answer that is to iterate every key in the chaincode, which is slow and, worse, produces a read set covering the entire ledger. Any concurrent write anywhere then invalidates your transaction.
The catch nobody warns you about: CouchDB queries are not part of consensus.
GetQueryResult is not re-executed at validation time, so a rich query is safe
for reads and dangerous inside a transaction that writes. Between endorsement and
commit the result could change, and Fabric will not catch it. Use rich queries to
read; use key-based access when a write depends on the result.
That was the single most confusing thing I learned on this project, and the documentation says it in one sentence you can easily miss.
Where the parallel arrays came from
The consent storage looks unusual, and I want to explain it rather than hide it.
// UserConsents holds all consent of a user
// (it can be said as a user - partner one to many relation)
type UserConsents struct {
ConsentIDs []string `json:"consentIDs"`
ConsentHashes []string `json:"consentHashes"`
PartnerIDs []string `json:"partnerIDs"`
MaskedDataIDs []string `json:"maskedDataIDs"`
MaskedDataHashes []string `json:"maskedDataHashes"`
}Five parallel arrays where a slice of structs would obviously be better. Lookup is by index.
indexOfPartner := IndexOf(userConsents.PartnerIDs, partnerID)
if indexOfPartner != -1 {
userConsent.ConsentID = userConsents.ConsentIDs[indexOfPartner]
userConsent.ConsentHash = userConsents.ConsentHashes[indexOfPartner]
// ...
}All five arrays have to stay the same length, and nothing in the type system enforces it. If an update appends to four of them and misses one, every consent after that index silently belongs to the wrong partner. That is a data correctness bug that no compiler catches.
I did it because I was thinking about the JSON shape in CouchDB rather than the
Go shape in memory, and arrays of scalars looked simpler to query. It was the
wrong trade. []UserConsent would have serialized fine and been impossible to
desynchronize.
The keys themselves are prefixed by type, which I do still like.
func GetConsentsIDFromUID(uid string) string { return "C-" + uid }
func GetDataIDFromUID(uid string) string { return "D-" + uid }
func GetDataAccessIDFromUID(uid string) string { return "DA-" + uid }A flat keyspace with typed prefixes. C- for consents, D- for data, DA- for
access logs. Simple, readable in the CouchDB console, and it makes range queries
by type possible.
Talking to the chain from Node
The application layer used the Fabric SDK. A client connects through a gateway using an identity from a wallet, gets a channel, gets a contract, submits.
import { FileSystemWallet, Gateway } from "fabric-network";
const wallet = new FileSystemWallet(walletPath);
const userExists = await wallet.exists(userID);
if (!userExists) {
return { success: false, message: "User is not registered." };
}
const gateway = new Gateway();
await gateway.connect(ccpPath, {
wallet,
identity: userID,
discovery: { enabled: true, asLocalhost: true }
});
const network = await gateway.getNetwork("secnodedefaultchannel");
const contract = network.getContract("consentManagement");
await contract.submitTransaction(transactionName, ...transactionArgs);
await gateway.disconnect();Two things here are worth pointing out.
submitTransaction and evaluateTransaction are not interchangeable. Submit
sends to the orderer and waits for commit. Evaluate only queries a peer and
changes nothing. Using submit for a read works and costs you a full consensus
round for no reason.
The identity is the passenger's own. Not a service account acting for them. That means the certificate that signed the transaction belongs to the person who gave the consent, and it is recorded on the chain that way. If you use one system identity for everything, your audit trail records that the system did it, which tells nobody anything.
What I would do differently
The hash function. SHA-256, not bcrypt. Explained above.
The parallel arrays. A slice of structs. Explained above.
Access control in the chaincode. Ours checks arguments, not callers. Fabric gives you the invoker's certificate through the client identity library, and the chaincode should verify that the caller is allowed to write this particular consent, rather than trusting the application layer to have checked. Right now any authenticated identity that reaches the chaincode can call any function.
Error handling. Almost every GetState in the file discards its error with
_. On a demo, a failed read returns nil, json.Unmarshal gives you a zero
value, and the transaction succeeds with empty data. That was a hackathon
decision that I would not repeat.
Private data collections. Fabric 1.4 supports keeping data with a subset of organizations, hashed on the shared ledger. That is arguably a better fit for the masked projections than an off-chain database. We did not use it because I did not understand it in time.
What it taught me
Most of what I learned was not about blockchain.
It was about deciding what not to put in the durable, replicated, permanent place. Once I understood that append-only is a promise you cannot take back, the whole architecture followed from it. Identifiers and hashes on the chain, data in a database you can actually delete from.
The other thing was determinism. Fabric forces you to write code that produces the same answer on every machine, every time, and it refuses your transaction if you get it wrong. That constraint is annoying for about a week and then it changes how you think about side effects generally.
I would not use a blockchain for most things. But for "prove to someone who does not trust you that this log is complete," I have not found anything better.