Brief

This article introduces a web front-end solution based on blockchain wallet, which can encrypt large files and store them in decentralized storage nodes through the Internet while ensuring the security of end-to-end transmission.

This solution can solve the security and exclusivity issues of file exchange and data transactions on the Internet, providing a reliable solution for data ownership and circulation in the digital economy era.

Finally, this article proposes an idea combining NFT non-fungible asset proof technology to realize data ownership certificates, providing a more trustworthy way for data ownership and circulation.

These solutions and ideas has and will be implemented in the Chatpuppy project.

As an instant messaging application, it is necessary to support peer-to-peer encrypted transmission of media and other large files. Considering to protect users’ data security, Chatpuppy needs to solve the following problems compared to conventional encrypted chat apps such as Whatsapp, telegram:

• Excessive redundancy of encrypted files cannot be caused by achieving peer-to-peer encrypted transmission.

• The file size cannot increase after encryption.

• Even if the file is stored on a public storage service (such as IPFS, ARwave, etc.), security can be ensured.

• The encryption key must be secure enough.

• Must be convenient enough to users.

To meet the above requirements, we have adopted the following technical solutions:

• Symmetric encryption is applied to the file, with a randomly and strong generated encryption key corresponding to each file on sender’s computer locally.

• After performance comparison, we abandoned the more versatile AES128/256 algorithm and switched to the faster Rabbit algorithm.

• The above encryption key is encrypted with the recipient's public key, and the recipient decrypts it with their own private key to ensure that only the sender and recipient can obtain the encryption key.

• Before encryption, the file is first compressed. After comparison, the zstd compression algorithm was selected. After testing, even after compressing and encrypting jpg files, the file size is only 60-70% of the original file size.

• The encrypted file is stored on IPFS.

• All of the above operations are performed in a blockchain wallet (such as MetaMask wallet for Ethereum or Alby wallet for Lightning Network), which can complete compression, encryption, and uploading with one click, and downloading, encryption, and decompression with one click.

After implementing the above functions, we found that this solution is not only suitable for web3 encrypted chat applications, but also can solve the problem of data ownership verification and anti-piracy in the data circulation and transaction process. The cryptographic algorithms used in this solution have been widely applied and verified, and they are secure and reliable, with the characteristic of being pluggable, which means they can be easily replaced with other cryptographic algorithms or file storage systems. This makes the solution adaptable to different application scenarios and requirements, with greater flexibility and scalability.

Below is a detailed explanation of the implementation steps and related code:
Because it is a front-end application, for ease of understanding, TypeScript programming is used, but the same logic can be implemented using any other language.

Encryption

It is divided into the following steps:

• 1- Get the local file.

• 2- Compress the file.

• 3- Generate a random encryption key

• 4- Encrypt the file.

• 5- Upload to IPFS.

• 6- Send the encrypted key to recipient.

1- Get the local file.

Get the file through an component.

<div>
    <h2>Encrypt & compress</h2>
    <input type="file" ref={compressFile} onChange={handleCompress} />
</div>

2- Compress the file.

Here, we use the zstd-codec library, see: https://www.npmjs.com/package/zstd-codec.
Import the zstd-codec library in the code:

import { ZstdCodec } from 'zstd-codec';

Then compress the source file in ZstdCode.run(), which can compress ordinary files to less than 50% of the original file size.

const file = Array.from(e.target.files)[0];
const fileContent = new Uint8Array(await file.arrayBuffer());
ZstdCodec.run(async (zst: any) => {
  // Compress encrypted data, the file will be smaller
  const simple = new zst.Simple();
  const level = 21;
  try {
    const compressedFile = simple.compress(
      fileContent,
      level
    ) as Uint8Array;
    console.log('original', fileContent, fileContent.length);
  } catch (err) {
    console.log(err);
  }
}

Note that whether it is compression or encryption and decryption, it is all based on the file buffer(Uint8Array). All files, including text, images, videos, binary files, etc., are converted to Uint8Array for unified processing.

3- Generate a random encryption key

The following method can generate a key consisting of 128 characters which is very strong, and the number of characters can be customized.

  const generatePassword = () => {
    const len = 128;
    const charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789';
    let retVal = '';
    for (var i = 0, n = charset.length; i < len; ++i) {
      retVal += charset.charAt(Math.floor(Math.random() * n));
    }
    setEncryptionPassword(retVal);
    return retVal;
  };

4- Encrypt the file.

We use the most popular encryption library: CryptoJS (https://www.npmjs.com/package/crypto-js). After comparative analysis, we decided to use Rabbit symmetric encryption algorithm provided by the library instead of the most popular AES algorithm.

const encryptData = (data: Uint8Array, pwd: string): Uint8Array => {
    const wa = convertUint8ArrayToWordArray(data);
    const wordArray = CryptoJS.lib.WordArray.create(wa.words, wa.sigBytes);
    var encrypted = CryptoJS.Rabbit.encrypt(wordArray, pwd);
    return getBufferFromCipherParams(encrypted, data.length);
  };

Encrypt the Uint8Array obtained from step 2 after compressing the file with the encryption key obtained from step 3 as parameters.

Two conversion functions are used in the above method: convertUint8ArrayToWordArray and getBufferFromCipherParams.

The reason for using the first conversion function is that the data processed in the CryptoJS library is all 32-bit WordArray, so it is necessary to convert Uint8Array to WordArray. The specific conversion method is as follows:

export function convertWordArrayToUint8Array(
  wordArray: CryptoJS.lib.WordArray
) {
  var len = wordArray.words.length,
    u8_array = new Uint8Array(len << 2),
    offset = 0,
    word,
    i;
  for (i = 0; i < len; i++) {
    word = wordArray.words[i];
    u8_array[offset++] = word >> 24;
    u8_array[offset++] = (word >> 16) & 0xff;
    u8_array[offset++] = (word >> 8) & 0xff;
    u8_array[offset++] = word & 0xff;
  }

  // delete the latest 0 elements
  const lastWord = wordArray.words[len - 1];
  const lastWordArray = [
    lastWord >> 24,
    (lastWord >> 16) & 0xff,
    (lastWord >> 8) & 0xff,
    lastWord & 0xff
  ];
  let zeros = 0;
  if (
    lastWordArray[0] === 0 &&
    lastWordArray[1] === 0 &&
    lastWordArray[2] === 0 &&
    lastWordArray[3] === 0
  )
    zeros = 4;
  else if (
    lastWordArray[0] !== 0 &&
    lastWordArray[1] === 0 &&
    lastWordArray[2] === 0 &&
    lastWordArray[3] === 0
  )
    zeros = 3;
  else if (
    lastWordArray[0] !== 0 &&
    lastWordArray[1] !== 0 &&
    lastWordArray[2] === 0 &&
    lastWordArray[3] === 0
  )
    zeros = 2;
  else if (
    lastWordArray[0] !== 0 &&
    lastWordArray[1] !== 0 &&
    lastWordArray[2] !== 0 &&
    lastWordArray[3] === 0
  )
    zeros = 1;
  return u8_array.subarray(0, u8_array.length - zeros);
}

There are many examples online that implement this method, but they all overlook a point: when the length of the Uint8Array is not a multiple of 32, it will be automatically padded with 0 to make up the number of bits, and these additional bits will make the file different from the source file. Therefore, the above code will removing the zeros.

The other method getBufferFromCipherParams is used to package the encrypted cipherParams data into a buffer because cipherParams not only includes ciphertext but also includes data such as key, iv, and salt, which are needed for decryption. Therefore, we designed a data structure to store these data.

  const getBufferFromCipherParams = (
    cipherParams: CryptoJS.lib.CipherParams,
    originalLength: number
  ) => {
    const ciphertextU8 = convertWordArrayToUint8Array(cipherParams.ciphertext);
    const keyU8 = convertWordArrayToUint8Array(cipherParams.key); // 32
    const ivU8 = convertWordArrayToUint8Array(cipherParams.iv); // 16
    const saltU8 = convertWordArrayToUint8Array(cipherParams.salt); // 8

    // Format of Uint8Array header + body
    // 0-15: original file length of Uint8Array
    // 16-31: ciphertext length of Uint8Array
    // 32-63: keyU8
    // 64-79: ivU8
    // 80-87: saltU8
    // 88~: ciphertext
    const newBuffer = new Uint8Array(88 + ciphertextU8.length);
    newBuffer.set(numberToBytes(originalLength));
    newBuffer.set(numberToBytes(ciphertextU8.length), 16);
    newBuffer.set(keyU8, 32);
    newBuffer.set(ivU8, 64);
    newBuffer.set(saltU8, 80);
    newBuffer.set(ciphertextU8, 88);
    return newBuffer;
  };

Above, we used the simplest processing method, which is to package key, iv, and salt as is. However, in the official version, more secure optimizations will be made.

5- Upload to IPFS

This step is straightforward, as IPFS has libraries specifically for this operation. However, I prefer to use the web3.storage service, so I won't provide the implementation code for uploading here.

6- Send the key

The sender encrypts the encryption key by recipient's public key and sends the ciphertext to the recipient. Because this ciphertext can only be opened and viewed by recipient's private key, the ciphertext can be sent through an insecure and public channel securely.

Here, the recipient uses the RPC method getEncryptionPublicKey of metamask to obtain their public key, and then gives it to the sender in a public way. The sender uses the recipient's public key for encryption. The code is as follows:

  const getEncryptionPublicKey = async () => {
    const encryptionPublicKey = await window.ethereum.request({
      method: 'eth_getEncryptionPublicKey',
      params: [currentAccount]
    });
    setPublicKey(encryptionPublicKey);
  };
  
  const encryptByPubKey = async () => {
    const encryptData = {
      publicKey,
      data: encryptionPassword,
      version: 'x25519-xsalsa20-poly1305'
    };
    const enc = sigUtil.encrypt(encryptData);
    const buff = stringToUint8Array(JSON.stringify(enc)) as any;
    const encryptedMessage = ethUtil.bufferToHex(buff);
    const compressedEncryptedMessage = hexToString(encryptedMessage);
    setEncryptedPassword(compressedEncryptedMessage);
  };

Manually operating the above operations may be a bit cumbersome, but if it is integrated into a social application with a crypto wallet, it will greatly improve the user experience. In Chatpuppy, we have integrated the above functions, making the transmission of public keys, encryption of keys, and transmission of ciphertext very simple.

Decryption process

Divided into the following steps:

• 1- Download the file.

• 2- Decrypt the file.

• 3- Decompress the file.

1- Download the encrypted file.

Download the encrypted and compressed file from IPFS. As there are ready-made tool libraries available, the code is omitted here.

2- Decrypt the file.

This is the reverse operation of encryptData above. The code is as follows:

  const decryptData = (data: Uint8Array, pwd: string): Uint8Array => {
    const cp = parseCipherParamsFromBuffer(data);
    const cipherParams = CryptoJS.lib.CipherParams.create({
      ciphertext: CryptoJS.lib.WordArray.create(
        cp.ciphertext.words,
        cp.ciphertext.sigBytes
      ),
      key: CryptoJS.lib.WordArray.create(cp.key.words, cp.key.sigBytes),
      iv: CryptoJS.lib.WordArray.create(cp.iv.words, cp.iv.sigBytes),
      salt: CryptoJS.lib.WordArray.create(cp.salt.words, cp.salt.sigBytes)
    });

    var decrypted = CryptoJS.Rabbit.decrypt(cipherParams, pwd);
    const decryptedU8 = convertWordArrayToUint8Array(decrypted);
    const result = decryptedU8.subarray(0, cp.originalLength);
    return result;
  };

Because we need to convert Buffer to cipherParams, there is a method called parseCipherParamsFromBuffer. The specific implementation code is as follows:

  const parseCipherParamsFromBuffer = (buf: Uint8Array) => {
    // decrypt from Uint8Array
    const ciphertext = convertUint8ArrayToWordArray(buf.subarray(88)); // convertUint8ArrayToWordArray(ciphertextU8);
    const originalLength = bytesToNumber(buf.subarray(0, 16));
    const cipherLength = bytesToNumber(buf.subarray(16, 32));
    const key = convertUint8ArrayToWordArray(buf.subarray(32, 64));
    const iv = convertUint8ArrayToWordArray(buf.subarray(64, 80));
    const salt = convertUint8ArrayToWordArray(buf.subarray(80, 88));
    return {
      ciphertext,
      key,
      iv,
      salt,
      originalLength,
      cipherLength
    };
  };

In addition, because we need to convert the decrypted WordArray to Uint8Array, we also need the convertWordArrayToUint8Array method.

3- Decompress the file.

Use the following code to implement decompression and save the decompressed file.

    const decryptedU8 = decryptData(fileContent, encryptionPassword);
    // Decompress
    ZstdCodec.run((zst: any) => {
      const simple = new zst.Simple();
      try {
        const result = simple.decompress(decryptedU8) as Uint8Array;
        if (result !== null) {
          const blob = new Blob([result], {
            type: 'application/octet-stream'
          });
          download(blob, file.name.substring(0, file.name.length - 4));
        } else {
          console.log('Decompress error');
        }
      } catch (err) {
        console.log(err);
      }
    });

The source code for the entire process above can be found at: https://codesandbox.io/s/charming-breeze-v1mh97. (Note: when using codesanbox, due to the memory limitation of the browser by the platform, only files of about 500K can be encrypted and decrypted.)

Some Thoughts on the Ownership Certificate of NFT Data

When discussing the digital economy and data assets, we always cannot avoid a question:

How is data ownership confirmed?

Ownership is a prerequisite for the circulation, use, and processing of data.

Although blockchain technology has become the best tool for confirming data ownership, it can only establish ownership of native blockchain data at present, and cannot achieve trusted ownership of off-chain data.

Here, I proposes an idea, which is to combine encrypted file transmission with NFT (Non-fungible Token), which may provide a way to solve the problem of confirmation of data ownership.

The method is as follows: include in the metadata of the NFT the ciphertext of the file key encrypted by the sender with the recipient's public key, as well as the URL of the encrypted file. That is, only the recipient authorized by the sender can obtain the key for decrypting the file. When the recipient decrypts the file and forwards it to a third party, the third party needs to verify the NFT to confirm whether the recipient has the data’s ownership. At the same time, the earliest sender can also control the copying of the data through the metadata contained in the NFT to prevent data abuse and piracy.

This method shifts the focus of data ownership from the data itself to the NFT, turns the circulation of data into the circulation of NFT, and turns the transaction of data into the transaction of NFT. This not only protects the rights of data owners but also maximizes the value of data in the circulation.

Currently, this is just a simple proposal, and reader who are interested in this are welcome to communicate and discuss it.

This road may be long, but we will continue to explore the application scenarios of combining blockchain technology and digital asset technology, bring better digital experiences to users, and create higher value for data.