ISP Experiment Report
MixLabs
0x1F89
August 15th, 2022

Experimental Environment

This experimental environment consists of two types of canisters: ISP and Buckets. There are three canisters in the experiment, including one ISP which is responsible for distributing blob files to subordinate Buckets and two Buckets which is responsible for receiving blob files sent by ISP and storing them in stable memory. According to the order of creation, these buckets are called Bucket#1 and Bucket#2.

Experiment Procedure

Suppose a user continuously transmits files to ISP with a total size of about 12.5GB, each file does not exceed UPLOAD_SIZE_LIMITED (about 2MB). ISP distributes the Blob file to Bucket#1 in the beginning. When the stable memory usage of Bucket#1 exceeds RETIRING_THRESHOLD (about 5.5GB), ISP automatically distributes subsequent files to the newly created Bucket#2.

In the experiment, ISP will transmits 440 epochs of files, and sends 50 blob files in each epoch, so there are 22,000 blob files. Bucket#1 receives the first 234 epochs (11700 blob files), while Bucket#2 receives the next 206 epochs (10300 blob files).

Overview

This section shows the line graphs of the RTS parameters of ISP, Bucket#1 and Bucket#2 during the whole experiment. It is specially explained here that RTS Memory Size, RTS Heap Size and RTS Max Live Size use Kilobyte as the ordinate unit. RTS Total Allocation, RTS Reclaimed and Stable Memory Size use MByte as the ordinate unit.

Among the three line graphs of RTS Memory Size, Heap Size and Max Live Size, Figure ISP has the features:

  • RTS Memory Size will step after a period of several epochs, and the upper limit of RTS Heap Size will also follow the step. As for the reason for the step, we are not clear, and we need to further study the Wasm Memory to get a reasonable guess.
  • The lower limit of RTS Heap Size is approximately equal to RTS Max Live Size, which means that ISP hardly uses Heap Memory (about 2KB) after a GC.

Figures Bucket#1, Bucket#2 have the following differences from Figure ISP:

  • The lower limit of RTS Heap Size is not approximately equal to RTS Max Live Size, which means that Bucket always occupies more Heap Memory, and as the storage process continues to increase, it increases from about 1000KB at the beginning to 4000KB at the end
  • The timing and number of steps of RTS Memory Size of two Buckets do not seem to be correlated with ISP.

The meaning of each parameter in the line graphs below will be explained in the following sections.

Figure 1
Figure 1
Figure 2
Figure 2
Figure 3
Figure 3
Figure 4
Figure 4
Figure 5
Figure 5
Figure 6
Figure 6
Figure 7
Figure 7
Figure 8
Figure 8

RTS Memory Size

RTS Memory Size is the current size of the wasm memory array (mostly the same as the canister memory size). This can only grow, never shrink in Wasm, even if most of it is unused.

In Overview section, we have mentioned some features of RTS Memory Size: RTS Memory Size will step after a period of several epochs, and the timing and number of steps of RTS Memory Size of two Buckets do not seem to be correlated with ISP.

In Figure 10, 12 and 14, there are the line graphs of absolute difference of RTS Memory Size. We can find the following features:

  • RTS Memory Size is a multiple of 64KB because WASM_PAGE_SIZE = 65536
  • RTS Memory Size only increases by one wasm page most of the time
Figure 9
Figure 9
Figure 10
Figure 10
Figure 11
Figure 11
Figure 12
Figure 12
Figure 13
Figure 13
Figure 14
Figure 14

RTS Heap Size

RTS Heap Size contains the actual size of the current Motoko heap, i.e., the heap memory in use, and RTS Max Live Size, which is the largest heap size that has remained so far after a GC.

In Overview section, we have already mentioned some features of RTS Heap Size: when a step occurs in RTS Memory Size, the upper limit of RTS Heap Size will also step up. We have listed the difference between upper and lower limits of ISP and Buckets at each stage.

  • ISP：40KB、1800KB、3500KB、5200KB、7000KB
  • Bucket#1：0.75KB、1175KB、2350KB、3550KB、4700KB
  • Bucket#2：0.75KB、1175KB、2350KB、3550KB
Figure 15
Figure 15
Figure 16
Figure 16
Figure 17
Figure 17
Figure 18
Figure 18
Figure 19
Figure 19
Figure 20
Figure 20

RTS Max Live Size

RTS Max Live Size is the largest heap size that has remained so far after a GC.

The RTS Max Live Size in ISP is roughly around 40KB, while the RTS Max Live Size in Bucket steps with the step of RTS Memory Size, we list the approximate values of each stage of Buckets

  • Bucket#1：700Byte、1700Byte、2600Byte、3700Byte、4700Byte
  • Bucket#2：700Byte、1700Byte、2600Byte、3700Byte

It should be noted that, although the absolute difference of RTS Max Live Size has several steps, the line graph still looks like a straight line. After checking the log data, we find that RTS Max Live Size will increase by one more index interval after one more step. During the interval, RTS Max Live Size won’t increase.

Figure 21
Figure 21
Figure 22
Figure 22
Figure 23
Figure 23
Figure 24
Figure 24
Figure 25
Figure 25
Figure 26
Figure 26

RTS Total Allocation

RTS Total Allocation is the size of accumulative total bytes allocated.

The size of RTS Total Allocation of each epoch of ISP is roughly distributed around 240MB, while the size of RTS Total Allocation of each index of Bucket keeps increasing.

Similar to RTS Max Live Size, although the upper size of RTS Total Allocation of each index of Bucket keeps increasing, the line graph still looks like a straight line. We found the same reason as RTS Max Live Size, RTS Total Allocation will increase by one more index interval after one more step. RTS Total Allocation does not allocate any bytes in the interval, i.e., the size of allocated bytes is 0.

Figure 27
Figure 27
Figure 28
Figure 28
Figure 29
Figure 29
Figure 30
Figure 30
Figure 31
Figure 31
Figure 32
Figure 32

RTS Reclaimed

RTS Reclaimed is the size of accumulative total bytes collected.

The size of RTS Reclaimed of each epoch of ISP is roughly distributed around 80MB, while the size of RTS Reclaimed in Bucket steps with the step of RTS Memory Size,

Similar to RTS Max Live Size, although the upper size of RTS Reclaimed of each index of Bucket has several steps, the line graph still looks like a straight line. We found the same reason as RTS Max Live Size, RTS Reclaimed will increase by one more index interval after one more step. RTS Reclaimed does not collect any bytes in the interval, i.e., the size of collected bytes is 0.

Figure 33
Figure 33
Figure 34
Figure 34
Figure 35
Figure 35
Figure 36
Figure 36
Figure 37
Figure 37
Figure 38
Figure 38

Stable Memory Size

Since Bucket is responsible for actually writing data, Stable Memory Size is only related to Bucket.

During the experiment, the stable memory writing speed of Bucket remained constant, each index was written to 597,324 Bytes.

Figure 39
Figure 39
Figure 40
Figure 40
Figure 41
Figure 41
Figure 42
Figure 42

Conclusion

Based on the analysis of the above line graphs, it can be concluded that the growth of Canister's RTS parameters are predictable, which has guiding significance for us to design a new generation of ISP architecture in the future.

In addition, it can be intuitively seen from the line graphs that the status of Bucket's continuous writing data to Stable Memory is robust and stable, which provides confidence for ISP to support larger-scale business services.

Reference

  1. Motoko, Array, Memory - rossberg

    Motoko, Array, Memory

    As for the rts values: memory_size is the current size of the Wasm memory array (mostly the same as the canister memory size). This can only grow, never shrink in Wasm, even if most of it is unused.

    The other values come from here 19. The two interesting ones are probably heap_size, which contains the actual size of the current Motoko heap, i.e., the heap memory in use, and max_live_size, which is the largest heap size that has remained so far after a GC. The other numbers are accumulative total bytes allocated and collected so far, which probably is less interesting.

  2. motoko/rts/motoko-rts/src/memory/ic.rs

    https://github.com/dfinity/motoko/blob/master/rts/motoko-rts/src/memory/ic.rs

    /// Maximum live data retained in a GC. pub(crate) static mut MAX_LIVE: Bytes<u32> = Bytes(0);

    /// Amount of garbage collected so far. pub(crate) static mut RECLAIMED: Bytes<u64> = Bytes(0);

    /// Counter for total allocations pub(crate) static mut ALLOCATED: Bytes<u64> = Bytes(0);

    /// Heap pointer pub(crate) static mut HP: u32 = 0;

    /// Heap pointer after last GC pub(crate) static mut LAST_HP: u32 = 0;

    // Provided by generated code extern "C" { pub(crate) fn get_heap_base() -> u32; pub(crate) fn get_static_roots() -> Value; }

    pub(crate) unsafe fn get_aligned_heap_base() -> u32 { // align to 32 bytes ((get_heap_base() + 31) / 32) * 32 }

    #[no_mangle] unsafe extern "C" fn init(align: bool) { HP = if align { get_aligned_heap_base() } else { get_heap_base() }; LAST_HP = HP; }

    #[no_mangle] unsafe extern "C" fn get_heap_size() -> Bytes<u32> { Bytes(HP - get_aligned_heap_base()) }

    /// Provides a Memory implementation, to be used in functions compiled for IC or WASI. /// The Memory implementation allocates in Wasm heap with Wasm memory.grow instruction. pub struct IcMemory;

    impl Memory for IcMemory { #[inline] unsafe fn alloc_words(&mut self, n: Words<u32>) -> Value { let bytes = n.to_bytes(); // Update ALLOCATED let delta = u64::from(bytes.as_u32()); ALLOCATED += Bytes(delta);

    // Update heap pointer

    let old_hp = u64::from(HP);

    let new_hp = old_hp + delta;

    // Grow memory if needed

    grow_memory(new_hp);

    debug_assert!(new_hp <= u64::from(core::u32::MAX));

    HP = new_hp as u32;

    Value::from_ptr(old_hp as usize)

    }

    }

    /// Page allocation. Ensures that the memory up to, but excluding, the given pointer is allocated. #[inline(never)] unsafe fn grow_memory(ptr: u64) { debug_assert!(ptr <= 2 * u64::from(core::u32::MAX)); let page_size = u64::from(WASM_PAGE_SIZE.as_u32()); let total_pages_needed = ((ptr + page_size - 1) / page_size) as usize; let current_pages = wasm32::memory_size(0); if total_pages_needed > current_pages { if wasm32::memory_grow(0, total_pages_needed - current_pages) == core::usize::MAX { rts_trap_with("Cannot grow memory"); } } }

Team

Mixlabs is a leading-edge technology laboratory in the blockchain direction composed of top universities and community developers in Asia. It is mainly engaged in cutting-edge technology research, incubation and ecological support around Web3.

Website Twitter Discord Gitbook

Subscribe to MixLabs
Receive the latest updates directly to your inbox.
Mint this entry as an NFT to add it to your collection.
Verification
This entry has been permanently stored onchain and signed by its creator.
Arweave Transaction
28C1ZBb0gUXv0XK…xqFKro57hRdItJ8
Author Address
0x1F89AfC31f531f7…619A8b86353a0B6
Content Digest
Wpdvh4nahn0rIrs…7YPuXUa8xwOZWPE