ISP Experiment Report
0xC321
August 15th, 2022

Experimental Environment

This experimental environment consists of two types of canisters: ISP and Buckets. There are three canisters in the experiment, including one ISP which is responsible for distributing blob files to subordinate Buckets and two Buckets which is responsible for receiving blob files sent by ISP and storing them in stable memory. According to the order of creation, these buckets are called Bucket#1 and Bucket#2.

Experiment Procedure

Suppose a user continuously transmits files to ISP with a total size of about 12.5GB, each file does not exceed UPLOAD_SIZE_LIMITED (about 2MB). ISP distributes the Blob file to Bucket#1 in the beginning. When the stable memory usage of Bucket#1 exceeds RETIRING_THRESHOLD (about 5.5GB), ISP automatically distributes subsequent files to the newly created Bucket#2.

In the experiment, ISP will transmits 440 epochs of files, and sends 50 blob files in each epoch, so there are 22,000 blob files. Bucket#1 receives the first 234 epochs (11700 blob files), while Bucket#2 receives the next 206 epochs (10300 blob files).

Overview

This section shows the line graphs of the RTS parameters of ISP, Bucket#1 and Bucket#2 during the whole experiment. It is specially explained here that RTS Memory Size, RTS Heap Size and RTS Max Live Size use Kilobyte as the ordinate unit. RTS Total Allocation, RTS Reclaimed and Stable Memory Size use MByte as the ordinate unit.

Among the three line graphs of RTS Memory Size, Heap Size and Max Live Size, Figure ISP has the features:

  • RTS Memory Size will step after a period of several epochs, and the upper limit of RTS Heap Size will also follow the step. As for the reason for the step, we are not clear, and we need to further study the Wasm Memory to get a reasonable guess.
  • The lower limit of RTS Heap Size is approximately equal to RTS Max Live Size, which means that ISP hardly uses Heap Memory (about 2KB) after a GC.

Figures Bucket#1, Bucket#2 have the following differences from Figure ISP:

  • The lower limit of RTS Heap Size is not approximately equal to RTS Max Live Size, which means that Bucket always occupies more Heap Memory, and as the storage process continues to increase, it increases from about 1000KB at the beginning to 4000KB at the end
  • The timing and number of steps of RTS Memory Size of two Buckets do not seem to be correlated with ISP.

The meaning of each parameter in the line graphs below will be explained in the following sections.

Figure 1
Figure 1

Figure 1

Figure 2
Figure 2

Figure 2

Figure 3
Figure 3

Figure 3

Figure 4
Figure 4

Figure 4

Figure 5
Figure 5

Figure 5

Figure 6
Figure 6

Figure 6

Figure 7
Figure 7

Figure 7

Figure 8
Figure 8

Figure 8

RTS Memory Size

RTS Memory Size is the current size of the wasm memory array (mostly the same as the canister memory size). This can only grow, never shrink in Wasm, even if most of it is unused.

In Overview section, we have mentioned some features of RTS Memory Size: RTS Memory Size will step after a period of several epochs, and the timing and number of steps of RTS Memory Size of two Buckets do not seem to be correlated with ISP.

In Figure 10, 12 and 14, there are the line graphs of absolute difference of RTS Memory Size. We can find the following features:

  • RTS Memory Size is a multiple of 64KB because WASM_PAGE_SIZE = 65536
  • RTS Memory Size only increases by one wasm page most of the time
Figure 9
Figure 9

Figure 9

Figure 10
Figure 10

Figure 10

Figure 11
Figure 11

Figure 11

Figure 12
Figure 12

Figure 12

Figure 13
Figure 13

Figure 13

Figure 14
Figure 14

Figure 14

RTS Heap Size

RTS Heap Size contains the actual size of the current Motoko heap, i.e., the heap memory in use, and RTS Max Live Size, which is the largest heap size that has remained so far after a GC.

In Overview section, we have already mentioned some features of RTS Heap Size: when a step occurs in RTS Memory Size, the upper limit of RTS Heap Size will also step up. We have listed the difference between upper and lower limits of ISP and Buckets at each stage.

  • ISP:40KB、1800KB、3500KB、5200KB、7000KB
  • Bucket#1:0.75KB、1175KB、2350KB、3550KB、4700KB
  • Bucket#2:0.75KB、1175KB、2350KB、3550KB
Figure 15
Figure 15

Figure 15

Figure 16
Figure 16

Figure 16

Figure 17
Figure 17

Figure 17

Figure 18
Figure 18

Figure 18

Figure 19
Figure 19

Figure 19

Figure 20
Figure 20

Figure 20

RTS Max Live Size

RTS Max Live Size is the largest heap size that has remained so far after a GC.

The RTS Max Live Size in ISP is roughly around 40KB, while the RTS Max Live Size in Bucket steps with the step of RTS Memory Size, we list the approximate values of each stage of Buckets

  • Bucket#1:700Byte、1700Byte、2600Byte、3700Byte、4700Byte
  • Bucket#2:700Byte、1700Byte、2600Byte、3700Byte

It should be noted that, although the absolute difference of RTS Max Live Size has several steps, the line graph still looks like a straight line. After checking the log data, we find that RTS Max Live Size will increase by one more index interval after one more step. During the interval, RTS Max Live Size won’t increase.

Figure 21
Figure 21

Figure 21

Figure 22
Figure 22

Figure 22

Figure 23
Figure 23

Figure 23

Figure 24
Figure 24

Figure 24

Figure 25
Figure 25

Figure 25

Figure 26
Figure 26

Figure 26

RTS Total Allocation

RTS Total Allocation is the size of accumulative total bytes allocated.

The size of RTS Total Allocation of each epoch of ISP is roughly distributed around 240MB, while the size of RTS Total Allocation of each index of Bucket keeps increasing.

Similar to RTS Max Live Size, although the upper size of RTS Total Allocation of each index of Bucket keeps increasing, the line graph still looks like a straight line. We found the same reason as RTS Max Live Size, RTS Total Allocation will increase by one more index interval after one more step. RTS Total Allocation does not allocate any bytes in the interval, i.e., the size of allocated bytes is 0.

Figure 27
Figure 27

Figure 27

Figure 28
Figure 28

Figure 28

Figure 29
Figure 29

Figure 29

Figure 30
Figure 30

Figure 30

Figure 31
Figure 31

Figure 31

Figure 32
Figure 32

Figure 32

RTS Reclaimed

RTS Reclaimed is the size of accumulative total bytes collected.

The size of RTS Reclaimed of each epoch of ISP is roughly distributed around 80MB, while the size of RTS Reclaimed in Bucket steps with the step of RTS Memory Size,

Similar to RTS Max Live Size, although the upper size of RTS Reclaimed of each index of Bucket has several steps, the line graph still looks like a straight line. We found the same reason as RTS Max Live Size, RTS Reclaimed will increase by one more index interval after one more step. RTS Reclaimed does not collect any bytes in the interval, i.e., the size of collected bytes is 0.

Figure 33
Figure 33

Figure 33

Figure 34
Figure 34

Figure 34

Figure 35
Figure 35

Figure 35

Figure 36
Figure 36

Figure 36

Figure 37
Figure 37

Figure 37

Figure 38
Figure 38

Figure 38

Stable Memory Size

Since Bucket is responsible for actually writing data, Stable Memory Size is only related to Bucket.

During the experiment, the stable memory writing speed of Bucket remained constant, each index was written to 597,324 Bytes.

Figure 39
Figure 39

Figure 39

Figure 40
Figure 40

Figure 40

Figure 41
Figure 41

Figure 41

Figure 42
Figure 42

Figure 42

Conclusion

Based on the analysis of the above line graphs, it can be concluded that the growth of Canister's RTS parameters are predictable, which has guiding significance for us to design a new generation of ISP architecture in the future.

In addition, it can be intuitively seen from the line graphs that the status of Bucket's continuous writing data to Stable Memory is robust and stable, which provides confidence for ISP to support larger-scale business services.

Reference

  1. Motoko, Array, Memory - rossberg

    Motoko, Array, Memory

    As for the rts values: memory_size is the current size of the Wasm memory array (mostly the same as the canister memory size). This can only grow, never shrink in Wasm, even if most of it is unused.

    The other values come from here 19. The two interesting ones are probably heap_size, which contains the actual size of the current Motoko heap, i.e., the heap memory in use, and max_live_size, which is the largest heap size that has remained so far after a GC. The other numbers are accumulative total bytes allocated and collected so far, which probably is less interesting.

  2. motoko/rts/motoko-rts/src/memory/ic.rs

    https://github.com/dfinity/motoko/blob/master/rts/motoko-rts/src/memory/ic.rs

    /// Maximum live data retained in a GC. pub(crate) static mut MAX_LIVE: Bytes<u32> = Bytes(0);

    /// Amount of garbage collected so far. pub(crate) static mut RECLAIMED: Bytes<u64> = Bytes(0);

    /// Counter for total allocations pub(crate) static mut ALLOCATED: Bytes<u64> = Bytes(0);

    /// Heap pointer pub(crate) static mut HP: u32 = 0;

    /// Heap pointer after last GC pub(crate) static mut LAST_HP: u32 = 0;

    // Provided by generated code extern "C" { pub(crate) fn get_heap_base() -> u32; pub(crate) fn get_static_roots() -> Value; }

    pub(crate) unsafe fn get_aligned_heap_base() -> u32 { // align to 32 bytes ((get_heap_base() + 31) / 32) * 32 }

    #[no_mangle] unsafe extern "C" fn init(align: bool) { HP = if align { get_aligned_heap_base() } else { get_heap_base() }; LAST_HP = HP; }

    #[no_mangle] unsafe extern "C" fn get_heap_size() -> Bytes<u32> { Bytes(HP - get_aligned_heap_base()) }

    /// Provides a Memory implementation, to be used in functions compiled for IC or WASI. /// The Memory implementation allocates in Wasm heap with Wasm memory.grow instruction. pub struct IcMemory;

    impl Memory for IcMemory { #[inline] unsafe fn alloc_words(&mut self, n: Words<u32>) -> Value { let bytes = n.to_bytes(); // Update ALLOCATED let delta = u64::from(bytes.as_u32()); ALLOCATED += Bytes(delta);

    // Update heap pointer

    let old_hp = u64::from(HP);

    let new_hp = old_hp + delta;

    // Grow memory if needed

    grow_memory(new_hp);

    debug_assert!(new_hp <= u64::from(core::u32::MAX));

    HP = new_hp as u32;

    Value::from_ptr(old_hp as usize)

    }

    }

    /// Page allocation. Ensures that the memory up to, but excluding, the given pointer is allocated. #[inline(never)] unsafe fn grow_memory(ptr: u64) { debug_assert!(ptr <= 2 * u64::from(core::u32::MAX)); let page_size = u64::from(WASM_PAGE_SIZE.as_u32()); let total_pages_needed = ((ptr + page_size - 1) / page_size) as usize; let current_pages = wasm32::memory_size(0); if total_pages_needed > current_pages { if wasm32::memory_grow(0, total_pages_needed - current_pages) == core::usize::MAX { rts_trap_with("Cannot grow memory"); } } }

Team

Mixlabs is a leading-edge technology laboratory in the blockchain direction composed of top universities and community developers in Asia. It is mainly engaged in cutting-edge technology research, incubation and ecological support around Web3.

Website Twitter Discord Gitbook

Subscribe to MixDAO
Receive new entries directly to your inbox.
Collectors
View
#1
#2
#3
View collectors
This entry has been permanently stored on-chain and signed by its creator.