Review Board

beta

Add fancy caching for large blocks of data

Updated 2 months, 3 weeks ago

David Trowbridge Reviewers
trunk reviewboard
None Navi
memcached will only cache chunks of data below 1M. This is kind of crappy, since some of our files can be larger than 1M, and those are the ones which take the longest to fetch, patch, diff, and render.

This change adds a large_data keyword to cache_memoize, which will do some fancy pickling, compression and splitting to stuff these large blocks into the cache.
Looked at my giant diff and saw that things were getting cached correctly.
Posted 2 months, 3 weeks ago (October 14th, 2008, 4:28 p.m.)

   

  
Loading diff fragment...
Can you add a small comment documenting the resulting size?
  1. David Trowbridge 2 months, 3 weeks ago (October 14th, 2008, 4:37 p.m.)
    Yup. I realized that with the overhead of the list and pickling, we need to chunk smaller, so I added -1024.
Loading diff fragment...
Can you space this out a little, put blank lines before/after each block?

Also, we may want to use the multi-key get capabilities to get all the data in one go, since we know all the keys up-front.
  1. David Trowbridge 2 months, 3 weeks ago (October 14th, 2008, 4:37 p.m.)
    Sure.
Loading diff fragment...
Can combine these.
  1. David Trowbridge 2 months, 3 weeks ago (October 14th, 2008, 4:36 p.m.)
    Okay.
Loading diff fragment...
Can you add a comment describing the format of the data we're storing and how we piece it all together?
  1. David Trowbridge 2 months, 3 weeks ago (October 14th, 2008, 4:37 p.m.)
    Sure.
Loading diff fragment...
[data, ] should just be [data].
  1. David Trowbridge 2 months, 3 weeks ago (October 14th, 2008, 4:37 p.m.)
    Okay.
Loading diff fragment...
No need for pass.
Ship it!
Posted 2 months, 3 weeks ago (October 14th, 2008, 5:52 p.m.)
Looks good! Awesome change.