Memory and Performance

Handling an email costs memory, and how much depends on how large the email is. The gateway is configured out of the box for the memory of a standard installation and for the default maximum message size. If you raise the maximum message size, give the machine less memory, or enable extra services, the settings described in this chapter have to be reviewed together, because they all draw on the same memory.

The symptom of getting this wrong is an OutOfMemoryError in the back-end log under load, usually when several large messages are being processed at the same time.

No email is lost when this happens. The message is retried a number of times, which often succeeds on its own because the memory pressure is temporary. If it keeps failing, the message is stored in the error spool, which is shown as MPA Error on the Dashboard. Once you have addressed the cause, you can respool the message from there and it is processed again.

How the memory is divided

The gateway runs up to three Java back-ends, each of which gets a percentage of the total memory of the machine for its heap:

Setting	Default	Service
ciphermail_backend__max_ram_percentage	40	The gateway back-end, which processes email
ciphermail_portal_backend__max_ram_percentage	20	The portal for external recipients
ciphermail_company_portal_backend__max_ram_percentage	8	The Company Portal , disabled by default

These percentages come out of the same memory, so they should be considered together. With the defaults they add up to 60 percent, or 68 percent when the Company Portal is enabled. The remainder is not spare capacity: it is needed for the memory the Java processes use outside their heap (metaspace, code cache and thread stacks), and for MariaDB, Postfix, Dovecot, the web server and the operating system itself.

Warning

Raising one of these percentages takes memory away from everything else. If you raise one, lower another, or give the machine more memory.

Disable the portals you do not use

If you do not use a portal, disable it. The service is still installed and configured, but it is not started, and the memory it would have reserved stays available for the back-end that processes email:

---
# disable the portal for external recipients
ciphermail_portal_backend__service_enabled: false

# disable the Company Portal
ciphermail_company_portal_backend__service_enabled: false

The Company Portal is already disabled by default. The portal for external recipients is enabled by default, because it is required for PDF password retrieval, Webmail Messenger and the recipient’s “forgot password” flow. Only disable it if you use none of those.

Note

Disabling a service does not by itself hand its share to another back-end. After disabling one, you can raise ciphermail_backend__max_ram_percentage by roughly the percentage you freed.

The number of spool threads

ciphermail_backend__james_spool_threads sets how many emails go through the gateway at the same time. It defaults to 5.

This is the most important setting for memory, because the memory the back-end needs is the number of spool threads times what one message costs, and what one message costs grows with the size of that message. Ten threads do not use slightly more memory than five: they use twice as much.

The heaviest thing the gateway does to a message is converting it to an encrypted PDF, which needs about 2.4 times the size of the message. So the memory the back-end needs at worst is:

spool threads x 2.4 x maximum message size

Which gives the largest number of threads that fits in the heap:

spool threads = heap / (2.4 x maximum message size)

Important

The 2.4 is the point at which it runs out of memory, not a safe setting. Leave margin: use about two thirds of the number this gives you.

Memory is not the only limit. The work is mostly processor bound, so on a machine with 4 vCPUs the throughput stops improving somewhere around 5 to 8 threads, and anything above that only costs memory and adds latency. A thread waiting for the database, for LDAP or for a CRL download does hold a slot without using a processor, so somewhat more threads than processors is still reasonable. In practice, use the lower of what the processors and the memory allow.

On the Enterprise Gateway there is a third limit. PDF encryption runs a headless browser outside the heap, one per spool thread, and on a small machine that is what decides the number of threads rather than the heap or the processors. See PDF encryption .

A gateway which does DLP scans every message inside the heap as well, which is about 20 MB per spool thread on top of the calculation above. See DLP scanning.

Note

Above 10 spool threads, also raise ciphermail.hibernate.hikari.maximum-pool-size, which defaults to 10 and otherwise becomes the limit instead, and raise max_connections of the database with it.

Why the calculation is worst case

The calculation assumes that every spool thread is handling a message of the maximum size at the same moment. Normal mail is nowhere near that: most email is a few kilobytes, so in day to day use the gateway uses a small fraction of what the calculation reserves.

It is still the number to size for, because large messages tend to arrive together rather than spread out:

• A bulk send with a large attachment reaches many recipients at once.

• Scheduled reports, exports and backups are sent at the same time every day, are large, and often go to several people.

• One large message addressed to several recipients can be split into several messages when those recipients need different handling, for example when one recipient’s mail is encrypted and another’s is not. Those are then processed as separate messages, so a single incoming email can occupy more than one spool thread, each with its own copy of the message.

Sizing for the worst case is also cheap. Spool threads above what the processors can use add very little throughput, so lowering the number when the maximum message size is large costs almost nothing in practice, while getting it wrong in the other direction means messages land in the error spool and have to be respooled by hand.

The maximum message size

The maximum message size is the “Before filter message size limit” in the MTA settings of the administration console, and it defaults to 10 MB. It is the other half of the calculation.

The default of 10 MB and the default of 5 spool threads fit comfortably on the smallest supported machine. Raising the message size limit is what usually causes memory problems, because the cost per thread rises with it while the number of threads stays where it was.

Warning

If you raise the maximum message size, lower the number of spool threads with it, or give the machine more memory.

Examples

The heap of the back-end is 40 percent of the memory of the machine by default. The table below shows the largest number of spool threads that fits, and a recommended value which leaves margin and takes the processors into account.

Memory	Back-end heap	Maximum message size	Threads that fit	Recommended
2 GB	819 MB	10 MB (default)	34	5 (the default, processor bound)
2 GB	819 MB	50 MB	6	4
2 GB	819 MB	100 MB	3	2
4 GB	1638 MB	10 MB (default)	68	5 to 8
4 GB	1638 MB	50 MB	13	8
4 GB	1638 MB	100 MB	6	4
8 GB	3277 MB	100 MB	13	8
16 GB	6554 MB	10 MB (default)	273	8 to 16 (processor bound)
16 GB	6554 MB	100 MB	27	8 to 16 (processor bound)
16 GB	6554 MB	200 MB	13	8

The pattern to take from this is that with the default message size limit the processors are the limit and the default of 5 threads is fine on any supported machine, and that once the message size limit reaches tens of megabytes the memory becomes the limit and the number of threads has to come down. On a large machine the memory stops being the constraint again: at 16 GB even a 100 MB message size limit leaves far more room than the processors can use, so the number of vCPUs decides.

Note

Where the recommendation above is higher than 10, remember to raise ciphermail.hibernate.hikari.maximum-pool-size and max_connections of the database with it, as described in The number of spool threads.

Important

This table is about the heap. On the Enterprise Gateway, PDF encryption needs memory outside the heap as well, and on a small machine that is the stricter of the two: a 2 GB gateway which encrypts to PDF should run 2 spool threads and not the 5 this table allows. See PDF encryption .

DLP scanning

A gateway with DLP rules reads every message a second time: the text of every part is extracted and every rule is matched against it. That happens inside the back-end heap, so it comes out of the same budget as everything else in this chapter.

Measured on a 2 GB gateway with 5 spool threads, all of them scanning a 10 MB message at the same time, the whole scan needs about 90 MB, which is under 20 MB per thread. That is the number whatever the message is made of: one large attachment, 64 medium parts or 2000 small ones all come out the same, and so do a message of nothing but text and one of eight nested attached messages.

• A part larger than dlp-text-extractor-mem-disk-threshold, 1 MB by default, is written to a temporary file rather than kept in memory.

• The rules are matched over the message in chunks of 512 kB with a small overlap, so a rule never sees a whole message at once and a large message does not cost more per rule than a small one.

Two limits are there for a message which is built to be expensive, and neither is worth changing: dlp-text-extractor-max-part-size, 20 MB, and a maximum of 64 sub-documents per document.

So on a gateway which does DLP, add something in the order of 20 MB per spool thread to the calculation. On the smallest supported machine, where the back-end heap is 819 MB, five threads scanning at once are about a tenth of it.

Note

A rule which backtracks catastrophically costs processor rather than memory, but it holds a spool thread for as long as it runs. If messages stop moving and the machine is busy without the queue draining, look at what was last added to the rules before looking at the memory.

PDF encryption

The Enterprise Gateway renders a message to PDF with chrome-headless-shell, which runs as a set of processes of its own, eight to twelve per conversion. Its memory is not part of the back-end heap and is therefore not covered by the calculation above.

A conversion needs between 70 and 175 MB. That is a fixed cost: it does not grow with the size of the message, because only the message body and the list of attachments are rendered, never the attachments themselves. The body itself is limited to 1 MB before it is rendered.

Three settings decide what the conversions may use, and they work together:

ciphermail_backend__pdf_conversion_memory_max_mb

What one conversion may use, 384 MB by default. That is more than double what the heaviest ordinary message needed, and it is there for the messages which are not ordinary: a body of a megabyte of table cells was measured taking 1.8 GB, and 56 KB of overlapping shadows took 934 MB. Both are inside what the body limit allows, both come through the sanitizer which the body is put through first, and neither is refused by chrome. A conversion which reaches the limit produces no PDF at all rather than half of one, and the message goes to the error spool where it can be respooled.

cm-pdf.slice

What all the conversions together may use. It is set for you as the number of spool threads times the limit above, and it is not something to tune downwards.

ciphermail_backend__pdf_conversion_memory_swap_max

Whether reaching the limit stops a conversion or only slows it down. It is 0 by default, which means no swap and therefore a conversion which stops. Without it the limit is not a limit: the megabyte of table cells above still produced its PDF, in 8.3 seconds instead of 4.6. Set it to infinity if you would rather every message be converted and delivered, at the cost of a message which cannot be converted cheaply holding a spool thread for as long as it takes.

Warning

A slice smaller than the number of spool threads times the per conversion limit does not protect the machine, it stops it working. The conversions are each given less memory than they need, chrome answers memory pressure by working slower rather than by failing, none of them finishes inside the 30 second converter timeout, all of them are abandoned, and the next ones start on top of the leftovers.

A gateway with 2 GB of memory, 5 spool threads and a 392 MB slice was measured converting nothing at all: twenty conversions in a row hit the timeout, no PDF was produced, the queue grew and the load average reached 73. The same gateway with 2 spool threads converted every message in 190 milliseconds.

So on a machine which cannot spare the number of spool threads times 384 MB, lower the number of spool threads. Fewer conversions at a time is the only setting which sheds load; a memory limit cannot.

How many spool threads PDF encryption allows

Subtract both heaps from the memory of the machine, and divide what is left by the 384 MB one conversion may claim. Round down, and then take the lower of that and the number of processors:

Memory	Back-end heap	Portal heap	Left for conversions	Spool threads	Slice
2 GB	819 MB	410 MB	819 MB	2	768 MB
4 GB	1638 MB	819 MB	1639 MB	4	1536 MB
8 GB	3277 MB	1638 MB	3277 MB	8	3072 MB
16 GB	6554 MB	3277 MB	6553 MB	16	6144 MB

Where this is lower than the number of threads the heap allows, this is the number to use. A 2 GB gateway which encrypts to PDF should run 2 spool threads, not the default of 5. If you have disabled a portal, add its share back before dividing.

Nothing in this table is set aside for the operating system, the database or the MTA, and it does not need to be. It sizes the ceiling rather than the traffic: ordinary mail uses around 170 MB a conversion, so the slice of a correctly sized gateway sits about half empty and everything else runs in the room that leaves. The full slice is only claimed when the gateway is sent something expensive, which is the case it exists for and therefore the case to size for.

Note

The processors are what stops the right-hand column growing on a large machine. Above about one conversion per processor the conversions slow each other down: on an 8 core gateway running 10 spool threads a conversion took 450 milliseconds, against 385 milliseconds for 5 threads on 4 cores. Twice the machine, and each conversion slower. Memory decides on a small gateway, the processors on a large one.

What to expect of the throughput

Once the memory is right, the limit is the processor. Measured with small messages, every one of them PDF encrypted and every result opened and checked:

Machine	Spool threads	Messages a second
4 cores	5	7.1
8 cores	5	11.2
8 cores	10	10.2

The 4 core gateway delivered about 28 messages a second when nothing had to be encrypted, so PDF encryption makes a message roughly four times as expensive. It also makes it about thirty times larger.

Read the table by what it does not show. Doubling the processors bought about half as much again, not double. Doubling the spool threads on top of that bought nothing at all: the throughput fell by about a tenth and the round trip times did not improve either, because the conversions then spend their time waiting for each other rather than converting. A conversion which takes 385 milliseconds with five running at once takes 450 with ten, and ten of them do not make up for it.

The reason is that a conversion is expensive in a way more machine does not fix. It costs roughly a quarter of a second of processor time when nothing else is running, and about twice that when several run at once. Under half of it is the rendering; the rest is the sandbox each conversion is started in, which is paid per message and does not get cheaper in bulk.

The rendering is however all of the PDF encryption proper. Building the PDF took 390 milliseconds of which the conversion was 384, so embedding the attachments and encrypting the result cost about 6 milliseconds.

So: adding memory does not make PDF encryption faster. Adding spool threads past the point in How many spool threads PDF encryption allows does not either, and costs latency. Adding processors helps, but expect rather less than the number of processors suggests.

Note

There is no setting which makes one gateway substantially faster at PDF encryption. The cost is the rendering, it is paid per message, and it is already spread across every processor the machine has. If you need more throughput than one machine gives you, add machines: see the Cluster Administration Guide.

Diagnosing a memory problem

If the back-end reports an OutOfMemoryError:

1. Check the maximum message size in the MTA settings and the number of spool threads against the Examples table. These two together explain most cases.

2. Check whether all three back-ends are running and whether you need all of them. See Disable the portals you do not use.

3. Check whether the machine meets the requirements, and remember that other software on the same machine, especially the database, needs memory too.

4. If the numbers look right and it still happens, please contact support with the back-end log and the garbage collector log.

A gateway which is short of memory for PDF encryption does not report an OutOfMemoryError at all. The symptoms are different: the back-end logs Timeout running for the converter, the spool grows, the load average climbs well past the number of processors, and messages arrive in the error spool without ever having been converted. Check the number of spool threads against PDF encryption before looking anywhere else, because too many threads is the usual cause and lowering them is the fix.

Messages which ended up in the error spool while you were investigating can be respooled once the setting has been corrected and the back-end has restarted. Check the error spool after a memory problem, because a message which was retried until it failed is waiting there rather than being delivered.