What’s new in Coggle recently? Here’s your occasional update in what we’ve been working on!
We love green, but, we finally decided that the old homepage was, perhaps, just a tiny bit too green, so you might have noticed our refreshed home page and documents list:
Hope you like the new design - if you have any comments we love to hear them!
There have been some much less visible updates too - in particular, we’ve been working on improving the quality of the diagrams you find when you search for public diagrams on Coggle, by removing and hiding published diagrams that have been created for spam, and that don’t contain any useful content.
We’re not really sure why anyone bothers to try create spam on Coggle, because we’ve always had a number of ways in which we’ve suppressed it, and limited the ways that Coggle diagrams can be used to boost other sites.
For some reason though, people still try (and try quite a lot!), which can cause problems if these diagrams appear in search results instead of other useful content. So, we’ve been improving our detection, flagging, and removal of spammy diagrams, and the accounts that create them, with an even more robust system.
Hopefully you will notice an increase in the quality of public diagrams you find when searching, but if you ever come across public content which shouldn’t be on Coggle, then please email us at hello@coggle.it with a link to the page, so we can review it.
Of course as always there have been lots of other little fixes and improvements too, to make using Coggle just a little bit smoother, like improvements to the ways that buttons for adding branches in diagrams work, and improvements to text rendering in Safari.
That’s all for now! As always, if you have any comments or feedback we always appreciate them, just drop us an email at hello@coggle.it.
Today we’ve got another update from the tech behind Coggle: how we cut the average response time by over 40% with some fairly simple changes, and learned a lesson in checking default configurations.
First, a bit of architecture. Coggle is divided into several separate services behind the scenes, with each service responsible for different things. One service is responsible for storing and accessing documents, another for sending email, one for generating downloads, and so on.
These services talk to each other internally with HTTP requests: so for each request from your browser for a page there will be several more requests between these services before a response is sent back to your browser.
This all adds up to quite a lot of HTTP requests - and many of these Coggle services call out to further services hosted by AWS, using (yep you guessed it!) even more HTTP requests.
So, in all, an awful lot of HTTP requests are going on.
Coggle is written in node.js, and originally we just used the default settings of the node request module, and the AWS SDK for node for most of these requests. (At this point there are better options than the request module - we’d recommend undici for new development - but there isn’t a practical alternative to the AWS SDK.)
Why does this matter? Well, it turns out both of these default configurations are absolutely not tuned for high-throughput applications…
The Investigation Begins
A few weeks ago I came across this interesting interactive debugging puzzle by @b0rk - now, no spoilers here (go try it for yourself!), but when I finally got to the solution it did make me immediately wonder if the same issue was present in Coggle - as for a long time our average response time for requests has been about 60ms:
It didn’t take long to confirm that the problem in the puzzle was not occurring for us, but this made me wonder why exactly our average response-time graph was so consistently high - was there room for any improvement? Are all those requests between the different services slowing things down?
What About the Database?
The first obvious place to check is the database. While the vast majority of requests are very fast, we have some occasionally slower requests. Could these be holding things up due to slow trains? Tweaking the connection pool size options of the mongodb driver showed a small improvement, and this is definitely a default configuration that you should tune to your application rather than leaving as-is (note maxPoolSize, not poolSize, is the option that should be used for unified topology connections).
No dramatic improvements here though.
All Those Internal Requests…
Like the mongodb driver, nodejs itself also maintains a global connection pool (in this case an http.Agent) for outgoing connections. If you search for information about this connection pool you will find lots articles saying that it’s limited to 5 concurrent connections. Ahha! This could easily be causing requests to back-up.
Inter-service requests are generally slower than database requests, and just five slow requests could cause others to start piling up behind them!
Fortunately, all those articles are very out of date. The global nodejs connection pool has been unlimited in size since nodejs 0.12 in 2015. But this line of investigation does lead directly to the true culprit.
The global http Agent which our internal requests were using is constructed using default options. And a careful reading of the http agent documentation shows that the keepAlive option is false by default.
This means, simply, that after a request is complete nodejs will close the connection to the remote server, instead of keeping the connection in case another request is made to the same server within a short time period.
In Coggle, where we have a small number of component services making a large number of requests to each other, it should almost always be possible to re-use connections for additional requests. Instead, with the default configuration, a new connection was being created for every single request!
A Solution!
It is not possible to change the global default value, so to configure the request module to use an http agent with keepalive set, a new agent must be created and passed in the options to each request. Separate agents are needed for http and https, but we want to make sure to re-use the same agent for multiple requests, so we use a simple helper function to create or retrieve an agent:
For Coggle, this simple change had a dramatic effect not only on the latency of internal requests (requests are much faster when a new connection doesn’t have to be negotiated), but also on CPU use. For one service a reduction of 70%!
The AWS SDK
As with the request module, the AWS SDK for nodejs will also use the default http Agent options for its own connections - meaning again that a new connection is established for each request!
To change this, httpOptions.agent can be set on the constructor for individual AWS services, for example with S3:
const https = require('https');
const s3 = new AWS.S3({
httpOptions:{
agent: new https.Agent({keepAlive:true, rejectUnauthorized:true})
}
});
Setting keepAlive when requests are not made sufficiently frequently will not have any performance benefit. Instead there will be a slight cost in memory and cpu of maintaining connections only for them to be closed by the remote server without being re-used.
So how often do requests need to be for keepAlive to show a benefit, or in other words, how long will remote servers keep the connection option?
When keepAlive Makes Sense
The default for nodejs servers is five seconds, and helpfully the Keep-Alive: timeout=5 header is set on responses to indicate this. For AWS things aren’t so clear.
While the documentation mentions enabling keepAlive in nodejs clients, it doesn’t say how long the server will keep the connection open, and so how frequent requests need to be in order to re-use it.
Some experimentation with S3 in the eu-west-1 region showed a time of about 4 seconds, though it seems possible this could vary with traffic, region, and across services.
But as a rough guide, if you’re likely to make more than one request every four seconds then there’s some gain to enabling keepAlive, and from there on, as request rates increase, the benefit only grows.
Combined Effect
For Coggle, the combined effect of keepalive for internal and AWS requests was a reduction from about 60ms to about 40ms for the median response time, which is quite amazing for such simple changes!
In the end, this is also a cautionary tale about making sure default configurations are appropriate, especially as patterns of use change over time. Sometimes there can be dramatic gains from just making sure basic things are configured correctly.
I’ll leave you with the lovely graph of how much faster the average request to Coggle is since these changes:
I hope this was an interesting read! As always if you have any questions or comments feel free to email hello@coggle.it
Noticed a message saying to refresh the page to get the latest version, and wondering what’s changed? Here’s your occasional update on what’s new in Coggle recently!
The Coggle servers now respond 40% faster on average to requests! - watch out for a separate post with the technical details of this soon.
Large diagrams also load substantially faster (yet again) - we’ve really been on a performance optimisation binge recently.
new keyboard shortcuts to paste style, modifying the colour, shape, etc of some items to match the copied one. Check out the knowledge base article for details.
SAML single sign on users can now seamlessly re-join an organisation after being removed (as long as they are still provisioned in your single sign-on system). This makes it easier to manage your costs for a Coggle Organisation by temporarily removing inactive members from the Organisation dashboard, as the people you remove can easily re-join when they need to use Coggle again.
Changed diagrams are indexed more quickly, especially for users with large numbers of diagrams.
Improved handling of slightly malformed URLs due to typos and copy+paste errors (like additional trailing slashes, or partial URL titles), so it’ll now be harder to accidentally end up with ‘diagram not found’ when copying a link someone has sent you for a diagram.
Other little bugfixes also included fixes for an issue where the size of icons would not always be calculated correctly, where the back button from the diagram sometimes wouldn’t take you to the right folder, and for an annoying bug in Safari where sometimes the first click after a page was loaded would be ignored.
That’s all for now! As always, if you have any comments or feedback we always appreciate them, just drop us an email at hello@coggle.it.
The biggest change behind the scenes recently, has been improvements for hotlinked images (that is, images hosted on other websites and used in Coggle diagrams).
Hotlinked images are now proxied, with the files forwarded by our own servers instead of requested directly by your browser. They are also restricted in types to png, jpeg and gif images. This improves security and privacy when viewing diagrams (more on this below), and also improves performance as images can be served by Coggle’s content delivery network, which can serve content very quickly anywhere in the world.
Unlike images uploaded to Coggle, there’s a much bigger size restriction in the free plan for images you hotlink from your own hosting, so it’s a good way to get even more out of the free version of Coggle! To hotlink an image you can use the markdown image syntax with a URL to your own image hosting, like this: .
If you run an image hosting service and want to specifically block or allow images hotlinked from Coggle, then note that the referrer for our images requests will always be https://coggle.it/ - the specific diagram URL where images are used is no longer included.
How does proxying these images protect security and privacy?
A hotlinked image is loaded directly from the server which hosts it each time it is viewed: this made it possible (at least in theory) for the server to track when and how many times someone loaded the image, and by implication when they viewed the diagram. They would have also been able to collect some information about who was viewing it (including the URL of the diagram, and which browser was being used). Now that image requests are proxied, the URLs of diagrams and your browser information is completely hidden, and only a small number of requests for the image will be made regardless of how times you view a diagram, or how many people view it.
It was also not possible to know what the external server would send when a browser requested an image - the external server could theoretically send a file of any type, in an attempt to compromise your browser. While your browser should be able to safely handle any file type, there have been cases in the past where this was not the case, so it is useful to have another line of defense in our servers, which now narrowly restrict the sorts of file types which will be loaded from external servers to your browser when viewing Coggle diagrams.
It’s worth noting that we aren’t aware of any cases where Coggle diagrams have been tracked or compromised using these methods, so this update is a precaution to close off these possibilities pre-emptively - it adds another line of defense to the security and privacy of your data in Coggle.
If you run a web service like Coggle, and need to proxy external images for the same security and privacy reasons we do - then drop us a line at hello@coggle.it: the proxy service we’ve built to do this is extremely scaleable, efficient, and globally distributed, and we’d be interested in seeing if it’s useful for you too!
Other recent changes:
Keyboard Shortcuts: You can now use [ctrl] + [s] to close and save item using the keyboard: unlike just pressing [enter] this works even if an item has multiple lines of text, and makes it easier to use Coggle entirely with the keyboard.
Improved performance of the documents list, and diagram loading for certain kinds of diagrams.
As always, lots of other little bugfixes and improvements, including style tweaks for code blocks and youtube previews, cases where the history slider wouldn’t select the correct version, and in some cases did not reset correctly when closed. Fixed issues with padding of task lists, and improved performance when adjusting the permissions of people a diagram is shared with.
That’s all for now! We always welcome any kind of feedback, but we’re particularly interested in knowing if Coggle is missing keyboard shortcuts you need, so please reach out to us if there’s something missing that you’d find useful.
New year, new updates to Coggle! We’ve been sticking to our new years resolutions and fixing lots of little bugs which were annoying us, but there’s a couple of new features too:
new keyboard shortcuts for adjusting colour, shape, line style, and thickness (command can be used instead of ctrl on Mac):
Your preferred zoom level for each diagram is now saved. You can also link to diagrams at specific zoom levels by adding the ?zoom=0.25 URL parameter on links
Invitation links, and other sign-in links to diagrams owned by an Organisation with Single Sign On enabled will now redirect to the correct single-sign-on page, instead of the generic login page, and support has been added for SAML deep links.
Better handling of text pasted from Microsoft office applications (no more annoying thumbnail images of text caused by image data on the clipboard)
Improved editing of markdown links (you can now easily toggle and un-toggle the link for selected text)
As well of lots of other little bugfixes and improvements, including improvements to display of very long URLs, improved display of buttons while holding the hotkey for item removal, and a bug which could cause the cursor to jump when using some CJK input methods, and which cause cause text-input to be slow.
That’s all for now! If you have any feedback about how we can improve Coggle we’re always listening - just drop us an email at hello@coggle.it
Noticed a banner telling you to refresh to update to the latest version of Coggle recently? Here’s a roundup of what’s new!
Faster Documents List For people with lots of diagrams, we’ve significantly improved the speed that the diagram list pages load: we no longer count all of your private diagrams before showing the page (you’ll notice the private diagrams counter in the top left of the page now stops at >10, instead of showing an exact count). When you have thousands of diagrams even just counting them all takes quite a long time!
Organisation Admin Panel The organisation admin panel is now faster and more responsive, especially for Organisations with large numbers of diagrams and members, and it also now includes the guest’s email address as a tooltip for guest members.
Improved ShortcutsYou can now use control + backspace to delete single words again, instead of this deleting the whole item. Use control + shift + backspace to delete the whole item. Thanks for all of your feedback about this!
Better email address handling Several commonly used email address formats were being rejected by our email inputs – now you should be able to copy and paste addresses from more applications without having to edit them first or re-type them.
Presentation Mode Links can now configure which level of a diagram embedded or linked in presentation more should be expanded by default, with the ?expand_level=2 query parameter.
Improved search Improved search indexing of recently modified diagrams, and for users with large numbers of diagrams.
As always we’ve also been fixing little bugs, including issues sorting and filtering guest members in an Organisation, an issue where it sometimes wasn’t possible to change the order of diagrams in a folder, improvements to the Microsoft Visio export format, and some cases where our on/off switches weren’t switching as easily as they should.
With the new world of remote work, remote collaborative editing is more important than ever. We’ve launched a big improvement to the way this works in Coggle: now you can see which item everyone else in the diagram is currently editing, or interacting with, including when they’re typing and when they’re just idle.
This makes it even easier to collaborate on your mind maps and flowcharts with your remote team. If you have one of our paid plans this also includes one-click guest editing, where each guest editors have a unique colour for each editing session to help identify them.
Do you live your work life with zoom open in one tab and Coggle in another? Let us know what you think!
We’ve added some neat updates in the latest version of Coggle for Windows touchscreen laptops and tablets!
Pinch zoom and pan work naturally and seamlessly on windows touchscreens.
The eraser on touch-screen pen/stylus can be used to erase connections and items.
To make this snappy and seamless we’ve also improved the performance of zooming on all devices, whether using the zoom menu, mousewheel, or keyboard shortcuts.
Do you use a Coggle on a Surface? Let us know what you think!
Today we’re back for another look behind the curtain at the tech behind Coggle. This time, the economics of running a web service, and how we’ve made sure Coggle will be around for many years to come.
One of the most important things about Coggle has always been that we’re building for the long term: a sustainable service that you can rely on far into the future. We don’t have venture capitalist investors looking for a quick return, we just want to build a service that you find useful, provide it to as many people as possible for free, and charge a low sustainable price for those upgrade from free to our more advanced plans.
To make this work it’s really important that Coggle is hosted efficiently, so that we can sustain a large number of free users, and keep everyone’s data safe and secure without paying huge hosting costs.
Since its inception in 2013, Coggle has always mostly been on track with that, but until recently there was one Achilles’ heel, a creeping cost which threatened to undermine our sustainability.
The Graph
The first thing to notice about this is that it’s going up. That’s great, right! Right? Well, sort of. This graph is showing our database storage over time, going back five years to July 2015. An increase means more Coggle documents being created and stored, which means more people making documents, finding Coggle useful, and sharing it with their friends, which is all hunky dory.
However, database storage is *expensive*.
This might be surprising: a lot of the writing about tech companies and startups is about how hardware is cheap, how the cloud has made server costs vanishingly small, and how it’s people who are the expensive part of running a business.
Well, for the growth path of venture-capital-backed companies that’s often true: the team is growing just as fast as the data that’s being stored, gearing up the whole company to succeed – or to fail – fast. If your company might not be around in two years, you’re not worrying about how much it costs to store your data for decades to come.
But for a sustainable company like Coggle, it’s different. The durability of Coggle documents is very important: we think that one of the most important aspects of a web application that replaces something as simple as a pencil and paper is that you must be able to rely on it. And rely on it not to be just as durable as the physical document alternative – but even more so.
That means we have always stored all the data of Coggle documents across multiple physically separate servers in separate ‘availability zones’ of a datacenter, and since February 2017 we’ve also stored data across entirely separate datacenters in two different countries (Ireland, and either the UK or Germany). A single failure or disaster would never destroy a Coggle document.
This durability is why database storage is expensive, any why the graph was potentially such a problem. By 2019 the simple storage cost of five copies (plus backups) of this data, stored on high-speed disks for quick database access, was the single biggest line item in our monthly server bills. And this cost was never going to go down by itself.
Costs Up and to the Right
It was apparent that this threatened the sustainability of Coggle. So, what could we do? One option would have been to either set a time limit for the storage of free Coggle documents, or add incentives for people to delete documents they no longer need.
The problem with this is it makes Coggle diagrams seem fragile, and not like a durable physical document. It’s important that even the free version of Coggle is something you can use sustainably, because the free version of Coggle is the way most people first use it, and it sets your expectations of the paid version.
If the free version deleted your data, wouldn’t you fear that the paid version might also delete it? What if you ever miss a payment or circumstances mean you want to downgrade again?
So deleting data, even of just free customers, is not an option for us. The only other possibility was to look at how the data is stored. Can we store documents durably, and reliably, without using such expensive storage?
Blob Storage, vs Database Storage
In short, yes.
Most of the data in Coggle documents does not need to be stored in a database. While we store each individual change that was made to Coggle diagrams (this is how collaboration and history mode work in Coggle), Coggle documents are more often accessed like files, with all of the data loaded at once, and new data added only at the end of the file.
Cloud services have always supported 'blob storage’ for storing this kind of data – which is about ten times cheaper than database storage for each gigabyte stored, and has the advantage of being completely flexible, with built in archival options for infrequently accessed data: no disks need to be provisioned in advance for data which grows over time.
The challenge with blob storage is that access to the data is the primary cost, instead of the storage data size: each time a file is updated or viewed there is a small cost, and that there is no simple way of adding data onto the end of an existing file (technically speaking, blob storage is 'eventually consistent’, which means all data will be saved safely, but it might be temporarily unavailable: there is no built-in way to append data without being sure something is not overwritten).
This means that to store Coggle documents in blob storage, we’ve built a new back-end storage service that saves and loads this data. This service does the bookkeeping necessary for adding data onto files as changes are made, and ensures every change is saved immediately.
The Great Coggle Blob Storage Migration
Over the past year, we’ve migrated all of the Coggle diagram data to be stored in blob storage using this new storage service. This has been a substantial undertaking, involving migrating the format of all the data being stored, and the design of a completely new storage service, but now our costs are much better matched to the way Coggle is used, with editing and viewing documents forming the majority of our monthly bills instead of simply storing data.
All data is still saved across multiple independent locations, encrypted with independent encryption keys, and in each location data is stored across multiple independent disks with 99.999999999% durability: that’s a million times less likely that data is lost due to hardware failure than the chance of being struck by lightning. And, even if a tsunami, comet, or other disaster completely wipes out our primary datacenter in Ireland, a copy of everything would still be safe in Germany (hey given the progress of 2020 so far we’re not counting out anything!).
I hope this was an interesting read! Here’s the graph again with annotations of different points in the migration:
As well as introducing Sign in With Apple for privacy-discerning Cogglers, we’ve added some more new things recently which you might have missed! Here’s a roundup:
Profile Images since Apple (and Organisation SAML Single Sign On) do not provide user profile pictures, we’ve added some built-in profile icons to choose from in Coggle, so you don’t have to be an anonymous silhouette. Find them in the Profile tab of your settings page.
Sharing Your Awesome If you have a Coggle Awesome subscription, people you’re editing diagrams with can now use most of the Awesome features too.
ID-provider initiated sign-on support for SAML authentication for Coggle Organisation single sign-on.
Updated Emoji Support: Now supporting all the latest emoji (that’s Emoji version 13.0).
Improved Shortcuts: You can now delete items using the [control] + [backspace] shortcut (or [command] + [backspace] on Mac), and when you delete an item using a keyboard shortcut, we’ve improved the way the next item is selected.
As always we’ve also been fixing little bugs, including one which would prevent some .mm freemind files from being imported correctly, where pasting images wouldn’t always work correctly, and also made some performance improvements.
With the new world of remote work, remote collaborative editing is more important than ever. We’ve launched a big improvement to the way this works in Coggle: now you can see which item everyone else in the diagram is currently editing, or interacting with, including when they’re typing and when they’re just idle.
This makes it even easier to collaborate on your mind maps and flowcharts with your remote team. If you have one of our paid plans this also includes one-click guest editing, where each guest editors have a unique colour for each editing session to help identify them.
Do you live your work life with zoom open in one tab and Coggle in another? Let us know what you think!
coggle