Why should I not put all my data in one CosmosDB collection?

Multi tool use
Multi tool use


Why should I not put all my data in one CosmosDB collection?



The problem



I have discovered that Cosmos DB is priced very aggressively and can be expensive if used with many data types.



I would think that a good structure, would be to put each data type I have in their own collection, almost like tables in a database (not quite).



However, each collection costs at least 24 USD per month. This is if I choose "Fixed", that limits me to 10GB and is NOT scalable. Hardly the point of Cosmos DB, so I would rather choose "Unlimited". However, here the price is at least 60 USD per month.



60 USD per month per data type.



This includes 1000 RU, but on top of this, I have to pay more for consumption.



This might be OK if I have a few data types, but if I a fully fledged business application with 30 data types (not at all uncommon), it becomes 1800 USD per month, at least. As a starting price. When I have no data yet.



The question



The structure of the data in the collection is not strict. I can store different types of documents in the same collection.



When using an "Unlimited" collection, I can use partition keys, which should be used to partition my data to ensure scalability.



However, why do I not just include the data type in the partition key?



Then the partition key becomes something like:


[customer-id]-[data-type]-[actual-partition-value, like 'state']



With one swift move, my minimum cost becomes 60 USD and the rest is based on consumption. Presumably, partition keys ensure satisfactory performance regardless of the data volume. So what am I missing? Is there some problem with this approach?





No there will be no problem per se. It all boils down to whether you're fine with having 1000 RU/s for your whole system. In fact you can simplify this even more by having your document id to be the partition key to enable the maximum distribution and scale in CosmosDB. That's exactly how Collection sharing works in Cosmonaut and I have noticed no problems, even on systems with many different data types.
– Nick Chapsas
Jul 3 at 12:39






@NickChapsas your comment sounds more like an answer, so maybe post it as one?
– Haspemulator
Jul 3 at 12:50





@Haspemulator Oh sorry i wanted to write it as an answer but I clicked on the wrong area.
– Nick Chapsas
Jul 3 at 12:57





You can always start at fixed collection and migrate to unlimited collection once you're getting near the limits. Overprovisioning before you actually need it is to be expected to be expensive.
– Imre Pühvel
Jul 4 at 13:11




1 Answer
1



No there will be no problem per se.
It all boils down to whether you're fine with having 1000 RU/s, or more specifically a single bottleneck, for your whole system.



In fact you can simplify this even more by having your document id to be the partition key. This will guarantee the uniqueness of the document id and will enable the maximum possible distribution and scale in CosmosDB.



That's exactly how collection sharing works in Cosmonaut (disclaimer, I'm the creator of this project) and I have noticed no problems, even on systems with many different data types.



However you have to keep in mind that even though you can scale this collection up and down you still restrict your whole system with this one bottleneck. I would recommend that you don't just create one collection but probably 2 or 3 collections with shared entities in them. If this is done smart enough and you batch entities in a logical way then you can scale your throughput for specific parts of your system.





Is it really a bottleneck if we have different partitions? Let's say we have 10 documents in one partition. Is that faster than having 100 documents in 10 different partitions, if we just need the data from one partition?
– Niels Brinch
Jul 4 at 10:48





The throughput is collection wide, not partition wide. It doesn't matter how many partitions you have. If you are constantly pumping and querying for data you might hit those 429s. Knowing the partition value only makes your queries faster and more cost efficient.
– Nick Chapsas
Jul 4 at 11:26





"your document id to be the partition key" - beware, this would cause a inefficient fan-out queries across all partitions should you ever need a select by other than id.
– Imre Pühvel
Jul 4 at 13:08





@ImrePühvel Which will be a problem anyway if he goes with the [customer-id]-[data-type]-[actual-partition-value, like 'state'] approach.
– Nick Chapsas
Jul 4 at 13:09


[customer-id]-[data-type]-[actual-partition-value, like 'state']





Nick, throughput is OK, I will of course buy more RUs if I need more. Regarding the partition strategy that I propose, it would have thousands of documents in the same partition. As opposed to your proposed strategy where each document has their own partition. So it is not the same structure in terms of partitions.
– Niels Brinch
Jul 4 at 14:41






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

XgsCyop1j,2oYG
-
qSdX Kt,3 bI,Lx 2PLq1vCtDSFFEXR32KXEgX5glYWFbH5,XW,lMk1t7,E,D9fsI,xCM4S3n,3mHYt,oeat

Popular posts from this blog

PHP contact form sending but not receiving emails

PHP contact form sending but not receiving emails I have created a contact form using PHP to send the data. The form seems to send fine as there are no errors and success message appears, however I am not receving the emails into the designated inbox. I am using PHPMailer as I originally tried just using the php 'mail' command which I now understand is a bit hit and miss. I cannot see why I would not be receving the emails so I would be very grateful for any help that could be given. I'm quite new to PHP so please be patient with me :) <?php use PHPMailerPHPMailerPHPMailer; use PHPMailerPHPMailerException; $msg = ""; if (isset($_POST['submit'])) { require 'phpmailer/src/Exception.php'; require 'phpmailer/src/PHPMailer.php'; require 'phpmailer/src/SMTP.php'; function sendemail ($to, $from, $body) { $mail = new PHPMailer(true); $mail->setFrom($from); $mail->addAddress($to); $ma...
Read more

Do graphics cards have individual ID by which single devices can be distinguished?

Do graphics cards have individual ID by which single devices can be distinguished? I have several graphics cards of the same manufacturer and same model and I want to distinguish them not only by the pcie socket they reside in but by some individual value I can read from the hardware. Is there such value and if so what would it be called and how could I access it? Right now I'm using modern nvidia cards (10xx series) and nvidia inspector.but I'd prefer if there was any value that was also applicable to amd. I have programming and scripting knowledge. I am running Windows 10 1 Answer 1 This is a powershell example to pull information from video cards. The DeviceID is supposed to be unique but only if the manufacturer supports Windows Display Driver Model (WDDM). (I copied this directly from docs.microsoft.com) $strComputer = "." $colItems = get-wmiobject -class "Win32_VideoC...
Read more

Create weekly swift ios local notifications

Create weekly swift ios local notifications I can create daily notifications but not weekly ones using swift 3 ios 10 User Notifications framework. Daily works fine but when I add in the .weekday parameter it doesn't send me a notification. The code is: for i in 1 ... 7 { let center = UNUserNotificationCenter.current(); let content = UNMutableNotificationContent(); content.title = "Title"; content.body = "content"; content.sound = UNNotificationSound.default(); var dateComponents = DateComponents(); dateComponents.weekday = i; // hours is 00 to 11 in string format if (daynight == "am") { dateComponents.hour = Int(hours); } else { dateComponents.hour = Int(hours)! + 12; } // mins is 00 to 59 in string format dateCom...
Read more