.net - Azure Table Storage Performance from Massively Parallel Threaded Reading -

-

short version: can read dozens or hundreds of table partitions in multi-threaded manner increase performance orders of magnitude?

long version: we're working on system storing millions of rows in azure table storage. partition data small partitions, each 1 containing 500 records, represents day worth of data unit.

since azure doesn't have "sum" feature, pull year worth of data, either have use pre-caching, or sum data ourselves in azure web or worker role.

assuming following: - reading partition doesn't affect performance of - reading partition has bottleneck based on network speed , server retrieval

we can take guess if wanted sum lot of data on fly (1 year, 365 partitions), use massively parallel algorithm , scale number of threads. example, use .net parallel extensions 50+ threads , huge performance boost.

we're working on setting experiments, wanted see if has been done before. since .net side idle waiting on high-latency operations, seems perfect multi-threading.

there limits imposed on number of transactions can performed against storage account , particular partition or storage server in given time period (somewhere around 500 req/s). in sense, there reasonable limit number of requests execute in parallel (before begin dos attack).

also, in implementation, wary of concurrent connection limits imposed on client, such system.net.servicepointmanager. not sure if azure storage client subject limits; might require adjustment.


Comments

Post a Comment

Popular posts from this blog

ASP.NET/SQL find the element ID and update database -

-
basically i've got update query runs when click submit button. i have asp:repeater in page layer inside this <asp:repeater id="dgbookings" runat="server" onitemdatabound="itemdb"> <itemtemplate> <div class="bookingscontent"> <div class="bookingdetails" id="<%# databinder.eval(container.dataitem, "booking_ref") %>"> <p class="infotext"><%# databinder.eval(container.dataitem, "adults") %> <asp:linkbutton onclick="editdetails" text="edit..." runat="server"></asp:linkbutton></p> <p class="infotext"><asp:linkbutton onclick="submitdetails" text="submit..." runat="server"></asp:linkbutton></p> </div> </div> </itemtemplate> <asp:repeater> and outputs ...
Read more

jquery - appear modal windows bottom -

-
i have modal windows in application, work fine <div id="modal" class="modal hide fade" tabindex="-1" role="dialog" aria-labelledby="mymodallabel" aria-hidden="true"> but when modal appears, appear top of windows, , want appear bottom the application joomla 3.0! , using template of meet gavern any idea? i found solution i modified de css in bootstrap .modal.fade { top:-25%; -webkit-transition:opacity .3s linear, bottom .3s ease-out; -moz-transition:opacity .3s linear, bottom .3s ease-out; -ms-transition:opacity .3s linear, bottom .3s ease-out; -o-transition:opacity .3s linear, bottom .3s ease-out; transition:opacity .3s linear, bottom .3s ease-out; }
Read more

c++ - Compiling static TagLib 1.6.3 libraries for Windows -

-
i having super hard time compiling , using taglib 1.6.3 in qt project. i've tried can think of. taglib claims supported through cmake i'm not having luck. furthermore, i'm confused kinds of files need qt libs! i've built *.a files, *.lib, , *.dll. understand far... believe since i'm working in windows *.lib want. no matter do, end "undefined references" taglib functions try use when try compile qt project. have tried mingw32, msys, visual studio 2008, , cross-compiling windows on linux. turning nothing. what makes less sense me if compile same taglib source qt on mac (g++ think?) works fine! somewhere in windows compilation procedures have going wrong. have been smacking face on desk 30 (on , off) hours trying figure out. since qt uses mingw must compile taglib same compiler? if compile *.lib's visual studio not compatible? are *.a libraries usable in windows? (assuming mingw) i'm still trying handle on c++ stuff, after reading coun...
Read more