{"id":7270,"date":"2016-10-17T14:08:24","date_gmt":"2016-10-17T12:08:24","guid":{"rendered":"https:\/\/blog.redbaronofazure.com\/?p=7270"},"modified":"2016-10-17T14:32:24","modified_gmt":"2016-10-17T12:32:24","slug":"put-block-apis-and-c","status":"publish","type":"post","link":"https:\/\/blog.redbaronofazure.com\/?p=7270","title":{"rendered":"PutBlock APIs and C++"},"content":{"rendered":"

Ingesting data into storage may be a non exciting task but it is a task that you get back to again and again. There are many tools for uploading files to Azure Storage and\u00a0there are also many SDKs so you can implement it in most popular languages. I’ve written how you can use the PutBlock APIs in both Java<\/a> and javascript<\/a> before and recently I had the chance to implement it i C++ to create a portable solution.<\/p>\n

PutBlock and PutBlockList APIs<\/strong><\/p>\n

The Azure Storage APIs PutBlock and PutBlockList are very basic but also very powerful. With PutBlock you upload a chunk of a file which you then repeat until all of the file is uploaded. You commit all of the chunks in one call to PutBlockList which takes all the chunk ids and their respective order. The beauty is that you can upload a\u00a0large files in parallell by using multiple threads, there by avoiding the\u00a0sequential processing of first reading a chunk and sending if over the network, then read some more, etc etc. How many threads and how large the chunk size should be may vary from machine to machine, but the goal is to maximize the utilization of the network card.<\/p>\n

Azure Storage C++ library<\/strong><\/p>\n

Microsoft do not have full a complete C++ Azure SDK as they have for C#, Java and other languages, but there is a C++ library for Azure Storage called azure-storage-cpp<\/a>. In a previous post<\/a> I wrote about how to leverage that and another Microsoft C++ REST library named Casablanca<\/a> for implementing Azure KeyVault functionality in C++. The sample program I develop in this post builds on the same principles, so instructions for how to build azure-storage-cpp and Casablanca will not be repeated here.<\/p>\n

Parallell uploading Sample<\/strong><\/p>\n

Each chunk of the file holds the starting offset in the file, the length of the chunk and its identity.<\/p>\n

class FileChunk\r\n{\r\npublic:\r\n\tint id;                                             \/\/ seq id of the chunk to read\r\n\tunsigned long startpos;                             \/\/ offset in file where to start reading\r\n\tunsigned long length;                               \/\/ length of chunk to read from file\r\n\tint threadid;                                       \/\/ marked by the thread that pulls the piece from the queue\r\n\tbool completed;                                     \/\/ marked by the thread when completed\r\n\tunsigned long bytesread;                            \/\/ actual bytes read from file\r\n\tfloat seconds;                                      \/\/ time it took to send this chunk to Azure Storage\r\n\tutility::string_t block_id;                         \/\/ BlockId for Azure\r\n<\/pre>\n
The list of chunks is stored twice. First in a list so we can keep track of them, and then also in a queue that serves as a way to feed all background threads with work to do.<\/p>\n
std::list<FileChunk*> chunkl;\r\nstd::queue<FileChunk*> queueChunks; \r\n<\/pre>\n
Before creating the threads, we create a list of FileChunk objects and populate the list and queue.<\/p>\n
\/\/ create chunks and push them on a queue\r\nwhile (remaining > 0)\r\n{\r\n    chunksread++;\r\n    long toread = remaining > this->chunkSize ? this->chunkSize : remaining;\r\n    FileChunk *fc = new FileChunk(chunksread, (unsigned long)currpos, (unsigned long)toread);\r\n    chunkl.push_back(fc);\r\n    this->queueChunks.push(fc);\r\n    remaining -= toread;\r\n    currpos += toread;\r\n}<\/pre>\n
Then it’s time to create the threads and put them to work.<\/p>\n
\/\/ create threads that process tasks in the queue\r\nstd::list<std::thread*> vt;\r\nfor (int n = 1; n <= countThreads; n++)\r\n{\r\n    std::thread *t1 = new std::thread( threadproc, n, this );\r\n    vt.push_back(t1);\r\n}<\/pre>\n
The background processing is pretty simple. It just pulls the next chunk to be processed off the queue, reads the file as specified in the FileChunk info, creates a BlockId and sends the chunk to\u00a0Azure Storage via the PutBlock API, which in the C++ library is called upload_block. Since it is a common queue, we can use as many threads as we want, but there is really no point in having to many since the bottleneck will be throughput on the network cards.<\/p>\n
azure::storage::cloud_block_blob blob = this->container.get_block_blob_reference( this->blobName );\r\nstd::vector<uint8_t> buffer( this->chunkSize );\r\n\/\/ get the next file I\/O task from hte queue and read that chunk\r\nwhile (!this->queueChunks.empty())\r\n{\r\n    FileChunk *fc = (FileChunk*)(this->queueChunks.front());\r\n    this->queueChunks.pop();\r\n    \/\/ read the specified chunk from the file\r\n    file.seekg(fc->startpos, ios::beg);\r\n    file.read((char*)&buffer[0], fc->length);\r\n    fc->bytesread = (unsigned long)file.gcount();\r\n    \/\/ create Azure Block ID value\r\n    fc->block_id = utility::conversions::to_base64(fc->id);\r\n    auto stream = concurrency::streams::bytestream::open_istream(buffer);\r\n    utility::string_t md5 = _XPLATSTR(\"\");\r\n    unsigned long t0 = clock();\r\n    blob.upload_block(fc->block_id, stream, md5);\r\n    fc->seconds = (float)(clock() - t0) \/ (float)CLOCKS_PER_SEC;\r\n    fc->threadid = threadid;\r\n    fc->completed = true;\r\n}\r\n<\/pre>\n
When the queue is empty, the threads terminate and the main program continues with the next step which is building a list of the BlockIds and calling the final commit to the PutBlockList API. This method is named upload_block_list in the C++ library.<\/p>\n
\/\/ wait for all threads to complete\r\nstd::list<std::thread*>::iterator itt;\r\nfor (itt = vt.begin(); itt != vt.end(); ++itt)\r\n{\r\n    (*itt)->join();\r\n}\r\n\/\/ create the block list vector from results\r\nthis->total_bytes = 0;\r\nstd::vector<azure::storage::block_list_item> vbi;\r\nstd::list<FileChunk*>::iterator it;\r\nfor (it = chunkl.begin(); it != chunkl.end(); ++it)\r\n{\r\n    azure::storage::block_list_item *bli = new azure::storage::block_list_item((*it)->block_id);\r\n    vbi.push_back(*bli);\r\n    this->total_bytes += (*it)->bytesread;\r\n    delete (*it);\r\n}\r\n\/\/ commit the block list items to Azure Storage\r\nblob1.upload_block_list(vbi);\r\n<\/pre>\n
 <\/p>\n
Running the program<\/strong><\/p>\n
To specify the storage account to be used, you can either pass the account name and key on the command line or you can set it as environment variables called STORAGE_ACCOUNT_NAME and STORAGE_ACCESS_KEY. Then you pass the local filename and the name of the Storage Container on the command line to execute the program. The below is output from running it on a D1 VM with 4 threads uploading a file of 1GB. It’s a pretty impressive performance. To get even better perf, you would need to create a VM in Azure with multiple NICs.<\/p>\n
\"cppblockupload_run\"<\/a><\/p>\n
\"cppblockupload_syntax\"<\/a><\/p>\n
Building the program<\/strong><\/p>\n
The C++ program can be built on Windows, Linux and Mac. The github repo contains a Visual Studio 2015 solution and a makefile that you can use on Linux\/Mac. On Windows, installing the azure-storage-cpp is automatic via NuGet (or if you do Install_package wastorage yourself). On Linux\/Mac, you have do git clone and build Casablanca and azure-storage-cpp manially.<\/p>\n
Summary<\/strong><\/p>\n
The PutBlock\/PutBlockList APIs are quite versatile in that you can build data ingestion solutions in almost all popular languages. With the C++ library solution, you can build your own tools that can run on a wide array of machines and devices. C++ might not be the language that you write a lot of code in today, but if you need raw performance and portability, you can interact with Azure Storage quite easily and achieve good performance.<\/p>\n
References<\/strong><\/p>\n
Github repo – Source code
\nhttps:\/\/github.com\/cljung\/azblockupload<\/a><\/p>\n
MSDN documentation – PutBlock APIs
\nhttps:\/\/msdn.microsoft.com\/en-us\/library\/dd135726.aspx<\/a><\/p>\n
Github \u2013 azure-storage-cpp
\nhttps:\/\/github.com\/Azure\/azure-storage-cpp<\/a><\/p>\n
Github \u2013 Casablanca C++ REST SDK
\nhttps:\/\/github.com\/microsoft\/cpprestsdk
\n<\/a>Make sure to look at the wiki page for documentation on how to build it<\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Ingesting data into storage may be a non exciting task but it is a task that you get back to again and again. There are many tools for uploading files to Azure Storage and\u00a0there are also many SDKs so you can implement it in most popular languages. I’ve written how you can use the PutBlock […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[405,131,121,321],"tags":[407,31],"_links":{"self":[{"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=\/wp\/v2\/posts\/7270"}],"collection":[{"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7270"}],"version-history":[{"count":4,"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=\/wp\/v2\/posts\/7270\/revisions"}],"predecessor-version":[{"id":7274,"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=\/wp\/v2\/posts\/7270\/revisions\/7274"}],"wp:attachment":[{"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7270"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7270"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.redbaronofazure.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7270"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}