CODING & ANALYTICS by Gbenga Oridupa

  The Azure Storage platform is Microsoft's cloud storage solution for modern data storage. The services are: Durable and highly available: Redundancy ensures that your data is safe in the event of transient hardware failures. You can also opt to replicate data across datacenters or geographical regions for additional protection from local catastrophe or natural disaster. Data replicated in this way remains highly available in the event of an unexpected outage. Secure: All data written to an Azure storage account is encrypted by the service. Azure Storage provides you with fine-grained control over who has access to your data. Scalable: Azure Storage is designed to be massively scalable to meet the data storage and performance needs of today's applications. Managed: Azure handles hardware maintenance, updates, and critical issues for you. Accessible: Data in Azure Storage is accessible from anywhere in the world over HTTP or HTTPS. Microsoft provides client libraries for Azure ...

Since the inception of Relational Databases in the 70’s, schema on write has be the defacto procedure for storing data to be analyzed. However recently there has been a shift to use a schema on read approach, which has led to the exploding popularity of Big Data platforms and NoSQL databases. Any data management system belongs to one of two types: Schema-on-write: Probably a lot of you have already worked with relational databases and you understand that once we have configured the schemas, created the tables, we can begin to ingest the data. Remember just because the data is structured doesn’t mean it starts out that way. It is likely to be something like bulk upload data from a text or csv file whose structure we know in advance because it somehow matches the schema of the tables, and once the data is loaded into the table, we can begin to execute analytical queries on our tables. This ...

Let us start with a quick introduction. Amazon simple storage service or Amazon S3 provides developers / IT teams / Organizations / Individuals with secure, available, price friendly object storage in the cloud. Ok. Slow down. What is object storage? It is a computer data storage architecture that manages data as objects, as opposed to other storage architectures like file systems, which manage data as a file hierarchy, and block storage, which manages data as blocks within sectors and tracks. Amazon S3 is easy to use and can be used to store and retrieve any amount of data at any time. All data transfers are over automatic encryption and SSL. Data is secure, available when needed and will scale as your needs grow. Last thing you want to worry about is losing valuable data. Amazon S3 automatically replicates your objects on multiple devices across multiple facilities. In Amazon S3, you create bucke...

In today’s post, I will like to talk about considerations for choosing to use Azure Blob Storage or Azure Data Lake Store when processing data to be loaded into a data warehouse. Here is a Data Warehouse Architecture published by Microsoft, where it suggests loading data from your source into Azure Blob Storage. Pause for a second!!! Ok, Let's continue!!! So, here are my thoughts on why you may choose one over the other based on my experience in some projects. It really "depends".In most cases you can’t go wrong either way because they are both powerful storage systems.Let's go: Firstly, Text files. ADLS is better with text files compared to ABS.When you talk about non-text data like media files,database backup files etc, are better off with ABS.There are trade-offs with both. Secondly, Geographic redundancy. ABS gives you that out of the box. For ADLS,i...

Microsoft recently introduced the use of templates in Azure Data Factory(ADF). I believe this is a very solid implementation. By having this addition, data engineers can easily get started quickly with building data factory pipelines and improve developer productivity along with reducing development time for same workflows repeatedly. Ready to explore?. Let's jump in for a cool ride: 1. Just click Create pipeline from template on the Overview page or click +-> Pipeline from template on the Author page in your data factory UX to get started. 2. Select any template from the gallery and provide the necessary inputs to use the template. You can also read detailed description about the template or visualize the end to end data factory pipeline. 3. You can also create new connections to your data store or compute while providing the template inputs. 4. Once you cli...

So people have been asking me to give some details on MapReduce concept. This is a very interesting topic to write about. If you have read my previous post, you would have seen my introduction to Big Data and Hadoop. Now I am going to talk about MapReduce has the heart of Hadoop. Some of you might be new to this, but do not worry, it is going to be described in a way you will quickly understand. To Java developers, it might be much easier, but if you do not have experience in java skills, you can still learn some basic java and master MapReduce. MapReduce is a programming framework that allows performance of distributed and parallel processing on large data sets in a distributed environment. I am talking massive scalability across hundreds or thousands of servers in a Hadoop cluster. Just imagine that for a second. If you see in the diagram above, we have the “Input, Map task, Reduce task ...

What is Scala?: Scala combines object-oriented and functional programming in one concise, high-level language. Scala's static types help avoid bugs in complex applications, and its JVM and JavaScript runtimes let you build high-performance systems with easy access to huge ecosystems of libraries. What is Python?: Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. Both Python and Scala programming languages offer a lot of productivity to programmers. They are useful tools among data scientists. Most learn both languages for Apache Spark. However, majority prefer Scala to ...

Big Data! Yes, do you really know what exactly it is, and its influence to the world today?. In order to understand the term 'Big Data' , we first need to know what 'Data' is. Oxford dictionary defines 'data' as - "The quantities, characters, or symbols on which operations are performed by a computer, which may be stored and transmitted in the form of electrical signals and recorded on magnetic, optical, or mechanical recording media. " Now imagine what ‘Big Data’ is. Big Data is a term used for a collection of data sets that are large and complex, which are difficult to store and process using available database management tools or traditional data processing applications. Examples of Big Data: The following are some of the examples of 'Big Data'- The New York Stock Exchange g...