Apache Spark

Analytics With Apache Spark Is Here

As promised earlier this year, we at Vidya are proud to officially announce our newest course Analytics with Apache Spark. Spark is a cool technology making an enormous–and growing–impact in the Big Data space, so naturally there are a lot of courses out there. Ours is different. Naturally we spend a lot of time on Spark itself with numerous code examples and challenging exercises, but we also stress the importance of things that have always mattered and still matter–architecture, security, and software engineering concepts like unit and integration testing, continuous integration, and continuous delivery.

Continue reading

Welcoming TRSS

Vidya is proud to be working with Thomson Reuters Special Services, a leading provider of threat detection solutions. Their software analyzes billions of public and proprietary records with innovative technology to deliver realtime, actionable intelligence to support sound decision making. We have joined a team of senior engineers with a wide variety of expertise. Currently, we are using Play Framework in Scala as the web application framework with an AngularJS interface to a backend MongoDB database.

Continue reading

Analytics With Apache Spark Is Coming

At Vidya we currently offer two courses, Software Engineering in Java and Agile Software Project Management with Scrum. In response to popular demand…OK, like eight or nine people…we are currently working on a third course to be ready by Summer 2015 tentatively called Analytics with Apache Spark. As “Big Data” becomes more and more of a thing, there just aren’t enough software engineers who know the tools and techniques for doing meaningful, performant, cloud-scale analytics.

Continue reading

Getting Started With Hadoop

Please take a look at my latest column for Government Computing News where I gently introduce you to the basics of Hadoop and describe cool technologies that build on it. Just to give you an idea, here is the unedited introduction. It’s a funny word. You have only a vague notion of what it is. You’ve heard that it takes a lot of work but is potentially really beneficial. Maybe if you learned more about it you too could enjoy its benefits.

Continue reading

Lighting a Spark with HBase

Although most developers and users are still feeling their way through Hadoop and (more specifically MapReduce), the truth is Google wrote that paper in 2004. That’s ten years ago! Million Dollar Baby won Best Picture that year. Yeah! by Usher, Lil John and Ludacris topped the charts in the United States. And Facebook had only just started to kill work productivity and violate your privacy. As long ago as that feels, it is an eternity in technology.

Continue reading

Java is Dysfunctional with Big Data

Let me first say I love Java. There is a reason it’s the most popular programming language in the world. For me personally, I made a career out of building systems in Java, and I even teach a course in Java. But when it comes to Big Data, Java simply doesn’t cut it. Everybody knows functional languages have enjoyed a renaissance as Big Data has become a thing. And for good reason: immutable (or mostly immutable) state, lazy evaluation, the natural fit with recursion, and so on.

Continue reading