Deploy Solutions

We Build Great Space Apps, for the World.

Menu
  • Home
  • Solutions
    • Software Factory for building Space Apps
    • Close
  • Projects
  • Company
    • About Us
    • Careers
    • Thanks & Recognition
    • Close
  • Partners
  • News & Events
    • Space Apps in Canada: 2020 and Beyond2020-12-19
    • GeoIgnite 2020: New Horizons in Space Apps Development2020-11-30
    • Deploy Sponsors Space Apps Ottawa 20202020-09-16
    • Space Apps Specialist Re-Launches At GeoIgnite2020-08-18
    • COVID-19 Special Edition Hackathon, and Impact on Space Apps Ottawa 20202020-05-24
    • Close
  • Contact Us

Big Data Reporting Solution using AWS Cloud and Tableau

2020-01-28

Project Overview

A client receives 100M+ transactions a year from external users of their flagship product.

The client's existing reporting processes were ad-hoc, complicated and time-consuming, due to the well-defined but very complicated schema their transactions use, and the wide variety of secondary data sources required to derive business meaning from their transactions.

The client's business analysts wanted a flexible, easy-to-use reporting tool that could provide predefined as well as ad-hoc queries and visualization on large and growing data sets. Their IT department wanted a scalable, secure cloud-hosted solution that allowed long term storage.

We developed an end-to-end big data solution hosted entirely using Amazon Web Services (AWS) cloud. Data is piped through a collector micro-service, parsed into a data lake made up of numerous S3 buckets, and then automatically copied into Redshift data warehouse using Lambda. Tableau Server and Desktop are used to host and present 50+ detailed visual reports to the business analysts.

Project Goals & Objectives

The business goal was to drive reports around the functionality and health of the client’s flagship product, identify capability or process gaps, and provide longer-term reporting that may assist with help desk and IT operations. The reporting helps identify common patterns in end user submission practices, procedures, and errors.

The IT goal was to use AWS cloud hosting and services wherever possible, and scalably store a large and growing number of transactions for years to come.

Project Results

The solution we provided involved:

  • Regular client workshops;
  • Frequent prototyping;
  • AWS infrastructure setup and hardening;
  • Detailed architecture and design work including with third party organizations;
  • Data processing and analysis;
  • Custom development;
  • Deployment;
  • Ongoing testing and support.

We produce the end-to-end solution in approximately 12 weeks of elapsed effort.

One of our major success factors was that we spent a great deal of time working closely with the principal business analyst to obtain a deep knowledge of their business model, goals & objectives, and data schema.

Once we had that shared understanding, we identified their desired reports and processes using mock reports. Instead of working with their raw data formats, we  mocked up the "desired data schema" that would facilitate their ongoing reporting. We then worked backwards to map their actual raw data with their desired state.

By keeping the myriad of complex data and technology issues "under the hood", and optimizing the solution performance for reading (reporting) instead of writing (data intake), we thus ensured that the end user reporting and analysis experience was never compromised, regardless of the state of their raw data.

Another major success factor was our decision to use a variety of best-of-breed server and serverless technologies including AWS S3, Lambda, Redshift, EC2, Tableau Server, and Tableau Desktop. These ensured our solution contained the most supported and feature-rich big data technology.

Where required, we developed code and scheduled PowerShell scripts as "glue" to integrate the various components. But we limited the custom coding and attempted to use industry standards such as SQL wherever possible, to support future upgrades.

Finally, we acknowledged up front that business queries will change over time (as they have). To accommodate this, we designed the system to handle a wide variety of future data sets and foreseeable reporting needs. In particular, we built our data warehouse using the radical MAD schema approach (see "Going MAD with Tableau and AWS Redshift").

At a Glance

End-to-end big data reporting solution using Amazon Web Services cloud and Tableau, produced in 12-weeks of elapsed effort.

Project type: High Impact Consulting Services, Custom Solution.
Status: Complete
Start date: 2017-05-25
End date: 2018-12-31. Support and enhancements are ongoing.

  • Read "Going MAD with Tableau and AWS Redshift"

    LinkedIn post on how we used the MAD approach to design a flexible Redshift data warehouse, with Tableau reports.

Filed Under: High-Impact Consulting Services, Projects, Snapshots Tagged With: aws, big data, tableau

Software Product Management Support

2018-03-26

We listen closely to our clients. Sometimes that takes us in directions we didn’t expect.

In addition to our software project management assistance, we’ve been providing consulting services that would be considered “software product management”. Until recently, most organizations did not develop their own software products. While it is still somewhat rare, the increasing power, simplicity, and importance of software products, services, and apps has resulted in more and more businesses taking on this function. Once it reaches a certain scale and importance within the organization, a software product usually requires dedicated management from a product manager.

We’re seeing this evolution with some of our own clients who have asked for our assistance with their growing product management needs. The good news is that this added service fits quite well with our software project management offering. Our mission is to help companies manage their software projects better, and some of our goals apply directly to software product management.

The role of a product manager tends to vary a lot from organization to organization. Generally speaking product management is recurring work that occurs as part of an ongoing lifecycle, rather than as part of a software project that has a defined start date and end date. Still, the product manager often contributes to (or manages) multiple smaller projects that are required as part of the product development. And depending on the client need and the product manager’s own skillset, he or she may focus more on the technical/development side or on the non-technical marketing side. Some of the common tasks include researching and defining the product vision and corresponding sales and marketing objectives, then translating those into technical specifications for the software engineers to build. Once a product is created, a great deal of time and effort goes into identifying and responding to competitors, maintaining a release roadmap, determining and communicating the desired user experience, prioritizing features and bugs, assisting with marketing material, attending trade shows, as well as providing pre-sales assistance.

This is a complicated role because it requires a great deal of software experience, strong communication and negotiation skills, the ability and willingness to sit at the junction of the client, upper management, and the technical, marketing, and sales teams. A good product manager evangelizes the product vision, within the company as well as with the outside world.

We’ve expanded our service offering to assist clients with software product management needs. We can do this directly by providing part-time product management assistance as you mature your product and processes. We can do this indirectly if you need help shaping your initial product vision and roadmap, or extra muscle defining and prioritizing your product features. We can template your product marketing, sales, technical, testing, and installation material. We offer a high-flexibility, high-value, high-impact consulting service that will be tailored to your unique product management needs.

So, please book a call, choose the “What can we do for you” option, and let’s see how we can help you!

Filed Under: High-Impact Consulting Services

Consulting Support Grouped on Periodicity: Before, During, and After a Project

2017-04-13

We’ve been busy over the past few months working on our project management and delivery methodology, and helping customers on a number of different types of project. Since we offer a variety of professional services, we wondered how best to organize and present them.

We have now observed that the project support services we provide naturally group themselves based on periodicity: that is, they are logically organized into intervals which occur before, during, and after a project.

Here’s how that works:

Before A Project Starts

This kind of assistance is preparatory: creating proposals for customers; producing detailed statements of work defining what will be done on a project; or turning  business goals and objectives into multi-year, multi-phase roadmaps, with supporting strategies, high-level implementation plans, resourcing, and costs.

We hypothesize that some key offerings before a project are:

  • Proposals & RFPs
  • Statements of Work
  • Roadmaps & Strategies

You can learn more about our “Pre-Project Support” offerings here.

During A Project

This is our “core” consulting support work, assisting project managers and teams with their project delivery and coordination work.

In some cases we act like second-level project support, for instance assisting the team with avoiding or overcoming specific project issues on an “as requested” basis. For more complex or long-running work like migration and software releases, we provide ongoing technical advice and project management help, for example coordinating between several groups including IT staff, the software vendor, and the application service provider.

We hypothesize that some key offerings during a project are:

  • Scheduling & Tracking
  • Migrations & Cleanups
  • Deployments & Releases

You can learn more about our “Project Support” offerings here.

After A Project

This is what we call “future investments”.

It’s our vision to reduce global software failure rates, and if we don’t measure something, we can’t reliably improve it. Objectively gathering project feedback and metrics, analyzing the results, and then suggesting areas of improvement is therefore vital to our methodology.

By providing templates, content, and tools to accelerate project work, we should be able to simultaneously improve its quality while reducing its effort and expense, which in turn should serve to reduce the risk of project failure.

We hypothesize that key offerings after a project are:

  • Feedback & Lessons Learned
  • Project Analysis
  • Templates & Accelerators

You can learn more about our “Post-Project Support” consulting here.

As a result of our observation, we’ve redesigned our website to showcase our project support offerings more intuitively. We’d love your feedback – please comment below, contact us via the “Feedback” widget at the bottom of this site, or book a call.

Filed Under: High-Impact Consulting Services Tagged With: after project, before project, during project

Deploy's New Company Website Launches

2016-11-18

Welcome to Deploy Software Solution’s newly-launched corporate website.

Our vision is to reduce the remarkably high rate of software project failures worldwide, by transforming the way companies manage them.

To that end, we have just launched this company website to showcase the growing list of consulting services and products we can offer our clients.

In addition, we are currently working hard on our software project delivery methodology, accelerators and tools, many of which we will share on a dedicated micro-site / blog. We plan to open that up to the public in the first quarter of the 2017. You can sign up to our “Software Projects” newsletter to get notified the instant it is live, or check back regularly for more news.

Filed Under: Events, High-Impact Consulting Services

© 2021 Deploy Solutions. All Rights Reserved. | Privacy Policy |Terms of Service | Site Map

We're a proudly Canadian company, eh!