SDLC Series: Continuous Integration

, ,

You may have heard the term “continuous integration” or “continuous deployment” or even “continuous delivery” tossed about in your department as a catch-all phrase for “we need to ship code quickly and constantly”. It’s true that a well-honed continuous integration (CI) program can result in rapid, hyper-agile delivery of software but in order to reap the philosophy’s rewards you have to establish and adhere to a disciplined protocol that is based on a true understanding of what CI actually is.

In order to understand CI, let’s look at the way software used to be shipped in the years leading up to the golden age of Agile, say, the aughts (2000 to 2010-ish). During these years, even the smallest feature change to an in-place application was a major undertaking. Budgets had to be approved, designs made, code written and tested, bugs fixed, user acceptance granted, and then a big monolithic chunk of code was released as a new version of the software. Because making a release was such a big undertaking, the needs of most stakeholders from the business fell to the wayside; there simply wasn’t enough time or budget to cater to all of their needs. 

Things began to change, however, as we entered the teens (2013 to present).  Web and native apps intended for consumption on smartphones exploded filling more and more niche needs and the typical business stakeholder became ever savvier in all things software as they grew accustomed to having myriad software features to solve problems in their personal life. This created a demand that spilled over into the workplace and became common in just about every conference room around the world:  “Amazon sends me updates about the location of my order every step of the way! Why can’t we do that with our replacement part orders??” or “Searching for information on Google is so intuitive–it should be the same when we search our inventory” or how about, “We should make a mini-game like Angry Birds to promote this new ad campaign”. Overnight, software delivery professionals–from developers to quality assurance to analysts–were overwhelmed with requests and outnumbered by throngs of stakeholders with wishlists a mile long. The age of carefully planned, waterfall-like software release delivery schedules and the age of “I want it all and I want it now” Agile methodology had begun.


In the years since that critical inflection point in the art of software delivery Agile, a stream-of-consciousness approach to software delivery has proliferated in response to stakeholder demand/impatience and this has in turn given rise to CI which is the process of streamlining and automating the business of requirement specification, development, quality assurance, testing, user acceptance and, finally, production deployment. 


In a CI world, a stakeholder may express a desire for a new feature in the company intranet during the Monday morning meeting. By lunchtime, the business analyst has gathered detailed requirements and placed the requirements into a ticketing system such as Visual Studio Team Services or Jira. This alerts the dev team automatically so that they can step away from the foosball table and get back to their workstations. By Monday afternoon, the developer(s) has accepted the ticket and used it’s automatic integration with the source control repository to create a new “branch” of the code. The developer’s job is done within the hour and her code is checked in which triggers an automatic execution of unit and end-to-end tests and then an automatic build to the QA environment and Slack notification to the QA testers. Once the QA staff has approved the build, the CI pipeline takes over once more and automatically handles the placement of the new branch into the production environment while maintaining the ability to easily roll back to the previous build if necessary.  By Tuesday morning, the stakeholder is happily using the feature he requested during the previous day’s morning meeting. This would never be possible without an established CI program in the organization.

A well-developed CI program isn’t just for the benefit of the stakeholders; it has plenty of deep technical advantages as well. For example, most projects have multiple developers working in isolation. Adherence to a CI protocol forces a degree of work atomization which limits the ability for discrete tasks to become too large. This means more frequent code check-ins and integration with the production environment which means fewer nasty merge conflicts and bugs.

By now you probably get what CI does, but you may be asking yourself what exactly it is. Is it a tool? A platform? A philosophy? In reality, it’s a little bit of everything. DevOps professionals create “build definitions” using popular build engines like Visual Studio Team Services, Team City by JetBrains, Jenkins, or Octopus. You can think of these definitions as scripts that have hooks into both the source control repository where your application’s code resides as well as into the environments (servers) that run the working code. In a sense, these build definitions are a collection of IF THIS THEN THAT statements: “If a new ticket is added then create a new branch and email the dev team”, “If a developer checks in their code, then run unit tests”, and then, “If all unit tests pass, then deploy to the QA environment and email the testers”. 

Different build engines have different strengths and it is possible that your organization uses more than one of them. In fact, we developed our NoOps/digital developer product called Catapult to help abstract and alleviate the stress of managing multiple build servers and other resources in order to further streamline the continuous integration process.

The other aspect of a solid CI program is to enforce a protocol to be followed by all team members. This is very important because if the team does not use the correct toolchain, the CI program won’t work and its benefits are lost. For example, if the stakeholder from the Monday morning meeting were to have simply emailed his request directly to the developer, that developer–eager to please–may have coded the feature and then checked it directly into the source control repository without following the proper branching protocol. This may have caused merge conflicts that then require a manual review and possibly fail to trigger automatic tests which would then result in the very real possibility that bugs slip through to production environments and create the need for site downtime.  The good news is, there are some pretty great tools out there to make adherence to a CI protocol pretty easy. We are, of course, partial to Catapult but regular ol’ VSTS or Jira are pretty good as well.

If you are interested in instituting a CI program at your enterprise but don’t know where to start, please feel free to contact us for help. We are experts in the field of CI and we can either help you design and roll out a custom CI program or implement a licensed instance of Catapult to make CI (and DevOps in general) feel like magic.

/by

Enterprise Uses of Machine Learning

, ,

According to a recent Gartner survey, only 37% of CEOs and other senior managers rate the efficacy of machine learning as being transformational to their enterprise. This isn’t too surprising, really. We commonly hear about tech giants such as Facebook, Google, and Netflix who are applying machine learning solutions to problems faced by hundreds of millions of people and so the notion somehow makes more sense when considered in that mass-market context.

As with most cutting-edge technologies, however, adoption tends to go from a trickle to a roar. Consider, for example, the automobile, the personal computer, and the cellular telephone. These were all technologies that seemed over-the-top at the time of their creation with few thinking that these things would be useful to a business, let alone a household. But of course, history has told us a different story. It turns out that the problem isn’t with the technology, the problem is with our ability to envision the use of the technology which leads to the point of this article: How is machine learning actually useful to most enterprises? In this article, we will answer that question, but it is helpful to start with an analogy.

Machine learning is great at detecting patterns in unstructured data and then formulating conclusions. This is the kind of thing that humans are generally good at doing. For example, if you go to a car dealership and notice that there is only one car remaining from the previous model year, and it’s the end of the month, you somehow realize that you can probably negotiate a great purchase price for this vehicle. But how did you know that? You knew it from prior articles you have read and advice your friends have given you with regards to getting a great deal on the purchase of a vehicle. You might have also picked up on visual cues at the dealership such as a “Must Sell!” sticker on the car.

This is what machine learning does except that instead of drawing only on individual experience, it can draw on the experience of hundreds or thousands of people who have been through similar scenarios in order to formulate positive insights and conclusions.

There are troves of data containers collecting dust at every enterprise stored by your CRM, ERP, email servers, and documents as well as external data sources such as news stories, weather events, commodity prices, etc. Implementing machine learning algorithms across this data can help you to develop insights about shortening your sales cycles, the communication cost around certain internal processes, employee and customer sentiment, customer buying habits, new product recommendations, and so forth.

Consider a machine learning algorithm that draws upon this information to help prompt you with tips on when to close a deal (and at what price). Or consider a machine learning algorithm that can help you spot communication gaps that are leading to quality defects on the production line. These things may seem like uniquely human problems, but they aren’t. Machine learning is excellent at detecting patterns, optimal timing for events, and even emotions. How would it impact your enterprise if you could predict the optimal time to launch a product, close a deal, raise your prices, discontinue a product, or even liquidate bill-of-materials stock?

Areas in which machine learning can help your enterprise

Here are a few other areas where machine learning can be readily applied to the enterprise:

Fraud Detection

Fraud detection via machine learning is already widely adopted for credit card companies and insurance companies, but what about putting that same pattern recognition technology to use in your enterprise to help resolve and steer customer service and return matters or perhaps where sales or productivity is being over or under-reported by employees?

Document Classification

Machine learning is excellent at classifying documents. You already use machine learning-based classification if you have a spam filter on your email client. This is no more than an algorithm that has learned to detect patterns in email text that has a high probability of being spam. What if you applied the same logic to help your employees classify and organize emails, comments, documents, Slack posts, etc by project, or recommend that they should not copy people to whom the information would be unnecessary while copying others who may have the ability to answer questions in their emails.

Machine learning can also make short work of crawling hundreds of thousands of historical documents, parsing their content, and storing them in a database for further analysis. This is not only true for word processing documents and spreadsheets, but it’s also true for images, video, and audio as well.

Automation

Machine learning can help to automate many processes within the enterprise by consuming unstructured data, such of scanned bills-of-lading, and taking the appropriate action to update your systems. What’s more, machine learning can even help you to identify which processes can easily be automated in the first place.

Diagnosis

The medical community is using machine learning for diagnosis more and more frequently but what about diagnosing equipment failures, detecting conflicts in a set of building design documents, and finding manufacturing processes that are the root cause of defects? Machine learning handles and solves this kind of diagnostic problem very well.

Machine Vision & Shape Detection

Machine vision is a trait enabled by machine learning wherein streaming video or images can be processed and certain objects within them can be recognized. This can help you sort through thousands of hours of footage when looking for a particular event, recognize installed quantities on a construction site, keep tabs on deliveries, and even let you know when the lawn should be tended to for a rental property.

Product Recommendations

Product recommendations are sort of old hat at this point, but is your enterprise using a good recommendation engine? You should be–targeted recommendations to your customers based on conclusions drawn by machine learning algorithms can increase sales by 50%-60%. Even if you don’t run an eCommerce business, you can use these algorithms to tip you off as to when to send regular mailers to certain customers.

There are some who believe that the next wave of enterprise software will be powered by machine learning and here at Polyrific, we agree. If you are interested in exploring the possibilities offered by integrating machine learning into your enterprise, we are here to help. We can be both a creative force driving your team towards greater heights with this ground-breaking technology and we can also handle the implementation. Working with us doesn’t have to mean that we are building software for you, though we’d like to; we can simply be a source of ideas and direction. 

Please contact us today to learn more.

/by

The Future of Machine Learning

, , ,

You probably already know that machine learning is an incredibly powerful technology that has the ability to solve difficult problems in a surprisingly effective manner.  What you may not have realized, however, is that since machine learning algorithmically builds its “gray matter” by learning from previous patterns, trends, and data models, we are at present witnessing only the very early stages of what machine learning can do for us.

Recently the science behind machine learning hit a significant milestone in fields that hadn’t really moved the needle for some time like speech recognition and image understanding. With the recent proliferation of sufficiently capable computing hardware, we witnessing a BIG BANG in machine learning technology that represents a major step forward in how computers can learn and perform.

For several years now, the use of machine learning has been used as a form of automation for low-value tasks that are easy to do but time-consuming when carried out by human hands. As we move into the near future, expect to see an explosion of applied machine learning as the necessary computing power and software implementation proliferate around you but it won’t all be easy; machine learning algorithms tend to have errors, and it is very interesting to know how we humans-in-the-loop in “coach” of those errors out of the result sets through training and deep-instruction in neural networks.

With that said, machine learning will have a great impact on all areas of business. One of the important things for enterprises to bear in mind is that they need to look beyond the AI hype for practical ways to incorporate machine learning into their operations. Expect too much too fast and we will find ourselves in another “AI Winter”: a season we have witnessed before during which confidence in machine learning plummets and investment stops. Machine learning algorithms should be regarded as a child in need of time and instruction to become truly effective. Goldcorp – a mining company that uses immense vehicles to haul tailings and other debris away from mining sites – is taking this step-by-step approach with great results by iterating a machine learning algorithm over time that now predicts with over 90 % accuracy, when will their machines need maintenance. Since a vehicle breakdown can cost Goldcorp over $2 million per day, it’s hard to argue the economy of this kind of applied machine learning; however, had Goldcorp expected for machine learning to first be able to make all of their monster vehicles self-driving, it is very possible that the program would have failed and the more simple, but extremely useful, algorithms would have never been implemented.

Short Term Predictions

More enterprises will begin their machine learning journey over the next 18 months than any other time in history. The smarter ones will create competitive separation for their enterprise by getting started with machine learning now while still learning from others’ mistakes. Resisting the urge to expect too much too fast will pay off handsomely – as was the case with Goldcorp – while machine learning quietly takes hold beneath a cacophony of AI marketing speak.

Employment

Some of the gloomier predictions will end on a higher note: machine learning will automate some human jobs out of the equation, but those jobs will be replaced with higher-value, more stimulating work. Retail and sales jobs are primed for machine learning implementation and automation. We will see robots in hospitals delivering medicines, materials, and meals. Self-driving cars will rule the highways in the next few years. In fact, we will see autonomous trucks, tractors, taxis, forklifts, cargo handlers, etc.

Automation of such large parts of our workforce is going to require that our governments come together in a very bipartisan way to avoid economic strive, but on the positive side, the world will see a tidal wave of creativity and innovation like never before due to the freeing of creative thought afforded by machine learning-based automation.

Culture Shift

Machine Learning will become so powerful in the future that it will shape culture by driving us to make better decisions and providing us a more profound vision for the pursuit of happiness and showing you the outcomes, explanations, or evidence that you might be missing in topics both big and small. And it will not only show you those missing elements but will also support you in weighing and making sense of them.

Machine learning will also bring about revolutionary personalization in the services and products based on your tastes, historical choices, location, even your DNA. This of course changes the way products are made, consumed, and marketed.

In conclusion, machine learning is changing everything quietly at the moment with the volume increasing dramatically over the next two years. Ignoring the technology is not an option, but it is important to measure your expectations and have a long game for machine learning in order to reap the highest rewards. 

It’s impossible to predict exactly where this phenomenon will lead us but in the words of Peter Thiel, 

“Not being able to get the future exactly right doesn’t mean you don’t have to think about it”. 

We are here to help you and think about your future and how machine learning can become a part of it as soon as possible. Please contact us to get started.

/by