Intelligent Processing at the Industrial Edge, a discussion with FogHorn's CTO

FogHorn CTO, Sastry Malladi, spoke with George Gilbert and Lisa Martin, co-hosts of theCUBE, SiliconANGLE Media’s mobile livestreaming studio, at the BigData SV event in San Jose, California. They discussed how FogHorn’s edge computing technology is being used by customers to predict conditions in real time, do condition monitoring or predictive maintenance.

 

Interview Transcript

Lisa Martin:  [00:14] Welcome back to theCUBE. I'm Lisa Martin with George Gilbert. We are live at our event Big Data SV in downtown San Jose, down the street from the Strata Data Conference.

[00:24] We're joined by a new guest to theCUBE Sastry Malladi, the CTO Of FogHorn. Sastry, welcome to theCUBE.

Sastry Malladi:  [00:30] Thank you. Thank you, Lisa.

Lisa:  [00:31] FogHorn, cool name. what do you guys do? Who are you? Tell us all that good stuff.

Sastry:  [00:36] Sure. We are a startup based in Silicon Valley right here in Mountain View. We started about three years ago, three‑plus years ago. We provide edge computing intelligence software for edge computing or fog computing. That's how our company name got started as FogHorn.

[00:49] Particularly, for IoT industrial sector, all of the industrial guys, whether it's transportation, manufacturing, oil and gas, smart cities, smart buildings, any of those different sectors, they use our software to predict failure conditions in real time, or do condition monitoring, or predictive maintenance, any of those use cases and successfully save a lot of money.

[01:10] Obviously in the process, we get paid for what we do.

George Gilbert:  [01:14] Sastry, GE populized this concept of IIoT and the analytics and the new business outcomes you could build on it, like Power by the Hour instead of selling a jet engine.

Sastry:  [01:29] That's right.

George:  [01:30] Wikibon and David Floyer did some pioneering research on how we're going to have to do a lot of the analytics on the edge for latency and bandwidth. What's the FogHorn's secret sauce that others would have difficulty with on the edge analytics?

Sastry:  [01:54] That's a great question. Before I directly answer the question, if you don't mind, I'll actually even describe why that's even important to do that. A lot of these industrial customers, if you look at, because we work with a lot of them, the amount of data that's produced from all of these different machines is terabytes to petabytes of data, it's real.

[02:12] It's not just the traditional digital sensors but there are video, audio, acoustic sensors out there. The amount of data is humongous. It's not even practical to send all of that to a cloud environment and do data processing, for many reasons.

[02:25] One is obviously the connectivity, bandwidth issues, and all of that. The two most important things are cyber security. None of these customers actually want to connect these highly expensive machines to the Internet. That's one.

[02:37] The second is the lack of real‑time decision making. What they want to know, when there is a problem, they want to know before it's too late. We want to notify them here is a problem that is occurring so they have a chance to go fix it and optimize their asset that is in question.

[02:54] Now, existing solutions do not work in this constrained environment. That's why FogHorn had to invent that solution.

George:  [03:01] Tell us, actually, just to be specific, how constrained an environment you can operate in.

Sastry:  [03:06] We could run in about less than 100 to 150 megabytes of memory, single‑core to dual‑core of CPU, whether it's an ARM processor, an x86 Intel‑based processor, almost literally no storage because we're a real‑time processing engine.

[03:20] Optionally, you could have some storage if you wanted to store some of the results locally there. That's the kind of environment we're talking about.

[03:27] Now, when I say 100 megabytes of memory, [inaudible] it's like a quarter of Raspberry Pi. Even in that environment, we have customers that run dozens of machine learning models. We're not talking about...

George:  [03:39] Like an ensemble?

Sastry:  [03:39] Like an anomaly detection, a regression, a random forest, a clustering, or a kmit, some of those. Now, if we get into more deep learning models, like image processing and neural nets and all of that, we obviously need a little bit more memory.

[03:54] What we have shown, we could still run...One of our largest smart city buildings customer, elevator company, runs in a Raspberry Pi on millions of elevators, dozens of machine learning algorithms on top of that. That's the kind of size we're talking about.

George:  [04:11] Let me just follow up with one question on the other thing you said, besides we have to do the low‑latency locally. You said a lot of customers don't want to connect these brown field, I guess, operations technology machines to the Internet, and I mean there was physical separation for security.

Sastry:  [04:32] That's right.

George:  [04:33] It's like security, Bill Joy used to say, "Security by obscurity." Here it's security by...

Sastry:  [04:38] Physical separation, absolutely.

George:  [04:40] Yeah, physical separation.

Sastry:  [04:41] Tell me about it. I was actually coming from, if you don't mind...Last week, I was in Saudi Arabia. One of the oil and gas plants where we deployed our software, you have to go to five levels of security even to get to there. It's a multibillion dollar plant and refining the gas and all of that. Completely offline, no connectivity to the Internet.

[04:59] We installed, in their existing small box, our software connected to their live video cameras that are actually measuring the stuff, doing the processing and detecting the specific conditions that we're looking for.

George:  [05:11] That's my question, which was if they want to be monitoring...There's like one low level, really low hardware, low level, the sensor feeds. You could actually have a richer feed, which is video and audio, but how much of that, then, are you doing the inferencing locally, or even retraining?

[05:32] I assume that since it's not the OT device, and it's something that's looking at it, you might be more able to send it back up the cloud if you needed to do retraining.

Sastry:  [05:43] That's exactly right. The way the model works is particularly for image processing because you need...It's a more complex process to train than create a model. You could create a model offline, like in a GPU box, an FPGA box, and whatnot.

[05:57] Import and bring the model back into this small little device that's running in the plant, and now the live video data is coming in, the model is inferencing the specific thing. Now there are two ways to update and revise the model ‑‑ incremental revision of the model, you could do that if you want, or you can send the results to a central location.

[06:15] Not Internet, they do have a local, in this example, for example a PIDB, an OSS PIDB, or some other local service out there, where you have an opportunity to gather the results from each of these different locations, and then consolidate and retrain the model, pull the model back again.

George:  [06:32] The one part that I didn't follow completely is if the model is running ultimately on the device, again and perhaps not even on a CPU, but a programmable logic controller...

Sastry:  [06:46] It could, even though a programmable logic controller also typically have some notion of a CPU there as well. These days, most of the PLCs, programmable controllers, have either an ARM‑based processor or an x86‑based processor. We can run in either one of those two.

George:  [07:01] Assume you've got the model deployed down there, for the local inferencing. Now, some retraining is going to go on in the cloud where you have...you're pulling in a richer perspective from many different devices.

Sastry:  [07:15] That's correct.

George:  [07:16] How does that model get back out to the device if it doesn't have the connectivity between the device and the cloud?

Sastry:  [07:23] If there's strictly no connectivity, what happens is once the model is regenerated or retrained, they put a model in a USB stick ‑‑ it's a low tech, USB stick ‑‑ bring it to the PLC device and upload the model.

George:  [07:35] This is how we destroyed the Iranian centrifuges.

Sastry:  [07:38] [laughs] That's exactly right, exactly right. Some other environments, even though it's not connectivity to the cloud environment, per se, but the devices have the ability to connect to the cloud. Optionally, they say, "Look, I'm the device that's coming up, do you have an upgraded model for me?" then it can pull the model.

[07:57] In some of the environments, it's super strict where there're absolutely no way to connect this device. You put it in a USB stick and bring the model back here.

[08:04] Other environments, device can query the cloud but cloud cannot connect to the device. This is a very popular model these days. In other words, imagine this, an elevator sitting in a building, somebody from the cloud cannot reach the elevator, but an elevator can reach the cloud when it wants to.

George:  [08:20] Sort of like a jet engine, you don't want the cloud to reach the jet engine.

Sastry:  [08:22] That's exactly right. The jet engine can reach the cloud if it wants to, when it wants to, but the cloud cannot reach the jet engine. That's how we can pull the model.

Lisa:  [08:29] Sastry, as a CTO, you meet with customers often. You mentioned you were in Saudi Arabia last week. I'd love to understand how your leveraging, engaging with customers to really help drive the development of Foghorn in terms of being differentiated in the market. What are those bidirectional symbiotic customer relationships? How are they helping Foghorn?

Sastry:  [08:53] That's a good question. We learn a lot from customers because we started a long time ago. We did an initial version of the product as we begin to talk to the customers, particularly that's part of my job. I go and talk to many of these costumers to give us feedback.

[09:07] My problem is really that I can't even give you connectivity to the cloud to update the model. I can't even give you a sample data. How do you do that modeling? Sometimes I say, "You know what? We're not technical people. Help us express the problem, the outcome. Give us tools that help me express that outcome."

[09:25] We created a bunch of what we call OT tools, operational technology tools. How we distinguish ourselves in this process from the traditional cloud‑based vendors, the traditional data science and data and tech companies is that they think in terms of computer scientists, computer programmers in expressions.

[09:42] We think in terms of industrial operators. What can they express? What do they know? They don't necessarily care when you tell them I've got an anomaly detection data science machine algorithm, they're going to look at you like, "What are you talking about? I don't understand what you're talking about."

[09:57] You need to tell them, look, this machine is failing. What are the conditions in which the machine is failing? How do you express that? Then we translate that requirement into the underlying models, the underlying Vel expressions, VEL is our CEP expression language.

[10:12] We learned a ton from using the interface capabilities, latency issues, connectivity issues, different protocols, a number of things that we learned from customers.

George:  [10:22] I'm curious with more of the big data vendors are recognizing data in motion and data coming from devices. The Horton Works data flow Nifi has a minified component written in C++, really low resource footprint. I assume that that's really just a transport. It's almost like a collector. It doesn't have the analytics built in.

Sastry:  [10:51] That's exactly right. Nifi has the transport. It has the real time transport capability for sure. What it does not have is this notion of that CEP concept. How do you combine all of the streams of everything there's a time to just get it for us from the devices.

[11:07] Whether it's coming from a device or whether it's coming from another static source out there. How do you express a recognition pattern definition across these streams. That's where our CEP comes in the picture.

[11:18] A lot of these seemingly similar software capabilities that people talk about don't quite exactly have either the streaming capability or the CEP capability or the real time or the low footprint. What we have is the combination of all of that.

George:  [11:31] You talked about everything is time serious to you. Is there a need to have an equivalent time series database in some central location so that when you determine what relevant subset of data to move up to the cloud or on-prem central location, does it need to be the same database?

Sastry:  [11:56] No, it doesn't need to be the same database. It's optional. In fact, we do ship a local time serious database at the age itself. If you have a little bit of a local storage you can downsample, take the results, and store it locally. Many customers actually do that.

[12:09] Some others, because they have an existing involvement, they have some cloud storage whether it's AWS, Microsoft, it doesn't what they use. We have connectors from our software to send these results into those existing environments.

George:  [12:22] You had also said something interesting about your tool set as being optimized for operations technology. This is really important because back when we had the net heads and the bell heads, there was a cultural clash and they had different technologies.

[12:42] Tell us more about how selling to operations is not just selling but supporting operations technology is different from IT sort of technology. Where does that boundary live?

Sastry:  [12:54] Typical IT environment, you start with the boss who is a decision maker. You work with them and they approve the project and you will execute that. In an industrial and OT environment it doesn't quite work like that.

[13:08] Even if the boss says go ahead and do this project, if the operator on the floor doesn't understand what you're talking about because that person is in charge of operating that machine. It doesn't quite work like that. You need to work bottom up as well to convincing them that you are indeed actually solving the pain point.

[13:25] The way we start with, rather than trying to tell them what capabilities we have as a product or what we're trying to do, the first thing we ask is, "What is their pain point? What's your problem? What's the problem you're trying to solve?" Some customers say, "Well, I've got yield, a lot of scrap. Help me to do my scrap. Help me to operate my equipment better. Help me predict these failure conditions before it's too late."

[13:49] That's how the problem starts. Then we start inquiring. "OK, what kind of data do you have? What kind of senses do you have?" Typically, they do have information about under what circumstances you have seen failures versus not seen failures out there.

[14:02] In the process of interrogation, we begin to understand how they might actually use our software and then we tell them please use our software to predict that. Just 30 more seconds on that. The other thing is that typically in an IT environment because I came from that.

[14:18] I've been in this business for 30 plus years, IT, OT, and all of that. Where we don't right away talk about CEP or expressions or analytics and machine learning. We don't talk about that. We talk about, look, you have these bunch of sensors. We have OT tools here, drag and drop your sensors. Express the outcome you're trying to look for. What is the outcome you're trying to look for?

[14:38] Then we drive behind the scenes what it means. Is it analytics, is it machine learning? Is it something else and what is that? That's how we approach the problem. Of course, sometimes you do surprisingly occasionally run into very technical people.

[14:52] For those people we can right away talk about here's analytics, here's machine learning, using an expression, all of that. That's kind of how we operate.

George:  [14:59] One thing that's becoming clearer is I think there's widespread recognition that data intensive and low latency work to be done near the edge. But what goes on in the cloud is actually closer to simulation than high performance compute if you want to optimize a model.

[15:18] It's not just train it but to have something that's prescriptive. That says here's the actionable information. As more of your data is video and audio, how do you turn that into something where you can simulate a model that tells you the optimal answer?

Sastry:  [15:42] That's actually a good experience. From my own experience there are models that require a lot of data. For example, video and audio. There are some other models that do not require a lot of data for training. I'll give you an example of some customer use cases that we have.

[15:56] There's one customer in a manufacturing domain where they've been seeing a lot of finished goods failures. There's a lot of scrap. The problem there was predict the failures, reduce my scrap, save the money.

[16:06] Because they've been seeing a lot of failures every single day, we did not need a lot of data to train and create a model to do that. In fact, we just needed one hour's worth of data. We created a model. We had completely eliminated their scrap.

[16:20] There are other kinds of models. Other kinds of model of the video where we can't do that on the edge. For example, we acquired some video files. Our simulated audio files. Take you to an offline model, created the model, and see whether it's accurately predicting based on the real time video coming in or not. It's a mix of what we're seeing between those two.

Lisa:  [16:42] Sastry, thank you so much for stopping by the cube and sharing what it is that you guys at Foghorn are doing, what you're hearing from customers, how you're working together with them to solve some of these pretty significant challenges.

Sastry:  [16:55] It's been a pleasure. Hopefully this was helpful.

Lisa:  [16:57] Definitely. Very educational. We want to thank you for watching the cube. I'm Lisa Martin with George Gilbert. We are live at our event Big Data SV in downtown San Jose. Come stop by Forager tasting room, hang out with us.

[17:08] Learn as much as we are about all the layers of big data, digital transformation, and the opportunities. Stick around, we will be back after a short break.