Description Transcript
Most organizations have the LLM part of AI implementation figured out. The real challenge is building AI systems that can reliably access the right data, make decisions based on complete context, and take action without creating new problems.
Teams getting real value from AI aren’t just analyzing data—they’re building systems that act on it. Support tickets that automatically route based on customer context. Forms that pull relevant information from your CRM. Sales processes that qualify and prioritize leads based on behavior patterns.
Analytics Engineer Sam Garfield sits down for a candid conversation about the messy reality of making data work for AI—and the practical steps that actually move the needle.
Read more 0:00 [Music] 0:17 Hi everyone. Uh, welcome to our webinar. 0:20 Is your data holding back your AI? And 0:22 here's how to fix it. Um, that was some 0:25 vibe elevator elevator music. Loved it. 0:27 I 0:27 >> was feeling it. 0:28 Uh, my name is Alicia Harper. I'm a 0:30 technical PMM here at Retool. And today 0:33 we're sitting down with Sam Garfield, a 0:36 staff analytics and data engineer at 0:39 Retool. And we're going to have just a 0:41 candid conversation about some of the 0:43 messy realities of making your data work 0:46 for AI and some of the practical steps 0:49 to get you there. So whether you're an 0:52 IC or a data leader or just interested 0:55 in this really cool intersection of data 0:57 and AI, I hope you guys can all walk 1:00 away with more confidence about your 1:02 data strategy and hopefully have some 1:04 concrete next steps that you can take 1:06 back to your team um and initiatives. So 1:11 before we jump in, a little bit of 1:13 housekeeping. This webinar is recorded. 1:16 So we are going to send a recording to 1:17 everyone at the end of the session. And 1:20 we're also going to hold a Q&A 1:22 afterwards with Sam. So, please share 1:24 your questions in the Q&A feature down 1:27 in Zoom. And if you're having any 1:30 technical difficulties, please reach out 1:32 to Daniel Kim, who I see just sent you 1:35 guys a little welcome. So, Sam, you 1:38 ready and get started? 1:41 >> Absolutely. 1:43 >> Okay. So, before we jump in, can you 1:45 tell me a little bit about yourself? 1:47 like what was your journey into data 1:49 analytics? What drew you into this 1:51 field? 1:53 >> Yeah. So, uh I actually was really 1:56 interested in data in university. Um 1:59 but, uh I hadn't really done the prep 2:02 work to, you know, major in a degree 2:04 that would would get me a job in data, 2:06 at least so I thought at the time. Um 2:08 but ended up studying economics where I 2:10 took as many applied stats classes as 2:12 possible. Uh, and then, uh, after 2:15 college, I actually became a high school 2:17 teacher, but between school years, I 2:20 needed something to do, um, over the 2:22 summer to make some extra money. And I 2:24 got an internship at Rocket Mortgage on 2:27 the data analytics team. Uh, and, you 2:29 know, by the end of summer, it was like, 2:30 this is what I want to do. Um, never 2:33 want to, you know, and then switch 2:34 careers from there. So, it's been, this 2:36 is my 14th year in data. 2:38 >> Wow. So, you but you've always had your 2:40 head in statistics and analytics, it 2:42 sounds like. 2:42 >> Yeah. Very interesting. I've also seen 2:45 you described as someone who is always 2:47 willing to tread the unknown territory, 2:50 which lines up with your history. Um, 2:53 what role do you kind of see yourself or 2:55 other data teams playing in this new AI 2:58 territory? 3:01 >> Yeah. Uh we'll get into much more detail 3:05 later probably, but um you know the uh 3:08 bottleneck for getting value out of AI 3:12 at your in the enterprise or nonprofit 3:15 or wherever you're trying to do it is uh 3:18 usually actually the data, not um you 3:21 know like the models or the uh prompt 3:24 engineering. Um and so you know there's 3:26 an opportunity for data teams to really 3:28 like seize the opportunity and open up 3:32 that bottleneck um and make AI something 3:35 that the enterprise gets value from. 3:38 >> Was there a moment that you kind of 3:40 realized AI was this next challenge you 3:42 wanted to take on or did it was it just 3:45 a personal interest or did it come from 3:46 a business challenge that you needed to 3:48 solve? 3:51 >> Yeah. um you know it's something that uh 3:54 has been a part of my career and 3:56 interest for a long time and say it's 3:58 something um the general theme of 4:00 algorithms and how data relates to 4:03 algorithms um you know I got my start 4:05 doing uh econometric forecasting which 4:09 is about algorithms um but then you know 4:11 after university um started working in 4:13 data I spent a while as like a 4:15 statistician and data mo or a machine 4:18 learning engineer um and I've seen you 4:21 know how uh algorithms can add value or 4:24 um create risks. Uh and that only kind 4:27 of increased with the advent of LLMs 4:30 which are kind of a new category of 4:32 system. Um so you know I've been 4:34 cautiously optimistic and interested uh 4:36 it says in the last year year and a half 4:39 that I really became convinced that 4:40 there's a lot of value there from LLMs u 4:43 that's waiting to be realized. 4:45 >> Interesting. It's interesting that you 4:47 were in ML and then you're now kind of 4:49 seeing the adoption of AI at a more 4:54 high level greater scale. 4:57 >> Yeah. Yeah. Seen similar commoditization 4:59 and like democratization like we had 5:01 with ML over time as well. 5:03 >> Yes. And with that comes a lot of 5:05 challenges in adopting that. So we want 5:08 to dive dive into the deeper questions. 5:10 Um, 5:12 I know like it seems like everyone is 5:14 dealing with like data connectivity 5:15 issues nowadays. I just data lives 5:17 everywhere. It's in a it's in 5:19 spreadsheets. It's in warehouses. It's 5:20 in other business softwares. What 5:23 actually happens to AI when your data is 5:26 scattered like that? And are there steps 5:29 you can take to start fixing that? 5:32 >> Yeah. So uh actually a friend of mine 5:35 who may be in the audience um he said 5:38 something that I stole and will now 5:41 steal from him um which is that there's 5:43 no AI without IIA. Uh IIA standing for 5:46 information architecture. 5:49 Um, and what that means is, uh, you 5:52 know, like I said earlier, the 5:53 bottleneck isn't generally the models or 5:57 the prompts, it's the data because you 6:00 need information and data in order to 6:04 get value out of the LLM that's specific 6:07 to the task you're trying to accomplish 6:08 or your business. 6:11 Um and you know the process of 6:14 architecting that data so that it's 6:16 accessible and consumable by an LLM is 6:19 um not always super straightforward. Um 6:22 so you know I think uh when you have 6:25 data that's scattered 6:28 uh it has a couple impacts on AI and AI 6:32 in the enterprise. So one is it makes 6:35 development or like reaching business 6:37 value much slower. Um, you know, it's 6:41 it's much easier to develop uh rapidly 6:44 develop valuable AI applications that 6:46 have business context if you have a 6:48 well- definfined set of data or small 6:52 number of resources to connect to to 6:54 understand what data is available for 6:56 answering a question or exploring a 6:58 domain. Um, 7:01 this is, you know, probably even more 7:03 true the more advanced the AI 7:05 application you have is. So example the 7:07 most basic level might be like a um a 7:11 enterprise GPT instance that has access 7:13 to your internal documents. But if you 7:16 are building um an AI agent that you 7:18 expect to like take some action on your 7:20 behalf um you those agents need to be 7:23 able to efficiently fetch uh and analyze 7:27 data as needed to make decisions on your 7:30 behalf. And the better the data is um 7:33 the better the decisions it'll make. 7:36 Um I think it's also you know uh it also 7:42 exacerbates one of the biggest risks I 7:45 think from enterprise data which is uh 7:49 debugging when things go wrong or even 7:52 identifying when things go wrong. Um, 7:55 you know, the more kind of data sources 7:57 you have to worry about, uh, the more 8:00 places there are for failures to occur, 8:03 the more likely it is that you aren't 8:05 going to have the right instrumentation 8:07 or observability in place to identify 8:09 when, you know, something changed and 8:11 uh, now your LLM or your agent is making 8:15 erroneous uh, decisions like, you know, 8:17 we've seen some outcomes like uh, agents 8:20 uh, giving people imaginary coupons that 8:23 don't exist. 8:24 uh for discounts on a company's products 8:26 for example. Um so I think the uh most 8:31 straightforward solution at least on its 8:33 face is to centralize your data sources 8:38 uh into as few places as possible. So 8:41 one way of approaching that is a 8:43 traditional data warehouse where you you 8:46 know can take in raw data from a variety 8:48 of sources. Then you can clean it, 8:51 transform it, maintain it um and monitor 8:54 it all in one place. Uh and then serve 8:57 that to an LLM or an agent. Um this is 9:00 particularly useful for you know if you 9:02 want to have context broad business 9:04 context uh you know you want to like 9:06 what are the trends in in the business 9:08 recently or or for across customers. Um 9:12 but you know you can also use solutions 9:13 like retool that allow you to connect to 9:16 different data sources in one place uh 9:18 and then standardize your governance 9:20 your access policies um as as well or 9:23 even better you can combine the two um 9:26 and uh get even better governance 9:28 >> right so I guess we're hearing a lot 9:30 about data using enterprise data 9:34 probably security is a really really 9:35 really top thing of mind is there 9:37 anything that you think teams should 9:39 consider when adopting AI and unifying 9:44 their data and getting an AI ready data 9:46 ecosystem. 9:48 >> Yeah. 9:49 Uh this is a big it's a big question. Um 9:53 so I think I'm a big first principles 9:55 guy. I like to think in terms of you 9:57 know what are the first principles that 9:59 you need to understand to understand a 10:02 system or make um make good decisions 10:04 about a system. And I'd say the key 10:07 principle for AI or is two. One is um AI 10:14 is not a destination, it's a tool. You 10:17 know, you you aren't just generically AI 10:19 ready. You have to have a sense of what 10:21 are we trying to accomplish with our AI 10:24 applications. Um and usually it's best 10:26 to start with a relatively narrow range 10:28 of applications so that you can make 10:30 sure you, you know, stick to landing. 10:33 The other key principle I think is um a 10:37 phrase that I'll steal from one of my 10:39 favorite writers, Max Reed, which is uh 10:41 epistemic humility. 10:44 Um so what that means is um there's two 10:46 kinds of epistemic humility. In other 10:48 words, uh being careful about, you know, 10:52 being overconfident about what you know. 10:54 Uh if you have epistemic humility, you 10:56 assume that you don't know a lot of 10:57 things. Um, so you have to have humility 10:59 a about what uh what do these LLM these 11:04 these systems know? What can they know? 11:07 How correctly do they interpret 11:08 information I provide them? Um, and then 11:11 two, you have to have humility about 11:12 what you yourself understand about how 11:14 the system behaves. You know, these are 11:16 highly complicated systems. Even the 11:18 foundation model developers like OpenAI 11:20 don't have great tools for understanding 11:22 what's happening under the hood. Um, and 11:25 so another way of framing that I've 11:28 mentioned in other places is to think of 11:30 the AI as like a, you know, well-meaning 11:34 CFO or even CEO who um is like 11:38 reasonably proficient with SQL. They can 11:41 write like pretty basic queries. They 11:43 can uh, you know, follow a link to 11:45 documentation to uh, understand the 11:49 data. uh they can even you know look at 11:51 examples to get a sense for how the data 11:54 is related to each other, how people 11:55 have solved problems in the past. But 11:57 you wouldn't just like expect your CEO 12:00 to understand all of the nuance of every 12:03 aspect of your business or to dive deep 12:05 into really complicated convoluted data 12:09 models and kind of come up with correct 12:10 answers by writing their own SQL. 12:14 Um yeah I think what that means in 12:17 practice uh is that to be AI ready uh 12:21 you have to have a number of facets to 12:24 your data. So one is you know unified 12:26 data ideally uh which we kind of 12:29 discussed above um you know making sure 12:31 it's easy for you to manage observe 12:34 transform document it all in one place. 12:38 Uh another aspect is data governance um 12:42 which helps you you know make sure that 12:45 you control access to data 12:48 uh at scale um you know one of the most 12:50 dangerous things about LLM's potentially 12:52 is regulatory risks um if imagine you 12:56 know LLM has access to private data it 12:58 shouldn't have or exposes private data 13:01 to end users um one of the best ways to 13:04 mitigate those risks is to have good 13:06 governance uh and systems that you can 13:08 use to manage access um and revoke it 13:10 where where needed and appropriate. Um 13:14 you also need to have quality. Uh it's 13:18 kind of a you know the importance of the 13:21 quality of your data is higher than 13:23 ever. Uh because you know an LLM it's 13:26 unlikely to look at the result and say 13:28 huh that doesn't align with my 13:30 understanding of how the business works. 13:32 This this uh metric clearly is 13:35 incorrect. Um, and so you really need to 13:38 have a human in the loop somewhere who's 13:40 notified when there are problems so that 13:42 you can intervene when an AI system is 13:44 like drawing incorrect conclusions. Um, 13:47 and then finally documentation. Um, you 13:50 know, it's been said that the most 13:52 valuable thing that uh a software 13:54 engineer can do at this point if you 13:56 believe that in the advent of AI powered 13:58 coding, the most valuable thing a 14:00 software engineer can do is document 14:01 their code rather than write it. And I 14:03 think the same is true um for both the 14:07 code that data engineers or analytics 14:09 engineers or data teams own but even 14:11 more so the data itself you know um if a 14:14 LM has access to really high quality 14:16 documentation to understand the data 14:18 then it can make really good decisions 14:19 about how to use it 14:20 >> right that makes sense so I guess can 14:22 you can you talk a little bit about 14:24 unstructured data and how that can 14:26 interact with AI? 14:28 Yeah. 14:30 So, um, unstructured data is really 14:34 interesting because there this tension 14:36 at the heart of LLMs and getting value, 14:40 um, out of unstructured data through 14:42 LLM. So, on the one hand, uh, the 14:45 highest value ad that an LLM can provide 14:47 is by automating the labor of getting 14:51 insights from unstructured data. you 14:53 know, um, LMS can make things possible 14:55 that were never possible before in terms 14:57 of, uh, crawling through and summarizing 15:00 huge reams of information. On the other 15:03 hand, however, the more extraneous 15:05 information you throw at an LLM, the 15:07 worse it performs on a given task. So, 15:10 these two things are in some of extreme 15:14 tension, you know, in practice, 15:15 especially if you're trying to develop 15:17 very high reliability business 15:18 applications. 15:21 So in practice what you you know I 15:24 usually have to do um especially if it's 15:26 really large unstructured data is to 15:28 walk a kind of a tight rope between 15:30 these these these two aspects of the 15:32 tension. Um so one you know you should 15:37 generally experiment with extracting key 15:40 pieces of information uh from the 15:42 unstructured data. So, one example might 15:45 be, you know, if you're building um uh 15:48 applications off of AI summaries of Paul 15:52 transcripts, um you might want to 15:55 pre-extract information from the 15:58 transcript like key topics that were 16:00 mentioned uh or the uh which of the 16:04 speakers are from within your company or 16:06 are not within your company. 16:09 Um, you also want to 16:13 try to generally transform the data that 16:17 you expose into a more LM friendly 16:20 format. Um, so for example like a vector 16:23 store is going to make it uh a the LM 16:26 can load more context into its memory at 16:29 once and access it more efficiently. Um 16:32 and then finally, you know, I've found 16:34 working on similar problems where we we 16:37 get insights from unstructured data at 16:39 scale, uh it's really useful to provide 16:42 an LLM with very clear examples and 16:46 particularly good um well doumented data 16:51 uh because you know it it helps the LLM 16:53 make sure it doesn't get overloaded in 16:55 the barrage of unstructured data um and 16:59 kind of interpret it correctly. 17:01 So, it's almost like you're you're 17:03 building this ETL, but now it's 17:04 specifically for LLMs 17:08 and giving it the right context that it 17:11 needs. 17:13 >> Yeah. Yeah. Exactly. Yeah. Sometimes you 17:16 can build an ETL for both humans and 17:17 LLMs. Um, but sometimes you just need to 17:20 build one for the LLMs. 17:22 >> We we serve the LLMs nowadays. 17:25 >> Yeah. 17:26 >> Okay. So I want to go back to that 17:28 centralized data warehouse topic that we 17:30 kind of talked about. I know this sounds 17:32 like the ideal state. Um obviously ideal 17:36 state but not not all teams are maybe 17:39 able to iterate that quickly or maybe 17:41 they're just working with a lot of 17:42 different APIs like Zenesk or Salesforce 17:46 or whatever. The tech ecosystem is 17:49 insane nowadays. But do you have any 17:51 advice on teams that are still either 17:54 working to centralize their responses? 18:00 >> Yeah. So um I think uh the key principle 18:05 here uh is that you need to identify 18:11 what are the aspects of our data that we 18:12 need to have the most confidence in that 18:15 we need to be the most reliable 18:18 in order to build the applications that 18:20 we need. Um you know when you're working 18:22 on centralizing your data there's a 18:23 million things to do and you really need 18:25 to prioritize. So again thinking about 18:27 what is the end state we're we're trying 18:29 to achieve and you know for example like 18:32 if your end state is starting by 18:34 creating a um support chatbot uh that's 18:37 going to have you prioritize um very 18:41 different data sources than if your uh 18:44 first application is going to be a uh 18:48 sales analysis agent or something. 18:51 Um so you don't necessarily even need to 18:54 store everything in one place. Um but 18:56 you know you need to be able to trust 18:57 your data. So you can find um other ways 19:01 of of creating governance and data 19:03 quality checks around uh that data. Um 19:06 you know it's hard to talk in two too 19:09 abstractive terms because it really 19:10 depends on the data source what's 19:11 available. So for example Salesforce 19:13 might have way more tools to help you do 19:16 that uh than say some other integration. 19:20 Um 19:22 so I'd also say that it's worth noting 19:24 that uh you know if you're if you want 19:26 to build highquality um agent 19:28 applications that really take action on 19:31 your behalf. Um integrating directly 19:33 with uh external systems is actually a 19:37 key aspect of that. So, you might 19:39 actually be helping yourself out um get 19:41 a get a boost up on developing good 19:45 agents that can write to or take actions 19:48 in external systems by setting up those 19:50 connections early. Uh you just need to 19:52 make sure that they're reliable and uh 19:55 well governed, 19:57 >> right? So, kind of having everything 19:59 under one hood, one platform can be 20:01 really helpful in getting started with 20:03 AI and getting giving it that whole 20:05 ecosystem. 20:08 Yeah, it's worth noting um you know 20:09 retool can be a very useful tool in this 20:12 situation because even if you don't have 20:14 something like centralized data 20:15 warehouse uh retools 20:18 connection uh library and query library 20:21 can be used as ways to um you know 20:25 quickly develop more of that governance 20:27 and centralization 20:29 >> right with sec yeah yeah right security 20:31 is kind of part of the platform is is 20:33 big as well. Um, speaking of agents, 20:36 like have you worked with agents? Can 20:38 you can you kind of describe and tell 20:40 the audience what some of the key 20:42 characteristics of an agent is and why 20:45 having this AI ready data is really 20:47 important? 20:49 >> Yeah, definitely. Um, yeah, I uh 20:53 worked on the launch for Retools AI 20:56 agent product actually. So, uh I think 20:59 I'm at least somewhat um familiar with 21:02 agents. Um but yeah, so I think the key 21:05 idea is that um agents are types of AI 21:10 applications that can act. Uh and you 21:14 know potentially there's potential for 21:16 agents to be kind of like one of the 21:18 keys to unlocking enterprise AI value 21:21 because if you uh set it up so that AIs 21:24 can act safely and securely um you can 21:27 really unlock all sorts of new 21:28 possibilities. uh you can really like 21:31 you know make a lot of people's jobs 21:32 easier um automate more types of tasks 21:35 that kind of thing. 21:37 So let's say in my view an agent has um 21:40 several aspects that make it separate 21:42 from like a regular LLM. Usually an 21:45 agent you know it's it's a particular 21:48 interface to an LLM that runs on a 21:49 special server. U there's several 21:52 aspects. One is going to have some kind 21:53 of goal orientation. So, you know, you 21:56 tell the agent what it must accomplish 21:58 before it finishes executing, whether 22:00 that's resolving a sport ticket uh or 22:03 finishing going through some decision 22:04 tree. Two, it needs to have reasoning 22:07 capabilities for it to make decisions 22:10 including when to terminate its process. 22:13 Three, it needs to have tools uh which 22:16 are the ability to ask the server it's 22:19 running on to run code for it. uh and 22:21 that includes fetching data from 22:23 external sources or say writing uh post 22:26 commands that uh hit an external API or 22:29 writing data to another database. So for 22:31 example, if you're writing um if you're 22:33 creating an uh a customer support 22:36 automation bot, it might need the 22:38 ability to write to your Zenesk to 22:40 update the status of a ticket. 22:43 Fourth, uh I think agents need knowledge 22:46 in memory. They need the ability to 22:47 fetch information to help them make a 22:49 decision using their reasoning 22:50 capabilities. And in some cases, they 22:53 need the ability to store information 22:54 permanently so that future uh agent runs 22:57 can then reference that information. And 23:00 then finally, you need an evaluation 23:02 framework. Uh so you can track, you 23:04 know, test different uh prompts and 23:07 different models and identify which ones 23:10 um are most efficient or effective at 23:14 accomplishing the task you need. 23:16 um and which ones are most cost 23:18 effective uh at accomplishing the task. 23:22 Now um 23:24 pools, knowledge and memory all depend 23:27 potentially in some part on having AI 23:30 ready data. So pools uh knowledge is 23:34 functionally a special kind of tool, 23:36 right? It's a tool that you provide the 23:38 LLM that allows it to fetch information 23:40 that helps it make a decision. If you 23:43 don't have well organized, centralized, 23:45 well-governed data, um the effectiveness 23:49 of the tools that you can provide your 23:51 agent to build knowledge are much more 23:55 limited. 23:56 Um you know so again uh in retool for 24:00 example you can kind of help manage this 24:03 better by using our connectors and query 24:05 libraries but you can also centralize 24:07 your data in a data warehouse that makes 24:09 it uh easier to write concise queries 24:12 that fetch the exact information that an 24:14 agent needs to make a decision. 24:17 Um but uh it's also easier to create 24:21 effective and safe tools if the data is 24:24 well governed. Um, you know, so again, 24:27 if you're talking about an AI system 24:29 that takes actions on your behalf, um, 24:31 believe it or not, there's some 24:32 companies that are even, you know, have 24:34 configured agents to, um, have access to 24:37 Stripe to make refunds or payments 24:39 automatically. You really want to make 24:41 sure that the data that's being used to 24:42 make that decision is extremely uh, well 24:45 governed and looked at very closely. 24:48 Um and then you know memory can be 24:50 related to the data team's work as well 24:52 whether that's u you know storing 24:54 information that an agent needs for 24:57 future reference in a vector store or a 24:59 database you maintain um again worth 25:02 noting ritual you get ritual DB retool 25:04 vector store for free uh which is one of 25:07 the reasons why uh I've done a lot of my 25:10 uh you know AI work in retool um and 25:13 then finally emails can also integrate 25:15 into kind of how a data team works Um, 25:18 ideally what you want to do is track, 25:20 you know, your evaluations for your 25:22 agents over time so you detect drift 25:24 from the original uh, performance and 25:26 then can take corrective action as 25:28 needed, 25:30 >> right? It's a bit of an experimentation 25:31 almost. Um, 25:34 >> yes. 25:34 >> Yeah, you've really been at the 25:36 forefront of a lot of these kind of 25:39 you've introduced a lot of AI solutions 25:41 even to retool itself as a company. Um, 25:45 do you think that like you being a data 25:47 expert, being on this data team, do you 25:49 find that you're best positioned to lead 25:51 this change? 25:53 >> Yeah. Uh, well, you know, I mean, 25:55 obviously I'm biased. 25:58 Um, but, uh, you know, as I said 26:01 earlier, there's no AI without IIA, 26:03 information architecture. The the 26:06 bottleneck generally is the data um, and 26:09 the context you're able to feed the LM. 26:12 So given that uh I see 26:15 data teams kind of across the world are 26:18 really like they're sitting at the mouth 26:20 of this bottleneck that's kind of 26:21 preventing 26:23 the realization of AI value in the 26:26 enterprise. Um and so you know data 26:30 teams particularly if they can upskill 26:35 um adopt tools that empower them to uh 26:39 kind of translate their data expertise 26:41 into AI applications. They can really 26:45 seize the opportunity and kind of open 26:48 up that bottleneck and uh have a lot 26:51 more influence uh within you know the 26:54 enterprise um and a lot more impact. So 26:57 uh you know yes but I'm I'm biased 26:59 >> right I mean well we are seeing a lot of 27:01 companies adopting AI but I know that 27:03 there's a challenge in actually getting 27:06 to a mature point with AI so 27:09 >> yeah I've and yeah I mean have you have 27:13 you well I've seen your apps so like can 27:15 you actually walk me through one of 27:17 these AI powered applications this like 27:19 actual mature AI solution that you've 27:22 managed to come up with while using 27:24 retool as a platform to do it. 27:27 >> Yeah, definitely. Um, yeah, I wish I 27:30 could talk about all of them. Some of 27:32 them, um, you know, kind of under NDA 27:34 given some of the cool stuff I'm working 27:36 on, but um, one example would be, uh, we 27:41 really 27:42 throughout my time at Retool, we really 27:45 developed uh, and worked spent a lot of 27:47 time working on developing the 27:49 enterprises ability to find the data 27:52 that someone needs for an application. 27:53 you know, um, we didn't have a 27:56 professional data team until the company 27:58 was like four years old or something 27:59 like that. So, there's a lot of data. 28:01 It's very hard to figure out if if what 28:03 you need is there and where it is. Uh, 28:06 we had an existing application, a ritual 28:08 application. 28:10 You know, one of the best things are 28:11 retool can build frontends really 28:14 quickly and cheaply. Uh, we had an 28:16 application that was a data search tool. 28:19 Um, and it allowed people to search 28:21 through the data in our Unity catalog, 28:25 uh, which is data brick system for kind 28:27 of managing metadata about your, um, 28:29 data warehouse to see if, you know, you 28:32 could find tables that matched, uh, the 28:34 use case you're looking for. Um, what I 28:37 realized is that, uh, we could actually 28:40 unlock new types of searches, 28:43 uh, by turning it into a rag app. rag 28:45 standing for recall augmented generation 28:48 where you uh fetch data in real time, 28:51 feed it to an LLM and have it help you 28:53 make some decision. 28:55 So uh what I did is I set it up so that 28:58 um the user could enter a semantic query 29:01 like uh I need data on selfserve AR 29:05 metrics 29:07 uh and we would fetch all of the column 29:10 descriptions all of the table 29:12 descriptions and other metadata in Unity 29:14 catalog and then for each data asset we 29:18 would feed that to an LLM. We would ask 29:20 it, hey, rate this on a scale of 0 to 29:22 100 in terms of how likely it is to 29:24 answer the um question being asked by uh 29:27 this this query string and then return 29:30 in a sorted list with some context on 29:32 you know how highly was it rated. And 29:34 this um you know increased the uh 29:38 ability for people to find data 29:40 effectively using the application um and 29:43 made it much easier for people who are 29:45 you know less technical who don't think 29:47 as much in terms of the kinds of uh ways 29:50 of framing questions that would work 29:52 based on the original apps um design to 29:54 access data more. Um however this does 29:57 illustrate you know the limitations of 30:02 your uh enterprise data um in the sense 30:04 that some of our earlier data sets were 30:07 less well documented uh for example some 30:09 of them had been built uh by people at 30:12 the company before there was a 30:13 professional data team and unfortunately 30:15 those kinds of data assets were less 30:17 likely to appear in the search results 30:19 because the LLM didn't have um access to 30:22 that data when it searched unity catalog 30:25 Right. So you were essentially able to 30:27 self-s serve data, give people a way to 30:32 ask creative questions about that data, 30:34 but still needed that data expertise to 30:37 kind of fill in the gaps and fill in 30:39 those 30:40 the unclean data bits. 30:43 >> Yeah, 30:44 exactly too, right? 30:47 >> Sorry, can you repeat that? 30:49 >> And there was like a front end to that 30:51 as well. 30:53 and actually save that themselves. 30:55 >> Yep. Yeah. So the front end retool app 30:58 uh you know big search box enter uh a 31:02 search query and then return a table 31:05 with uh metadata about tables in our 31:07 data bricks uh lakehouse uh kind of in 31:10 response to the query. 31:13 >> That's okay. How do you have you built 31:14 front ends before or was that 31:16 >> Oh, no. No. Never built a front end of 31:19 any kind except for 31:20 >> I mean like you I built like um front 31:23 ends using BI tools. Uh but they have 31:26 they're a little different from kind of 31:28 the full-fledged app development 31:31 framework that you have like RTOL. 31:34 >> Interesting. It's very cool. Um okay, 31:36 just in the in the interest of time, I 31:39 want to pause here. So we this is kind 31:41 of where we're at. We've finished our 31:44 questions for the day, but we do have 31:46 Q&A. Um, so we can jump into 31:50 some audience Q&A. 31:53 Okay, Sam, could you please elaborate a 31:57 bit on the multimodal unsaturated data, 32:00 so images and text for fine-tuning an 32:02 LLM? What are the inherent problems on 32:05 training with these types of data? 32:10 >> That's a really good question. 32:12 Yeah. Um, sorry. Can you The uh 32:16 multimodal saturated data you said 32:18 >> unsaturated data, so images and 32:20 >> unsaturated. 32:22 >> Yeah. So that's actually a great 32:24 question. I personally have never done 32:27 um really much heavy duty data 32:29 development work on um images and text. 32:33 Um I wish I could provide better advice. 32:37 Um, but I'm sure uh you know we can find 32:40 um you know can connect with me 32:42 afterwards on LinkedIn and I can see if 32:44 I can provide you some uh some help. 32:47 Okay. Um okay, next question. So when 32:50 building agents with complex qualitative 32:53 data, should we build one agent to do a 32:56 variety of steps or multiple agents that 32:58 work together? So like one agent to 33:01 clean the data to make it more useful or 33:03 one agent to look for themes, one agent 33:05 to edit the text etc. 33:08 >> Yeah. So this is uh from what I've 33:12 gathered kind of an open question in the 33:15 industry. You know we're kind of in 33:17 early days in terms of identifying what 33:19 are the best practices um how do you get 33:22 the most out of agents? Now, 33:26 >> from what I've seen, there are cases 33:29 where uh 33:31 breaking out an agent, what would be one 33:33 agent into a variety of sub agents that 33:37 handle more granular tasks um actually, 33:40 you know, performs much better than 33:44 having one master agent that is 33:47 responsible for implementing or 33:48 executing all the tasks in of itself. 33:51 And this makes sense on its face. Um, 33:53 generally speaking, you know, as I said 33:55 earlier, the uh less extraneous 33:59 information you throw out an LLM, the 34:01 better it is at making the right 34:03 decision. That also applies to, you 34:05 know, the narrower the range of tasks 34:08 you ask the LM to be responsible for, 34:10 the more likely it is also to make the 34:12 right decision. Um, now I'd say the main 34:16 counterveailing force for 34:19 breaking things out into sub agents more 34:21 frequently is that it becomes more 34:24 difficult to manage, maintain and scale. 34:29 Um, in one sense it might be more 34:32 scalable because some sub aents you can 34:34 actually treat like a aentic API. you 34:37 might have multiple workflows that kind 34:39 of query the same sub agent but also um 34:43 there's a existing you know kind of 34:44 inherent problem in um troubleshooting 34:48 and monitoring these systems and that 34:51 can become multiplied uh the more sub 34:53 aents uh you you create. So if you 34:58 prioritize performance um you should 35:00 definitely experiment with building sub 35:02 aents that handle more narotasks. Uh but 35:05 if you have fewer resources for 35:06 maintaining I would say you should 35:09 probably start by sticking with um you 35:11 know one super agent 35:14 >> right and you've worked a lot with um 35:17 agent evals. Did you develop evals for 35:19 our agents future? Were you working with 35:21 >> uh no I I didn't develop them. Um 35:24 actually I've only worked somewhat with 35:26 evals. You know I uh I I built a lot of 35:29 the uh data we use to measure how 35:31 customers are using evals. Um and you 35:34 know eval critical for um effectively 35:38 building and managing agents, 35:40 >> right? Okay. So that can kind of help if 35:42 you are managing multiple agents or 35:45 building even just a a mega agent. 35:48 >> Yeah. Yeah. Exactly. emails are going to 35:50 be the key to verifying that breaking it 35:52 out into kind of you know smaller sub 35:54 aents is actually more effective. 35:56 >> Okay, this might lead into the next 35:58 question which is can you speak to the 36:01 memory an agent can access to improve 36:03 its results over time and how can we do 36:06 this using retool? 36:08 >> Yeah. 36:10 So um I think this is the impression I 36:13 guess this is kind of a fledgling area 36:16 of agent development and best practices 36:20 as well but in principle 36:23 uh you know let's say uh I'll give you a 36:26 a more straightforward example from a 36:28 tool that many people might be familiar 36:31 with. So uh claude code uh is a tool 36:34 that some people use for developing 36:36 software. Uh one aspect of cloud code is 36:39 that you can create memory for it by uh 36:42 adding a file called cloud.md 36:45 that provides permanent context that the 36:48 agent uses every time it makes a 36:50 decision. So for example, if you're 36:52 working in a data engineering 36:54 repository, you might provide context on 36:56 here are testing standards, here's how 36:58 you um decide which tests to add to this 37:01 data set, here's the naming conventions, 37:04 that kind of thing. And then what's more 37:07 uh if you encounter a shortcoming in 37:10 terms of uh cloud code's ability to 37:13 write code in the way you want you can 37:15 actually have claude code itself update 37:17 its cla MD file uh and therefore you 37:20 have this kind of like self you know 37:22 improving and reinforcing agent. 37:24 >> Now you can apply that oh sorry what' 37:27 you say uh Alicia 37:28 >> it's so meta 37:29 >> yeah it gets super meta very quickly. Um 37:34 now uh what you can do is apply that to 37:37 say a business context with another type 37:40 of agent. So you know one example might 37:43 be that you are developing a um 37:47 customer support automation agent and uh 37:52 when the agent uh ingests a ticket or 37:55 triggers because a new ticket was 37:57 submitted by a customer. Uh there might 37:59 be things that you want to have future 38:02 agent invocations remember about how to 38:05 interact with that customer effectively. 38:07 Uh perhaps you know this customer 38:10 doesn't like being contacted via email. 38:14 Um stuff like that. Um and so what you 38:18 would want to do is set up first, you 38:21 know, identify what system you're going 38:22 to use to track um this this memory. So 38:26 you might set up a database where you 38:29 store um you know some summary string 38:32 about uh on a per customer level about 38:34 how to interact with this customer 38:35 effectively and then you create a tool 38:38 for the agent so that every time the 38:40 agent receives a ticket it checks do I 38:43 have anything in my memory store that 38:44 pertains to this customer if so fetch it 38:47 and load that into my context. if not 38:50 proceed um as normal. Uh there's a lot 38:54 of ways you can kind of architect this 38:55 but the same idea it's the same basic 38:58 idea you know like identify something 39:01 the agent needs to know in the future 39:02 next time it runs um or another time 39:04 that it runs create a system where it 39:06 can fetch that context when it runs 39:09 again um and then help it decide what to 39:11 do with that information. 39:14 >> Right? Right. So, kind of building those 39:15 tools around your agents workflows, etc. 39:19 >> Yeah. 39:21 >> Do you have any advice on how to reduce 39:23 LLM token consumption? 39:27 >> That's an interesting question. 39:29 H 39:33 >> uh 39:34 you know, that's actually not an aspect 39:37 of this that I've looked at in a lot of 39:41 depth. Um I mean one 39:45 general approach can be uh using concept 39:49 called compaction and is functionally a 39:52 kind of memory process 39:54 uh where you ask the LLM periodically to 39:59 uh summarize the uh text that's or other 40:03 tokens that are in its context window in 40:06 a much more concise format. then start a 40:09 new chat where the uh context provided 40:12 in the new chat is the summary of your 40:14 previous discussions. Um but you know 40:17 there's there's a lot of uh different 40:19 techniques for that. Um I'd say that's 40:22 traditionally something that you know 40:23 for example at retool more focused on by 40:25 the product engineers who are building 40:27 our AI products and and really want to 40:29 get the best value for our customers. 40:32 >> Right. Yeah. Which I think is why we we 40:34 make sure to expose a lot of that 40:36 information so that you can 40:38 Yeah, 40:38 >> keep an eye on it. Yeah, 40:42 >> shout out to uh the retool agents 40:45 product which has really cool 40:46 observability features that you can use 40:48 to track token usage by your agents on 40:51 your your keys. 40:52 >> Yes. If any of you have been to a 40:54 webinar, I'm sure you know who Kent is. 40:55 We all love Kent. 40:58 >> Yes. 40:59 >> Okay. Can you elaborate a little bit 41:00 more on implementing data oo 41:03 implementing data governance for AI apps 41:06 and are there any tools or patterns that 41:08 are already established that we should 41:10 be aware of? 41:12 >> Yeah. Um 41:15 so again I think this is a place where 41:18 um 41:20 best practices are still being kind of 41:22 figured out. Um now there's a couple 41:26 different perspectives or ways to think 41:28 about this question as well. 41:32 So um one question is how do we think 41:35 about the governance of the data that's 41:37 input into our our LLM from an external 41:42 system. Um so for example the tools that 41:45 are used by an agent to fetch 41:47 information or the vector stores that 41:50 you know a rag app uses to answer 41:51 questions. 41:53 Um 41:55 I think in this situation generally the 41:59 tools and techniques are pretty similar 42:02 to like a traditional data governance 42:05 system. Um, you might, you know, kind of 42:08 think about the, uh, 42:11 LLM or the service accounts that NLM 42:14 uses to access this data as kind of like 42:16 one of the accounts you pay the most 42:18 attention to. 42:21 Um, you know, the principle of least uh 42:25 permissions is going to be even more 42:27 important when you're talking about 42:29 something that a system that could be 42:32 kind of like um, you know, a 42:36 uh, polymath toddler who knows how to 42:40 fetch whatever write whatever code it 42:43 wants or fetch whatever data it wants, 42:44 but doesn't understand enough about the 42:46 world to know which information it 42:48 shouldn't divulge or which actions are, 42:51 you know, um, inappropriate to take 42:53 based on the powers it has. Uh, you 42:56 know, it's kind of like the the the one 43:00 account that you really want to make 43:01 sure you get the permissioning uh, 43:04 exactly right and even the permissioning 43:05 on who can kind of make changes to the 43:08 application itself. 43:10 So another aspect of this is thinking 43:12 about uh how do we manage the data that 43:14 is input by humans or users to a AI 43:20 system. Um this comes with uh 43:24 potentially a lot of privacy and other 43:28 concerns that can make this data 43:30 different from other types of data. 43:33 um you know, if you're talking about the 43:36 data that's getting from say your data 43:38 warehouse, 43:39 generally you're going to um hopefully 43:41 be making sure that PII or other 43:44 sensitive information isn't being sent 43:45 to your LLM. However, you can't, you 43:47 know, stop a human from just telling 43:49 your your AI chatbot that uh what its 43:52 address is and its birth date or it SS 43:55 social security number. Um, so I'm less 43:59 of an expert on this area, but there are 44:01 emerging tools like brain trust um that 44:04 allow you to kind of walk that fine 44:06 balance of tracking what is the data 44:10 that humans have entered into or or uh 44:13 introduced to our systems, how has that 44:15 caused our systems to behave? uh while 44:18 also um making sure you uh you ensure 44:23 that only people who should have access 44:24 to this information have access and that 44:26 sensitive information is kind of removed 44:28 before it's exposed to people, 44:32 >> right? Um okay, the chat is we got a lot 44:36 of questions coming in. Um okay, so 44:40 >> do you have any opinions on MCP or kind 44:42 of model context protocol? 44:46 >> Yeah. 44:47 So, um 44:49 it's interesting at retool, we have a 44:52 lot of um in-house tools that we've 44:54 built for our agents platform. 44:57 Uh and so, you know, I've had a lot of 44:59 experience um kind of looking at and 45:02 working with those tools. However, you 45:04 know, we also made sure that we added 45:07 MCP support to our agents product 45:10 because it is clearly um at the very 45:14 least a strong candidate for you know uh 45:17 the leading way to to kind of manage the 45:19 tools that you provide to an LL. Uh I 45:22 think MCP is very promising uh in terms 45:27 of being open source um very auditable 45:31 uh easy to kind of customize or fork if 45:33 needed um and you know a kind of really 45:37 impressive collaboration between um 45:40 competing companies in the space to 45:43 create a uh shared toolkit that anyone 45:46 can use. Actually, before I proceed, I 45:48 guess to make sure it was on the same 45:49 page, MCP stands for model context 45:51 protocol. It's open source software that 45:54 you can use to um define tools or 45:59 agents. Uh it comes out of the box with 46:02 a large number of connectors that you 46:04 can, you know, configure to connect to 46:05 your system. So, for example, I believe 46:07 it comes with a Salesforce connector out 46:09 of the box. Um 46:12 and uh it's standardized between um like 46:17 open AAI, anthropic um maybe XAI it's uh 46:22 like portable between models uh which 46:25 makes it you know super powerful. 46:28 Now I I have seen some concern about um 46:33 MCP some of the limitations potentially 46:36 of the MCP implementation. there might 46:39 be kind of the like lowest common 46:41 denominator um tools out of the box that 46:44 you get for each connector uh which 46:46 might not meet all of your business use 46:47 cases. Um and I've also believe you know 46:52 I might have seen some concerns about 46:53 like security or um governance in some 46:56 of the connectors MCP but um I don't 46:59 remember the details if that's the case 47:00 but you know very promising um in 47:03 principle. 47:05 >> Very nice. And do you think that the 47:07 future of AI lays in sort of querying 47:10 raw data or in working with a more 47:13 curated knowledge base? 47:17 >> So it's really tough to say because the 47:20 future of the underlying technologies is 47:24 uh very uncertain. um 47:27 you know I'd say let's say we uh achieve 47:30 uh super intelligence 47:33 uh like some people uh forecast in the 47:36 next several years you know at that 47:38 point actually maybe it doesn't matter 47:39 uh how you architected your data the uh 47:42 super intelligence will um rewrite 47:45 rewrite it to its own uh ends but um the 47:49 underlying idea I'm getting at is the 47:51 more if these systems get to a place 47:54 where they're able to independently 47:57 make reliably good decisions despite 48:00 having a lot of information to parse 48:03 then um querying you know less curated 48:07 more raw data might become the most 48:10 effective way of running the systems. 48:12 You know you might be able to just say 48:13 like hey 48:16 here's the uh database kind of have at 48:19 it you know here's the documentation 48:21 figure it out. uh but for the time being 48:24 in the foreseeable future 48:26 uh curated data is the name of the game 48:29 in my opinion. Um 48:32 I here's an example. Um, you know, so I 48:36 had a a stakeholder who 48:39 uh really wanted to 48:43 ask an LLM questions kind of like broad 48:47 questions about, you know, asking LM to 48:49 analyze really large amounts of 48:52 unstructured data. Um, and I set it up 48:58 uh, and the AI isn't wasn't actually 49:00 able to provide meaningful insight 49:03 because 49:05 it's too overwhelmed with context and it 49:08 just kind of ended up providing these 49:10 basic platitudes um, about the the data 49:14 it provided that that didn't really add 49:15 any insight. Um yeah I I have to be a 49:18 little you know can't get into too much 49:19 detail about what exactly I was working 49:20 on but suffice to say that you know 49:23 these systems really don't perform uh 49:25 well if you provide them a lot of 49:28 context a lot of information in order to 49:31 really be valuable and to make it past 49:33 that like proof of concept stage to a 49:35 real production application you need to 49:38 give it a very well- definfined narrow 49:40 task and very well- definfined narrow 49:42 information to use to like execute or 49:45 make decisions about that task. 49:47 So working with that problem, did you 49:49 approach that from more of a prompt 49:51 angle or were you going at it from more 49:53 of a data angle of how can I scrape and 49:56 grab the data that I need and then kind 49:59 of 50:00 >> Yeah. How did you approach that? 50:02 >> Yeah. Um so I approached that from a 50:05 data angle. So you know my my role was 50:08 uh okay how do we how do we architect um 50:12 a data set and then architect the system 50:16 we use to pass this context to an LLM so 50:20 that someone can ask questions uh of the 50:23 LLM. Um so it's functionally kind of 50:26 like um a rag app uh built on on some 50:31 like uh custom architecture. Um, so 50:33 yeah, there's really interesting 50:34 questions if you um get into 50:37 architecting data for AI of how do you 50:40 efficiently 50:42 um summarize the information, this 50:45 unstructured information and then pass 50:47 it to an LLM while still staying within 50:50 your context window and not providing, 50:52 you know, too much context or or uh not, 50:55 you know, um not wasting tokens by 50:58 repeating the same information over 51:00 again. Nice. 51:05 Nice. Um, I'm also looking at the chat 51:09 here. Um, okay. So, here's here's an 51:12 interesting question. What kind of 51:14 company is best positioned to benefit 51:16 from your offerings? 51:18 I might be able to answer this one. 51:19 Supposing a company uses real time data 51:22 heavy workflows and in-house solutions 51:24 compared to the more standard tools like 51:25 Zenesk and Salesforce. What kind of 51:28 support can RTOL give to help integrate 51:29 such workflows if that's supported? So 51:32 we like to say that RTOL is data 51:34 agnostic. So if if you are exposing 51:37 those workflows through an API or if 51:40 you're storing the information in a 51:41 database that is something that you 51:43 could connect and store within retool 51:45 and use within retool. So what kind of 51:48 companies are best positioned? I think 51:50 anywhere from people who are trying to 51:52 go from prototype to a production 51:54 application or even enterprise customers 51:57 who need that built-in security and 51:59 governance and even like source control 52:01 and auditability. 52:03 There's a range of needs that retool can 52:06 can fit and with whatever data you're 52:09 working with we can connect to that and 52:11 you will have governance over that. We 52:13 talked a lot about governance. I think 52:14 that's one of the main benefits of using 52:17 retools. have instead of having to 52:20 manage all of these API keys and doing 52:22 it in separate tabs, which I know we all 52:25 have crazy tab windows, you can do that 52:27 within the reach tool platform and it 52:30 makes it really easy to keep under 52:32 control who's using what, who has access 52:34 to what, which APIs you're working with. 52:38 Um, so yes, even if it's custom 52:40 workflows, that is supported. 52:42 Yeah, I would add um in some ways you 52:45 might be a better candidate for retool 52:47 or you might get some benefits that 52:49 other other types of uh retool customers 52:51 don't because you own the system that 52:53 you're connecting to entirely. Um and 52:56 then you can kind of tailor the like 52:59 affordances of say the API or whatever 53:01 system you need to connect to to match 53:04 whatever you need in reach. Um so that 53:06 could actually be a really great fit. 53:09 >> Yes. Yes. Um, okay. Gonna wrap up really 53:13 quickly. Oh, before that, my favorite 53:16 question. Sam, 53:18 what specific brand and shade of black 53:20 nail polish will maximize my ability to 53:23 data science this hard? 53:25 [Laughter] 53:28 >> Huh. I wonder it was the person who 53:30 answer asked this question named Mitch. 53:34 >> Uh, great question. I don't know. Uh, my 53:37 wife and I got manicures last week. Uh, 53:39 I don't know the the brand. She actually 53:41 kind of did all the she kind of chose 53:43 for me. Um, I can find out for you 53:46 though if you need to. Um, find, you 53:48 know, I'm on LinkedIn. Message me. Just 53:51 say you need information about uh, black 53:53 nail polish. 53:56 >> That was Mitch. 53:58 >> Love it. 53:59 >> That was not Mitch's question to my 54:00 surprise. 54:02 >> Okay, one more one more question. What 54:04 is something that excites you the most 54:06 about this new world of AI? and then 54:08 we'll wrap up. 54:12 >> That's an interesting question. Um 54:15 I'd say you know the most exciting thing 54:19 is uh possibility of kind of like 54:23 blowing open 54:26 what data teams can do and accomplish. 54:29 Um and you know kind of like some ways 54:32 maybe muddying the waters between 54:34 traditional job titles. uh might be that 54:37 software engineers need to become more 54:41 data data engineers and data analysts um 54:43 to become like truly effective uh at at 54:46 building the best AI applications. Um or 54:50 it might be that uh data scientists 54:52 start becoming you know more like 54:54 back-end engineers in certain ways uh in 54:56 order to to to deliver AI applications. 55:00 Um yeah it' be very interesting to see 55:03 how that goes. 55:05 >> Right. It's yeah the industry is 55:07 shifting a lot a lot. Um okay well Sam 55:11 thank you so much for your time for your 55:12 expertise. 55:14 Yeah you are so full of knowledge. It's 55:16 just amazing to hear you talk. 55:18 >> Well thank you 55:20 >> and 55:20 >> glad to sit down and chat. 55:22 >> It's great. Thank you everyone for 55:24 coming. Um 55:26 also for those of you who don't know we 55:28 are also hosting a live summit coming up 55:30 in October 7th. Uh we're hosting it in 55:33 San Francisco. So that's super exciting. 55:36 Uh Daniel's going to drop a link in the 55:38 chat. So if anyone is interested in 55:40 joining and attending 55:42 um we would love to see you there. 55:44 There's going to be a lot of great 55:45 sessions, a lot of great conversations, 55:47 good people. Um and we will follow up 55:50 with a link to this video recording and 55:53 please feel free to connect with both 55:55 Sam and I on LinkedIn. Reach out if you 55:57 have any questions. And Sam, everyone 56:00 loves the nail polish. 56:02 Love it. Um, 56:06 >> I'll make sure my wife knows and that uh 56:08 my nail pol my manicure place knows next 56:10 time I I go out. 56:12 >> Write a good Yelp review for sure. Um, 56:15 okay. Thank you everyone and yeah, look 56:18 forward to connecting soon. 56:20 >> Have a good day, guys.
